Slice fennel stems crosswise (not too thin) and place in a medium bowl. Coarsely chop fennel fronds (you want about ⅓ cup) and add to bowl. Crush olives with a flat-bottomed cup or side of a chef’s knife and remove pit. Coarsely chop olives (you want big, chunky pieces). Add olives, oil, vinegar, orange zest, and red pepper to bowl; season with kosher salt and black pepper, then toss to coat.
Halve fennel bulbs lengthwise. Using a mandoline, shave fennel crosswise (you can use a knife, but you want the slices to be no thicker than ⅛"). Transfer fennel to a large bowl. Zest one-quarter of lemon over. Halve lemon and squeeze in juice from both halves; season with kosher salt and toss to coat. Taste and adjust with more lemon juice, if needed.
Divide olive mixture among plates. Top with cheese to just cover olives. Arrange shaved fennel over so olive mixture is covered, then season with flaky sea salt.