As a jungle Pantheon main, yes, Panth does fall off a cliff late game. There are ways you can make yourself useful late game though, such as focusing your attention solely on the carries, and ulting to cut off the escape routes of your enemies.
The issue here is that guys like Renekton, Volibear, and Xin Zhao can splitpush late game, and Pantheon is only slightly better at taking towers than Mages are (hell, Ziggs does a better job at it).
Pantheon is far more team reliant on his successes late game than those three. If your team derps, those three can just split push. If your team derps when you're playing Pantheon, you're SOL.
Pantheon got indirectly screwed preseason with the lazer towers, but everyone was paying too much attention to his ability to block dragon shots to care, and after the dust cleared he was unplayable until Warrior got buffed and Black Cleaver got reworked.
Right now, some good ideas would be to normalize his ultimate's mana cost to 100 mana, and give him some more late game utility, like a slightly longer stun on his W from 1 sec at all ranks to 1.1-1.5. I don't know any other ideas right now that wouldn't push his early game over the edge.
Edit: Pantheon doesn't need a rework. He's an extremely well made champion that has a low skill floor but a high skill ceiling. He just needs some balance tweaks and some fresh ideas to reinvigorate him, and he'll be back to his good ol ass kicking top lane self.
Edit 2: Did I mention he counters Fiora? Because he counters Fiora.