Nasus, malphite or jax?
Late game teamfights?
Late game teamfights?

Malphite, then Nasus, then Jax.
Jax has the most natural DPS out of the theory (in theory and on paper), but he has no where near as much tank as Nasus and Malphite can afford to get. That being said, if he is able to single out the ADC threat, he does his job well.
I say Malphite because of his kit, he doesn't really need to do all that much damage. He just ultis in, knocks up some, then CCs and disables the ADC and possible bruiser.
Nasus is next because of his wither and infinite scaling means he can afford to build full tank. Focus him, and then his team is able to swoop in.
Malphite has the best transition into the mid-game and late-game imo.
Nobody's going to stand and trade with a Jax- ever, that's just stupidity to do and so the other team generally plays around his kitability with something like 
Nasus scales horrendously well into the mid-game and early late game if you can keep your Q farm up pretty well, but doesn't provide what Malphite does in the way of CC, hard engage and tankiness.
In short, either or between Malphite and Nasus. Also jax just got nerfed.
Nasus is a different kind of tank. He doesn't need cc because the team focuses the shit out of him anyway. Want to know why? Because his Q chunks half your health in one hit late-game.
Late game, an ulted Nasus at full build is the god of tanks.
You have to get to the insanely late game for that to be effective - and until then, Nasus is an inferior force.
Malphite still does a damn good job of being a great engage with ridiculous amounts of tankiness while still pumping out more than enough damage throughout the game

R > W > E > Q
Is his ability order supposed to prove something?
No? It's the specific Ability Order for that build path.
Out of curiosity, why opt for
over an on-hit item like
, which also frees up your trinity force slot (I feel as though trinity is pretty situational on malphite where Iceborn can be used throughout any scenario as he's still very kitable despite trinity force)
Read his W Passive.
AD steroid, I'm aware. Doesn't justify building one expensive AD item that he'll see usage out of in the early game over something that he's going to get a lot more usage out of both in the early game, mid game and late game all the same.
imo
Late game Jax and Nasus are better than Malphite
Malphite is best in the mid game
Correction to some silly things said in here-
Nasus has the highest tank AND damage potential of the three. Pick nasus to shut down AA based top laners, or hypercarry adcs Jax is the easiest to deal consistent damage on, and is slightly harder to kite. Malphite is the best initiator, and deals surprising damage when built for it.