When to use the following champs at top?

Nasus is strong either against full tanks who can't harm you anyway or against low engage targets. Think champions like Mundo, who lack the upfront burst potential to kill you past your sustain.

Malphite is good against all around AD. The real question should be how should you build him.

Maokai is well rounded for most match ups imo.

Personally, I would go Nasus when my team needs a strong mid to late game threat (Orianna mid, Ezreal AD and any tanky jungler) and I would be able to farm something, not even lane necessarily since you can farm the jungle as Nasus, if you have the jungler's permission.

Malphite is what you pick against full ad comps or when you want to wombo combo (Yasuo/Orianna mid Graves,Ezreal, MF as AD and a Leona/Braum/Alistar suppport) but with the unfortunate loss of DFJESUS full ap burst malphite is now only a pipedream, so I suggest going Frozen Fist/Abyssal into tank.

Mao is truly beastly in a coordinated 5v5, so unless you have faith in a random team I would only go him in a group setting where coordination around teamfghting and objectives is easier and because he has so little damage in the mid-late game he's very hard pressed to solo carry games. Also, in case you were wondering, Righteous Glory is a hilarious rush on Mao if you can group up your team, with the double active on it allowing your team a lot of teamfighting potential not even including your ult and if you are brave enough to Mao in Yolo-Queue Zz'rot Portal and Frozen Fist after your ROA/Righteous give you a bit of split push threat.