How does this even work??...

Well...show us his build? That very well could be either full AP or AP tank Malphite.

By ulting a bunch of squishies with no resistances multiple times.

High base damage abilities. If he constantly AoEs like 2 or more people all the time (Ult + E + W), coupled with Sunfire, he's bound to deal a ton of damage over the course of an almost 40 minute game. And because he's their team's engage, he's more liable to damage everyone first, so his carries might die while he engages and end up just dealing less damage than him overall.

I'd count your blessings. Maokai is a fighter secondary and has good AP scalings on his ult and Q. Had he built something like an Abyssal to deal more damage, he'd probably have dealt even more and killed you guys more often.

Well, tanks have a higher up time in a fight than squishies since they don't have to worry as much about blowing up. For instance, if your adc does 1000 dps but is only in the fight for 1 second , that's 1000 damage right? But if your tank does 100 dps but is in the fight for 15 seconds, that's 1500 damage.