Vayne or Kog'Maw?

Kog'Maw is better late for team fights because range. Vayne is better at dueling.

Kogmaw with tanks to protect him = better than vayne in my opinion

The answer tends to be Kog'Maw. Superior range, and more importantly, armor reduction (which means your team's AD fighters are suddenly much stronger as well) makes him more reliable in team based actions, specifically team fighting.

Vayne has the better solo potential though. Much, muuuuch more mobility, due to how true damage interacts with most defensive abilities in game, she's almost guaranteed to get kills if you know how to play her well, and unlike Kog'Maw, she even has hard CC in her kit.

If you have a reliable team with good defensive synergy, play Kog'Maw.

If you want to hyper scale hard but you may need to defend yourself by yourself, opt for Vayne.