I searched this up a few times and there were many people saying different things. Kog'Maw deals 6% of the target's max HP magic damage on every autoattack for 8 seconds with a cooldown of 17 seconds. Vayne deals 60 + 8% of the target's max HP true damage every three autoattacks on a single target.
I would have to argue KogMaw for a few reasons. First lets look at damage output. Vayne's DPS per second (I won't look at items, because that sorta changes things, so just looking at kits) from her W is 20+2.66% max health true damage. Kogmaw hits for 6% every auto attack. Lets take an average champion health =2600; This is a fairly large approximation, but lets just go with it. Against a champion with this health, vayne's W does approximately 90 true damage per attack. Kogmaw's W does 156. This means that the champion needs approximately 56% magic resistance to lower kogmaw's damage to equal that of vayne's. This is the equivalent of around 130 MR, which is around bruiser level but a dedicated tank might have a little more. So it looks like, on paper, Kogmaw ends up doing a little more raw damage than Vayne does; except on tanks that have stacked MR.
But the on paper values don't tell the full story. Kog'Maw doesn't have his W in 100% up time really don't itemize for cdr anyway, resulting in periods of time where he isn't a very good tank shredder. Vayne also doesn't get her damage on every attack; she MUST land 3 consecutive attacks, which can be difficult to do given the situation. If she cannot successfully get 3 attacks, she essentially doesn't do any tank shredding at all. One should also consider the relative safety of each of these champions. After all, if an ADC is dead, they are doing no shredding. Kogmaw, with an effective 710 attack range, is much harder to kill than the 550 range vayne. Also, I know I said I would keep items out of it, but trinity force is a common pickup on Kog, which increases his kiting potential by providing more movespeed than Vayne gets in her kit (which increases his chances of survival). Vayne, however can use tumble to reposition and dodge in ways kogmaw cannot.
All in all, I'd say the are about equal, actually giving the edge to Kog'maw because of his safer nature, no strings attached % health damage, and the fact that most champions don't build enough mr to reduce the magic damage to that of vayne's.
Kog'maw is a better tank shredder because he deals mixed damage which forces the enemy tank to build all three defense types (armor, MR, HP), has a % resist shred, % HP shred, and several built in steroids like increased range and AS.
Vayne (although very annoying because of how safe her early game is for how well she scales) does not shred as hard as Kog maw and is countered more heavily by armor (rather ironic really) and AS slows. Dealing % HP true damage may seem to have no counterplay, but since it takes her 3 hits to deal her damage, she is more heavily impacted by anything that delays the third hit, and someone stacking armor and HP will still last a very long time against her because her standard physical DPS is pretty weak (essentially, she relies more heavily on her true damage than Kog has to because she only deals physical damage, where Kog deals physical and magical).
8% every three hits (not even mentioning you have to get consecutive hits for it to count) is only 2.67% Max HP per hit.
6% magic damage every hit is more than 2.25 times that amount, per hit, and it deals more per hit until the enemy has 65.5% magic damage reduction, or ~125 MR. assuming you are targeting the same tank, they will have been hit by your armor+mr shred which means they have to have a minimum of 174 MR for Koggles to deal less % health DPS than Vayne. Vayne's % health damage just looks more impressive because it comes out in chunks, but when you're dealing with a tank, you want persistent DPS, not bursts of damage.
Kog is of course weaker early game than Vayne, and takes longer to scale while being less independent late game, but he is a much better team fighter than Vayne because of his range advantage on all fronts.
Tough question but dont forget Koggy's Q Can lower resistances..
so with just 
he can lower defences by about 60% and do 14% health and then some..
But kog maw works best with a team built around him to protect him.. he needs a lot of protection. he doesn't have an excape as vayne does..
if you are on your own in a 1v1 i would say vayne as she has a ton of self peel to keep the tank away while kogmaw will get killed soon...
but if you are talking about teamfights or 5v5 i would say kogmaw if you have a team that focuses on peeling tanks and assasins from you and keeping you alive
Kog is the best tank shredder if u just focus on damage. but unlike Vayne he instantly dies cuz he has no self Protection. and a dead Kog Maw does 0 damage
depends on the enemy's mr health and armor
I look at it this way:
if your chasing.
if your kiting.
