Consider the build she has:
Wants to Rush Blade of the Ruined King
Boots
Last Whisper
Looking at Champion.gg, two of her other main items are Phantom Dancer and Mercurial Scimitar, Auto Attack items with defensive utility, something that Guinsoo's will provide none of. Replacing these items with Guinsoo's puts Vayne at a higher risk, which at best we can argue is a break even.
What this essentially means the one item Vayne regularly builds now that can be replaced without significant changing in the rest of her build is Infinity Edge. Infinity Edge gives a bit of everything; bolstering your Burst potential with Flat AD and Crit Chance to multiply with Vayne's Ult and also doing decently well with Tank Busting with its Crit multiplier and synergy with Last Whisper. Guinsoo's on the other hand pretty much only puts her down the path of a Tank Buster.
But here's the thing.
Vayne is not the premier Tank Buster. Kog'Maw is. And Kog'Maw takes advantage of Rageblade both now and probably afterwards anyway, so trying to make Vayne compete with Kog here is just going to make a really inefficient Kog. Vayne's niche is indiscriminately tearing through everyone, squishies and Tanks alike, so a more rounded build like Infinity Edge suits her better. Not to mention, it doesn't have the +2 second delay to her powerspike in every fight she jumps in, especially since that delay is one that accentuates Vayne's biggest weakness of being focused down before she can properly do something.