Kog has more damage, higher range, AOE damage, poke and is tankier thanks to his build. He can't kite as well and is easy to dive though. Vayne is slipperier and kites extremely well, but has short range and is only single target damage, meaning she is extremely weak against AOE hard engage teams.
Early game, Kog has waveclear and ways to proc Sheen for his damage, while Vayne needs to farm with autos alone. Kog's build is also easier to complete and is very hard to keep down (he gets Triforce and BoRK, which come in small components) while Vayne needs a shiv, which has an 800 gold component. However, Vayne's better in a kill lane, since she actually has kill potential, while Kog doesn't, so that limits Kog's possible supports to pokers and sustainers.