Wits end. gives magic damage to on hit champions like Vayne and Lucian, making them able to bypass it anyways and make them even MORE of a threat to the tank because now the AD champions take less magic damage and deal MORE magic damage which they do not build against, but ends up being extremely overbearing. Not only that, but most champions (like they have mentioned) have built in magic damage into their stuff to help early on, but not really is it helpful later. For example, Yasuo's E can do around 150ish damage? i think if maxed. This would probably give plenty enough damage for you to not have to worry about the item at all... not including Static Shiv. How would it even stand when it would be so easily removed?
Also, a Vayne tried that vs Nasus. It didnt end well. All through the lane she was able to be dived and killed with little to no effort.
That said, then the tank should not be alone fighting the ADC. Even a non tank killer adc like Caitlyn can kill a tank solo.... thats because a tank isnt supposed to be 1v1ing and killing everyone. Tanks are there to possibly win lane, or to provide utility and a front line late game. If they win lane, then that means the opponent either is bad, or the tank player is really good.
ALSO Vayne is a late game champion who excels after she gets a few items. She also has one of, if not the shortest ranged auto range which can be heavily abused by champions like Caitlyn.
You need to focus less on "how can this person solo kill everyone" and focus more on "how can i help my team win vs that person".
Heck, Banner of Command used to do similar vs AP champions but on a minion.... It was removed because it was too hard to balance for the very fact that an item nullifies all the damage to a certain target. That is what you are going for, and next thing you know if they add that... you and everyone else will be asking for it to be removed or never build it.
It is a good attempt at an idea, but it for sure needs refined to make it even seem like it would be worth while. Some things you would need in an attempt to sell it is how will it be activated? will it have an activation timer? how much magic damage would it take? how long would it last? In an actual what way would it be useful? (for example Vayne or any other adc will just kill everyone else rather than go for the person immune to them, same as they do with Kayle and Taric etc.)