Normally if you're able to tank a few people and the others are still hurting you a lot, it's down to one of two things:
- The people who are hurting you are doing a lot better than the rest of their team, so they have more damage
- You've focused very hard on preventing one type of damage, so people who do the other type of damage are hurting you much more comparitively
The best advice I can give is to make sure you don't skimp on health - while you really do need to get armour and magic resist too, getting health alongside your resistances means that you're getting at least some bulk against both damage types. For example, buying a Dead Man's Plate is something you'll do to prevent physical damage, but the 500 HP on that item will also be a big help against mages.
The most important thing to remember when building defensively is that physical damage is much more important to itemise against than magic damage, in almost every case. Largely one magic resist item (Banshee's Veil or Spirit Visage, plus possibly Mercury's Threads for your boots) will be all you need to deal with the enemy team's mages, but you'll want multiple slots dedicated to armour. This is because mages do a lot less damage than ADCs overall, so if you have enough health and MR to survive a mage's initial burst, then that's all you need. However, ADC damage never turns off so you need a TON of physical bulk to be able to tank them for sustained periods of time.
A typical build for a tank like Shen can look like:

- this build gives you about 2800 HP, 155 armour (will be 160 next patch as Sunfire is getting buffed), 70 MR, 10% crit damage reduction and a ton of health regeneration. Combined with your runes + masteries and natural tankiness and you become extremely difficult to kill no matter what type of damage the enemy is doing with this build.