I thought that magic pen was overall less important than armour pen because of auto attacks.
You get two main types of AD users - casters and attackers. Attackers build a lot of AS/Crit for that multiplicative scaling, casters prefer flat AD and armour penetration to increase the damage their abilities do.
The two classes of champion cannot share the same item system because they do things quite differently. If you let a caster do their job well while buying crit then their attacks would do far too much damage. Similarly, if you let an auto-attacker do high sustained damage while building AD and armour pen their abilities would get a little bit out of hand.
Ignoring LW (Which most AD champions should buy if there i a tank on the other team) from a design perspective, low overlap in items is the best.
Then you consider AP champions. With the exception of Azir, Kayle and maybe Teemo, all of them use their abilities primarily. This means they all fall into the same class of item users.
AP can increase the damage an assassin type champion deals, increase the shielding etc of a utility type champion, the burst of a burst mage, the DPS of a sustained damage mage... You get the picture. As AP ratios are changeable (unlike auto attack ratios, which are 1 in all cases except for Kalista), only one class of item is really needed.
Tl;dr - On AP champions that want high damage, building more AP has exactly the same effect as building more magic pen. This is untrue on AD champions due to the presence of basic attacks - crit and AS are different to AD and armour penetration.