mPen does not scale with AP or anything at all, unless you mean void staff, which gets stronger the more MR someone has.
Flat pen
is best against low-MR targets, as you can even go into negative MR with enough, doing more damage than your base.
Percent pen
is for targets with high MR, and it applies before flat pen. A target with 150 MR loses 60 to void staff and then 33 to sorc/liandry, leaving 47.
Pure AP is best when your champion has high AP ratios compared to their base damage and you are not dealing with a high MR target.
Each 100 MR/Armor acts like having +100% HP (100 is twice as survivable, 200 is three times, ect).
Note mPen is being moved around and changed in the upcoming rework of AP items.