Magic Resist is calculated like this:
If you have 50 Magic Resist you will receive 33% less damage (50/(100+50)=.33
If you have 100 Magic Resist, you will receive 50% less damage (100/(100+100)=.50
If you have 200 Magic Resist, you will receive 67% less damage (200/(100+200)=.67
So the formula is MagicResist / (100 + MagicResist) = Damage reduction
The way Magic Pen is taken into this is by taking the Magic Resist and subtracting the Magic Pen.
For example, if your opponent has 100 Magic Resist, and you have Sorc Shoes (+15 Magic Pen), they will only be taking 46% damage reduction from you instead of the 50% that it would be without the Sorc Shoes
Their Magic Resist is 100 minus 15 (for the Magic Pen from Sorc Shes) = 85.
Then you throw 85 into the calculator.
85/(100+85)=46% damage reduction.
Make sense?