Like others have said, armor penetration stacks multiplicativly.
If you have 50% + 50% armor penetration, you get 75% armor penetration.
The first 50% penetrates 50% of the targets armor. The second 50% penetrates 50% of the target's remaining armor.
As for the graphical user display showing 39% or 39% regardless of a 1 point mastery difference - the graphic user display in most games rounds information.
Seeing as runes (and per level stats) go down to thousandths (or previously 4 decimal places in Sion's AD scaling), it's safe to assume attribute bonuses that go into decimal places are indeed factored into formulas.