The most common example of diminishing returns is armor and magic resistance. In this instance, the more you invest in a certain stat, the less each new stat increase gives. For example, you have 100 armor, which reduces physical damage by 50%. Let's say you buy thornmail and you now have 200 armor. Instead of the 50% doubling, it only increases to 75%. So instead of a direct relationship between the two, one is diminishing in comparison to the other.
On the other hand, the stat really isn't "diminishing" so much as being effective relative to your health. 100 armor means 50% damage, which is half damage. So 100 armor is like +100% health to physical damage. and 200 armor is 75% reduction, which means 200% of your effective health.
There are other types of diminishing returns, such as slows. In this case, the first slow is 100% effective, but any slow thereafter is only 50% effective.
As for penetration, it functions oppositely of armor and magic resistance. If they have little armor/mr, then flat penetration is exceptionally strong. for example, if the enemy has 30 MR, then they will reduce damage by 23%. 20 penetration means they have 10 MR, which reduces that to 16% reduction. That almost cut the reduction in half!
But say the enemy has 100 MR, for that 50% reduction, and you penetrate 20 again. That goes to 80 MR, which reduces it to 44%. That was only a 6% decrease...
I hope that helped!