Let's take a look at the patch notes back when they changed the item.
5.16
New Passive: 10% reduced damage from Critical strikes. This affects the total damage dealt by a critical strike, not an additive 10% reduction in their critical strike damage.
Important to note here is that they say that the damage reduction is reducing the total damage, it is not additive like the damage enhancement from Infinity Edge.
7.9
Reduced damage taken from critical strikes increased to 20% from 10%.
That shows that your math is already flawed since you are still calculating with the old 10% instead of the actual 20% damage reduction.
The fact that it is reducing the total damage of the attack means that a critical strike will just deal 80% of its damage.
Let's calculate an example:
A marksman has 100 AD and Infinity Edge (just to make it easier to calculate).
A critical strike from this marksman is supposed to deal 250 damage, 250% of his AD.
Now the damage reduction from Randuin's Omen.
Taking 20% less total damage means, that the marksman will deal 80% of his original 250 as damage.
0,8*250=200
Therefore, the marksman is just dealing 200% of his AD as damage after Randuin's damage reduction.
This is the same amount a critical strike would deal, if the marksman didn't have the passive from Infinity Edge.
Key here is that the damage reduction from Randuin's is not additive, it works multiplicative.