1 armor = 1% EHP (against physical damage)
1 MR = 1% EHP (against magic damage)
This never changes at any point. The only thing that limits your tankiness is the limited item slots. You can calculate the maximum possible EHP against physical/magical damage using spreadsheets and then build those items based on the enemy team comp but that's unwise given other factors to consider. The only things that will ever change up your EHP value from there are true damage and defense penetration, but barring those you always gain a static 1% EHP of your listed HP value per point of armor and MR against their respective damage type.
Actual diminishing returns are when your points are just inherently less valuable per point than they are previously. Say that 10 points of dodge in a game gives y dodge chance up until z value. After z value is crossed the points now give a reduced value of x per 10 points. This means after the threshold of z has been crossed, the value of dodge has been diminished and mathematically reduced by calculation. Armor and MR in League never cross such a threshold, and as such they never face diminishing returns.
Now, there is a point where you would decide to build more HP over armor/MR, but that's not due to diminishing returns that's due to the now-increased value of HP in this case. If you have 200 armor and MR but only 2500 HP then getting 600+ HP is going to be more valuable to you than, say, 50 armor or 50 MR because it gives you defense against not only physical or magical damage but also the opposite type and defense pen, and not only that but it also gives a larger health buffer against true damage. But the value of armor/MR never diminishes, HP just becomes more valuable, like crit does when you have more AD/AS.