Clearly you have never played a slow melee champion. Or been a jungler. 5 ms is a lot. Given that, in normal scenarios (i.e. Everyone is in "base" form (Kled is mounted, Gnar is not enraged, no abilities are active), using base values (level 1, no boots, no boosts), and excepting Pantheon and Yi, everyone's speed is between 325 (notable champions: Ashe, Mordekaiser, Azir, Caitlyn) and 350 (Akali, Jax, Trundle, Shyvana), 5 is 20% of the range. 5 ms is the difference between escaping a gank and being worn down. 5 ms allows you to kill or be kited to death. It's a minor number, but a huge impact.
It's even moreso on a jungler, who's very existence is roaming from camp to camp. Mordekaiser can be slow because he just sits in lane. A slow jungler will be worse because said jungler inherently cannot clear as fast, nor can he gank as fast.
Most impactful changes are usually not big, huge ones. +20 mana to a Q, -5 ms, +5 AD, increased AA delay…all of those have WAY bigger impacts than you imagine. You see the -5 ms and see a small nerf in speed. I see it and can automatically tell that it's a significant change, because Lee can no longer stick to Sion without applying CC.
And let's not pretend that Riot only took off 5 ms. We both know they also nerfed his ult's CD in early game by a significant amount. And that they cut off 25% of its base damage at the start of the season.