Lethality vs Towers

I'm not sure if I'm coming off biased here and I'd like to bring up something about lethality that feels rather alarming. We all know by now that the majority of users who can utilize lethality well get it pretty good until they fall off in game. Getting straight to it, since the tower changes and lethality implications- I've seen lethality users take down towers waaay easier than everyone else. I believe that since these champions have the ability to roam, gap close, annihilate enemies, and have good ability refreshes, they should at least have some kind of downfall- like doing damage to towers apparently. From what I've seen a champion like zed for example, would do more damage per auto compared to a top lanner procing demolish with a frozen mallet and an ability. Yes, that's a lot of damage. You know, I'd be okay with this if this kind of damage was relative with everyone else, but it isn't and like most of competitive play- people go for the unfairly abusable things. Now I've asked around, searched on the famous league wiki and found out that it is the penetrative damage from the items which also applies to the tower. And I was thinking to myself, how is that at all remotely balanced? It is only them who can do this. I mean hey, I get it; assassins and burst are meta right now- but that shouldn't mean that towers are also just as malleable. Its just, imagine if every lane had that kind of pressure. Do you have any idea how fast every game would be? Well, I'd like shorter games, but I don't speak for everyone here. Anyways, I digress. Shouldn't Lethality's ability to apply to minions and towers be removed? And if no, why? you sparked me a reason to be intrigued.