The buffs are pretty strong, though.
1st: Stat steroid that is always indisputably beneficial.
2nd: It makes a huge difference between "your siege" and "their siege".
3rd: Influences map pressure and engagements + retreats.
4th: Same as the 2nd, and split-pushing + Baron + that Final Dragon is done a lot faster.
5th: You may never see it happen, but when it does, it almost always results in a won game.
In the time it takes you to kill a turret, you can kill dragon and already make it back to base. If you're left behind after grabbing a turret, they can potentially catch you or at least push up a lane (including the one you just pushed yourself) and you'll probably be spending even more time counter-acting it.
Turrets stay where they are, which is why you shouldn't risk a death just to take it. But Dragon's a neutral objective that can be stolen, and is always better taken when they can't contest it. You might say that they would be spending too much time compensating the lost turret to contest, but it's easy to kill a single wave and immediately roam to dragon when your team is either spent because they still haven't backed, or are still trying to make it there because they did.