I agree with you, definitely. The first buff feels very polarizing in its current form. Perhaps even putting it as dragon buff 3 would be better. It feels too much like baron buff--a late game objective--except now it's permanent. It's too unbalancing for an early-mid game buff unless Riot is trying to make the length of games shorter with more snowballing. Whichever team doesn't get the first dragon buff seems to be on their heels, very similar to how the previous seasons have felt if you're the team being besieged by the temporary baron buff. Sure, the next time dragon's up, the team that's behind has a chance to get the buff too, but by that time, any halfway decent team that had the advantage from the first baron buff should have snowballed enough that they have their second dragon in the bag, too.
If they had the second buff (damage to minions) be the first buff, it would simply allow the various lanes to push harder to take down the first towers, which is much more of an early game objective.