How much "worth" is the first dragon buff?

The occasion for this discussion is as follows: I had a ranked game where we were behind. All of us. At some point the enemy team grouped and started drake (1st buff) with their jungler's smite ready. The only ones close to the pit were me (Vlad) and our support (Leona). Leona spammed pings and told me to come with her to the drake so we can try and steal it, whereas I told her that it was no use and we couldn't do anything about it. Our jungler was at our blue buff. We didn't have smite and we were 2 people. We ended up going and, of course, failing. The enemy team got their drake and 3 kills from us. Leona kept saying in the chat that it was my fault because I didn't come to the pit immediately. So my question is: How much worth is the first buff? How much do you need to sacrifice in order to get it? Do we need to get it first at all costs? I would really like your opinions, people. Note: This was around high Gold-Platinum elo, if anyone was wondering.