At the end of the day, it is the jungler's decision. It is his camps.
It also depends on how the mid laner reacts and who the jungler is. I don't play mana heavy champs in jungle so other than the first 1 I really don't need it anymore, but if the mid laner is demanding or raging at me, then he sure as hell isn't getting one.
It also depends on how the mid laner is, if they are 0/4, I am not really sure I will give it to them since they will most likely give it away immediately, likewise even if the 6/0 mid laner is doing amazing and gets caught and dies right after getting it, I am not likely to give it to them again if they did something stupid, if they got ganked or something is one thing, if they are overextended or lose a 1v1, yeah nope.
Same applies to red buff for me. If the ADC is doing well and isnt an ahole, I don't mind them taking the reds. If they aren't doing good or are raging I will make sure to take it everytime. I can assure you my smite is harder to beat than an ADC crit or Mage spell.
Overall yes, you generally should give your buffs to who makes use of them most, but it is the junglers decision at the end of the day. If they want them just because or they refuse to give them to you since you were rude, you literally got no right to complain about it since that is their cs, it is no different than a jungler being in your lane taking cs. So unless you let them take it and not be rude, you got no right to be rude to the jungler that you are taking cs away from. I remember when Sated was a thing, I would take almost every camp without question unless the person was there waiting for me already, there was no if ands or buts, if I was walking to blue, the mid laner better be going there too since once I start the fight, I am not stopping unless they are there.