Early First Tower Question

So basically I just played a match where I was Darius against Yasuo. I wasn't worried because it's an easy matchup for me. My Graves wanted to gank top, and after running towards the Yas with no effect turned around and took an entire wave from me as I tried to ping him to back off, then sat there as their Trundle and Talon came up to kill him (I did make a mistake and tried to save him instead of just running. I died too). He then came back top and we killed the Yas but then he pushed the wave all the way to the turret and destroyed it, getting the first turret bonus gold. Normally I'd be ok with this, but in this case, it seemed to ruin any pressure I had in the lane as Darius and the Yas just got to freeze the wave at his turret and free farm. I couldn't successfully roam becuase the enemy team was too mobile (champs like Kat, Talon, Azir). I was pretty pissed this game.

Was I wrong to be pissed at the jungler's behavior? I've always thought that if you have the kill pressure in top lane and can make the enemy miss gold and xp you want to delay taking the turret so you can bully the enemy laner.