The problem with his tankiness is that its stacking. Meaning that A) before youre first quickdraw, youre just as squshy as any other ADC with a similar build, B) you need to use quickdraw 4 times to get max stacks, and C) his E has a 14 second cooldown reduced by .5 for each pellet.
What does this mean? First, your build has no crit, so Graves will only be firing 4 pellets each auto. Second, it has 20% CDR, so his 14 second E is lowered to a little over 11 seconds. If he is within melee range of you, his E cooldown gets lowered by 2 seconds per auto. So in the FASTEST of scenarios where he uses quickdraw at the start of the fight (not for the quick reload), he would need to: E - auto x2 (at this time, around 5 seconds left on the cooldown for E) - reload - auto - E - auto x2 - reload - auto - E - auto x2 - reload - auto - E. Youre talking minimum 9 autos and 3 reloads, which is easily around 10 seconds in combat.
But realistically, as you said, he is ranged. He isnt going to hit you with all 4 pellets - he will most likely only get 2 with a non-crit, non IE build. Meaning that youd have to have twice as many atuos and twice as many reloads between uses of his E. From my experience, the only time ive been able to consistently and quickly get max stacks with him is with a traditional ADC build that has 40% CDR and 70% crit, but even then the fight is usually over before the 4th quickdraw.