From what I understand, one of the biggest things that makes Graves such a great jungler is that 1. His Auto-Attacks push Jungle monsters away, and 2. His Quickdraw gives him damage resistances as long as he's doing what it takes to get the most damage in - which is to say, getting up close and personal.
From what I can wager from the last time I played him (which was a while ago), a lot of his damage comes from landing the return blast of his End of the Line, since it deals big AoE physical damage coming back. The only problem though is that the shell takes long enough on its own to hit the powder that most people will be able to dodge the second wave of damage, so for the most part you have to try and catch your enemies between a rock and a shotgun to get the most out of it.
His Collateral Damage, at least in my memory, should do a fair amount of damage too, but, as I said, it's been some time since I've played Graves, so I don't think I'm totally capable of figuring out what may be wrong. Generally, since Graves needs to get really close to get the most damage out of his New Destiny shells, I'd imagine landing Collateral Damage isn't really the problem. (or, it could be, depending on your playstyle).
While I know this probably doesn't do much to help you, I hope it does help somewhat.