I've played a lot of Fiddle before and I don't think he's garbage but he has certain problems.
First of all, he is quite Blue reliant and that can be a problem if your mid laner is mana hungry. It also means he is set back more than other junglers by a Blue steal early.
Secondly, he requires teams to make mistakes in order to be effective (either through a lack of vision or poor positioning).
Third, he is vulnerable to counterjungling because it's not as if he can use his ult. in a duel. An Xin, Poppy, Lee etc. can easily kill him or just steal his camps and run away. FIddle can protect himself with Fear but he can't actually duel them and sometimes Fear isn't even enough to escape. Even if your team is collapsing to help, Fiddlesticks has no dash or movement speed ability so he pretty much can't stop them from escaping if his Fear is on CD.
Fourth, the lack of a dash over a wall means it's harder for Fiddle to invade successfully even if it's just to ward. It puts you at a high risk but a Lee can hop away or a Gragas can jump over the wall. Vision, especially deep in the enemy jungle is crucial for strategy and helping laners stay safe so this is a bit of a negative due to the additional risk when Fiddle does it.
Fifth, he isn't particularly tanky and many team comps. expect the jungler to provide some beefiness.
Sixth, Fiddlesticks' damage does fall off a bit late game and he isn't really a huge DPS threat if his ult. isn't effective (and he can be knocked back or a Talisman active can save everyone etc.). You also can't ult. immediately and it can be interrupted. This makes him a bit unreliable and snowball dependent.
Seventh, his clear isn't the best and he will get outfarmed/behind a jungler like Elise or Nidalee if he doesn't have some big plays to make up for it. This goes back to his dependence on mistakes moreso than other junglers. It's a bit like Singed's problem but less exaggerated.
Eighth, he has poor waveclear and isn't good at split pushing. This isn't huge for a jungler but it's nice to be able to quickly push lanes when both laners backed. Having good waveclear and split-pushing/turrettaking ability gives your team another tool to win.
Lastly, he has many intricacies despite his simple kit. It requires excellent map awareness and knowing exactly where they see you channel, what walls you can jump etc. It's interesting that Fiddlesticks has the highest average number of games of any champion for those who play him.
With all that said, I'd still consider Fiddlesticks a top 10 jungler in the right hands and composition. He is very fun to play and guys like Nikkone have mained him to Challenger.