In short, no. Damn near every champion is annoying if you are playing what they're supposed to be strong against.
One of the most common things I see on the boards, and in game, is people substiting "toxic" for annoying. They're not the same thing, at all. As such one very important point, which is often left out of this discussion, is what 'toxic' gameplay actually means.
Toxic gameplay, the same as what used to be dubbed anti-fun gameplay, is removing interaction from a game / lane. It warps situations to either be foregone conclusions (If X then Y, regardless of player skill) or entirely removes the ability to 'play around' a champion. I'll give you an example:
Pantheon has a toxic kit. He operates on the level of "Is enemy in X range? Enemy receives poke." This is heightened by him having another clause of "Is Enemy with X-Y range? Enemy eats hard CC and an unwinnable all-in trade." There is no real skill expression involved with leveraging his strengths and there's no skill involved with avoiding them. It's purely "step within bubble, get hurt".
This is further exacerbated by the fact that this bubble is not small by any means. Old Darius used to have this design; he had a VERY small zone, but he controlled the hell out of it. This meant that even though 'step within the bubble, get murdered' applied, many champions could effectively harass him from outside of that bubble, or at the very least keep him from getting them within it.
Pantheon doesn't have weakness. His kit removes interaction, creating a dynamic between laners which is purely "don't get within range".
Another good example is old Fiddle Support. Back in S3 fiddle became a top tier support because his Q, when maxed, was a point-and-click hard CC of 3 seconds. It had no travel time, it had no pretenses, just click and apply. This was toxic because you couldn't dive him or his carry (unless you were Olaf), nor could you escape them.
There are more examples (pre-rework heimer, pre-rework assassins, pre-rework Taric, pre-rework-rework-rework Xin)
Annoyance, on the other hand, is Zoe and Yasuo (among others). These champions are healthy from a design perspective; they encourage both players to interact, have defined strengths, and come with incentives to take risks for greater gains.
Despite that a lot of people still hate them. Admittedly they have both had number problems at times, but beyond that people dislike how glaringly obvious their strengths are, as well as their moments of power. It's weird, but a lot of the league community has been conditioned to prefer 'hidden power' in a champion's kit, so long as they know it's there, than blatant displayed power of the same magnitude.
Anyway, to curve back, no. Champions should not be nerfed because people find them annoying, because a lot of people simply dub things they do not enjoy / which counter their playstyle as 'toxic'.