You can't report someone for being a troll if they don't say anything toxic
Yes you can, and people do actually get punished for being a troll. The downside to these punishments, though, is that it is often difficult to prove intent, meaning it can take a while before they actually receive punishments. I know this is quite unfortunate, however, when these players do get caught, they instantly receive a 14 day ban, with a permanent ban for a second offense.
And at the same time a %%%%% could be perfectly nice in chat but some bag of dick tips could report him and without any proof could get this %%%%% ban for doing nothing
If a player isn't trolling and they don't say anything punishable in chat, then any reports against them for that match will be thrown out as false reports and the player won't be punished. If for whatever reason they are punished for nothing, Riot has no problem lifting it provided it is indeed a false punishment.
Remember Dunkey. All he did was talk shit to a troll and you banned him
There's a difference between "talking shit" and being a flat out dick, and Dunkey was being the latter. I mean, he told someone they should be shot down and killed... OVER A VIDEO GAME. Now don't get me wrong, trolls are annoying and frustrating to deal with and they should absolutely be punished. However, there is absolutely no reason to tell someone that they should be killed, whether it's through suicide or by someone else killing them. That is absolutely unacceptable and should also be met with a punishment.
If someone's being a troll to a person who has put hours and hours of hard work into something they have the right to call that person dickhead or a %%%%bag
Sure, you absolutely have the right to do that, but you should know that Riot also has every right to dictate what kind of behavior is acceptable on their private servers, and they say that retaliating against toxic behavior with your own toxic behavior is not acceptable.
Putting that aside, though, have you ever heard of the saying "Don't feed the troll!" A troll only does what he does to get a reaction out of you. By reacting negatively and calling them a dickhead, idiot, moron, etc., you are simply giving them exactly what they want while also putting your account at risk of punishment. If you can learn to ignore the trolls and don't say anything to them, you are denying them this satisfaction.
A few more quotes that I believe also apply here:
"Never argue with an idiot. They will only lower you to their level and beat you with experience."
"Never wrestle with a pig in the mud. You will both get dirty, but the pig likes it."