Muting is no self-defense as it leaves the flamer party free reign to talk rest of team to report you.
Which has no effect on you.
as Support Singed case proved, the punishment system has no human involved.
You mean the support Singed that was manually banned by a human?
Thus enough players report you, your reports become false reports.
That's not how things work anymore... wait... Are you saying that being reported lowers your report value? Then nevermind. That's not how it ever worked.
Nobody check it, or if they do, they say "the punishment was correct, as you already got it". Riot won't admit their punishment is ever wrong. That is against Riot policy.
The plenty of cases where Riot reverted punishments beg to differ.
Now that we have the (frankly quite hilarious) first paragraph out of the way, let's cover the rest in a slightly more serious manner:
Muting might not be a perfect solution... but honestly, flaming back is not a solution at all. It's actually most likely going to make things worse.
And if you want to defend yourself in a civil manner without retaliating verbally? You can. I've not seen a single case of somebody being banned for that. It's pointless, but hey, at least it usually doesn't make things worse.
And last but not least: You don't need to defend yourself from other peoples' reports. This whole idea that you are going to get banned from false reports is laughable at best. Especially when you simultaneously claim it would take three months for the actual flamers to get punished.