First of all, there's a pretty big issue with your idea:
...their winrate should be compared to the number of games that person gets detected as troll in.
How does the game know they're trolling in a given match in the first place to save for comparison data? Better yet, I may as well just ask; where does the comparison data even come from?
You can't really compare winrate by items, because that'd get too specific to provide reliable data, and in higher levels of play, some items may generally not get touched by the community at large. So, that's out of the question.
And you can't compare it to optimal/traditional builds, either, because the winrate of traditional builds would be skewed pretty hard when compared to an off-meta build.
And comparing it to the player's regular winrate with the Champion is a bad idea, too, since it could fluctuate for any reason, and trying new things tends to tank players' winrates.
Second, your suggestion still does something that Riot doesn't want to do: Force a meta. Riot doesn't want to force players to build certain ways, or play certain ways, etc., and punishing players for doing poorly even in one match where they screw around testing something new is beyond absurd.
Like, what if Tank-&-Spank Gangplank is gonna be the next big thing, but nobody wants to try it because the moment their winrate dips because of Dead Man's Plate or any other tank purchase, they'll get sacked with a punishment for being "trolls"? Or a Hyper-Speed Isolation Skarner build?
Should they be counted as Trolls because they're doing something unusual, even though they wholly intend to win?
Third, you're completely disregarding intent, amongst several other factors, though intent is the foremost of them. How do you even know if the person intends to troll with their build? Do they say it outright? Do they follow through when they say it, or do they play it straight?
What if their Riven Support pick got spontaneously hard-countered, and they lost lane hard as a result? Should the Riven be counted as a troll simply because of an unforeseen variable or two? Or perhaps they got ganked obsessively by the enemy jungler/midlaner because they were seen as an easy target?
What if the consistent losses have less to do with the player's build, but more to do with the circumstances of the match itself? What if the build was actually godlike, but teams gave up before it could reach the height of its power and make a difference in the match?
There are way too many factors to consider, that even touching on the subject I know I'm doing it an injustice.
The point is, your suggestion does little to fix the troll problem, and does more to implicate people who play off-meta as "trolls".
