Ran and Experiment on Report System. Results not Surprising, System is Biased and has Flaws.

DunBanieMyAnnie·8/7/2017, 5:53:12 PM·9 votes·3,434 views

TL:DR Riots report system only punishes based off chat logs. It claims to be fair and asks players never to retaliate but only to mute people and report them, however it does not hand out punishments equally. Actions in game can physically ruin the game and you cannot prevent that loss, chat can tilt people but CAN be fixed by muting, however riot punishes chat but does not punish trolls. Something doesn't add up. I present a couple of options to mitigate this.

I do care about the issue and would love for the community to improve, as this is probably the only game where I've seen the trolls actually have the upper hand and are able to inflict punishment on good players with little to no consequence. It also promotes a "lets report so and so" environment just like how everyone in the real world tries to sue each other.

So, as someone who is/was utterly disgusted with Riots "chat scanning auto report bot" because it hardly gets rid of trolls and punishes good people. I understand it benefits riot not to have a tribunal where players decide whats best for the community and riot moderates it but rather replace a human interaction with a piece of code which they hardly have to oversee, but I'm not sure how that's best for or even helps the players. If an equal amount of resources from the cost savings was devoted to player issues Riot please feel free to let me know, in which case the return on your investment has not broken even yet.

However, despite all of this I'm a person of reason so even though I was pissed, when talking to some friends they wanted to try some impartial experiments I figured it might be a good idea Note: I did not participate in any of the games only looked at the data after. So no use in bashing me for ruining "normal" for random people I didn't order, execute or encourage it, just examined the results. Feel free to point out any flaws in the design or defend Riot if you want.

Three accounts were created

Account A: Lets call this the FLAMER - This account always played exceptionally well in game but said a lot of nasty things in chat. Account B: Troll -This account never said anything negative in chat, although did occasionally claim to be trolling or not care but would follow up by later saying they are not trolling Account C: Neutral Party

During each of the games the above scenarios would take place, Account B would run it down mid, build troll builds, intentionally feed or go to the wrong lanes, but was silent. Account A would get mad at account B. At the end of the game--not every game because it would diminish the report effectiveness account C would report one or both of the players-equal amount of reports over the total duration, and would encourage people to actively report the troll. Note that this was a biased hypothesis against the troll because I really wanted to stress test the system. Now I'm happy to provide all of the data for each game etc...but, basically what happened was by the end of 50 games,

Account A: received multiple reports and account bans, finally was perma-banned Account B: Absolutely NOTHING. Account C: Nothing-as expected.

There is a plausibility Account B would go straight to perma-ban or ban without warnings because a chat ban wouldn't have harmed them, but if that scenario exists 50 games seems a long time to not have a punishment. Now, during the games while account A had toxic chat, any player could have muted account A, the overall impact would be small but would negatively affect the player experience by prohibiting communication which can be useful in-game to coordinate strategy if used effectively, however Riot has probably determined this is negligible in random solo-q type games because players rarely can develop a good enough relationship to effectively use it, which is why I suspect they allow players to play with chat bans, which I'd hope they wouldn't do if the data showed that players with chat ban substantially lose more often from lack of communication. So Account A could have been muted and would not have affected the game. Player B did not say anything in chat, but there was no way to mitigate or "mute" the actions of a troll. There is no way to make it so their deaths don't negatively affect your team. There are likely three options. A) Ignore the troll- this may work if they get tired of trolling but it has no tangible effect and leaves it entirely up to the other player. Maybe you playing will inspire them to try, maybe not. You will lose LP for the loss but can report the troll hoping to clean up the community B) Try to reason with the troll via chat, be mindful that anything you say can and will be used against you to the fullest extent of the chat bot which scans logs. There can be a very fine line of what riot deems is aggressive towards an individual, and showing anger to someone wasting your time will likely get you reported for escalating the situation

Here I don't see a winning situation, if the report system did anything then option A would be a consolation to player C and A who were negatively affected by player B, but since it doesn't they lose in both scenarios.

Now in conclusion I FULLY understand why it's VERY VERY hard to report trolls and have any sort of CODE detect it. So I present 2 solutions to be pragmatic and practical.

Option A) Its nifty that Riot has a record feature. Now I know they cant possible save every single game that's out there, but if a game is flagged for multiple reports of trolling why not capture it? Why not put the decision BACK in the hands of the community or have a moderator examine it. In this case hand out a VERY HARSH punishment to demonstrate this behavior will NOT be tolerated, and the punishment can be harsher because it required riots human resources to enforce it instead of a system. Due to the new found stigma of reports for actions and trolling ACTUALLY having an impact it will quickly deter players, as currently I've seen many trolls say go ahead report me I've said nothing wrong and the system can't tell I'm a troll (or something similar). The only thing the system does detect is idling or AFK but it doesn't work if the person moves around in base every 30 seconds to crack the simple idle code.

Option B) Adjust the LP +/- according to the performance of the player! Now again I understand sometimes you have a case where someone truely has bad stats and they are a team player etc...well that's lovely but chances are its a fluke if they are ranking up solely by that game after game, Have a +/- 5 LP for performance. If I lose and I'm flawless on the game, give me only -10 LP and a troll who gives up, give them -20. Make them feel the wrath of their decisions more than those you've stuck them in the pot with. I understand riot wants there to be a shared risk/reward, and there is. Losing 10 LP still sucks, so theres still plenty of incentive to work together and not alone. Alternatively, if someone wins and is the carry give them +25 instead of +20 and give the troll they carried +15 instead of +20. This system can scale with your ELO standard LP awarded and also can scale with how bad you troll... Say you were only slightly better than everyone, maybe you only get +1 added to your score. If you were slightly worse? only -1 from your base. This would also work better than having to win multiple games in order to see a change in your ELO.

Bonus Option C) Stop using the stupid chat scanner to report people from games where its a human response to be angry because your system is ineffective and you give them no solution or outlet. Bring back the tribunal where i know many people went years with no reprimand, yet were banned through the scan bot.

44 Comments

Riot Tantram8/7/2017, 6:20:09 PM18 votes

Riots report system only punishes based off chat logs

This is not accurate. Our systems penalize for harassment, intentional feeding, trolling, and AFKing.

However, it is much harder to detect intentional feeding or trolling. The obvious response most people think of is to just penalize people who have a poor KDA. The problem with that is I can guarantee you I could go through each and every players' match history and find multiple games that would match this criteria, and the person was just having a poor game. Man, it would suck to get banned when you're already tilted off the face of the earth for having a poor game.

Most of the possible solutions you outline, involve manual review. This just isn't possible given the large number of games. It would suffer from the same problems Tribunal had, which is why it was taken offline. The tribunal couldn't get through cases fast enough, and was eventually backed up with years of data needing to be reviewed. This resulted in players either never being addressed, or being penalized for something they had no memory of that happened a year ago. This pertains to your option A and option C.

Adjusting the LP of players according to performance has a whole host of problems. Why would you ever want to play that team enabling tank or initiator? You may be hard carrying the team, but you are going to lose more LP because your KDA isn't as good. As a support, why wouldn't I only play Brand or Vel'Koz? I'm sure to have a better KDA.

That being said, we are constantly updating our systems, we aren't just letting them run and calling it a day. We don't post it in the patch notes, but we are constantly deploying. It's frustrating when you lose games, it's even more frustrating when you lose a game where you played well (win lane lose game, am I right?).

HalcyonDweller8/7/2017, 6:23:47 PM4 votes

I like how you presented this. I came to this thread expecting to see, "I ran an experiment and have concluded that Riot sucks, lol!"

But I'm pleasantly surprised to see that you presented a problem / hypothesis, tested and proved it, and then presented analysis and solutions. Are you by any chance a scientist? Because this was written refreshingly like a thesis. S+ job amigo!

  1. On how good the system is at its job: I completely agree, the system currently is very good at punishing toxic players while it is terrible at punishing the players who troll them. I do, however, disagree with removing the existing system. It should still punish toxic chat, because that is still a problem regardless of trolling.

  2. On why the tribunal was removed: I really like the point that you have made that the reasons for the tribunal being removed (too many cases to process) is mitigated now that many cases of toxic chat are already handled by the automated system.

  3. On bringing the tribunal back: In addition to that I really like the idea of bringing the tribunal back solely for the cases of reported trolls. I believe that it could save Riot a lot of time and give them a much more effective way to handle that sort of thing. And if Riot is worried about accuracy, they could always apply some level of AI / machine learning to monitor the tribunal judgements and alert Rioters / give them the final say when it comes to handing out punishment. I believe that this system of filtering cases down as much as possible before they go to Rioters would be the best way to get accuracy out of existing and past systems of punishment, while still saving Rioters the most time. It's possible that Reporting and tribuning could become part of the same process, where a player is reported, then after enough times the tribunal steps in and does all the prep-work of analyzing the games and the player's behavior, then it gets handed to an AI or a Rioter to sift through and determine the best course of action.

YerroFever8/8/2017, 6:11:10 AM3 votes

This is actually a really inaccurate conclusion for judging the punishment system. Your three accounts:

  1. Verbal abuse is VERY easy to detect thus easy to punish.

  2. Trolling/inting are harder to detect because you can't just simply go by KDA or build. You have to observe the game and this takes time. This is why you probably didn't get punished at the same rate. You need people to sit there and watch your games vs being able to just pick out flagged language.

  3. I appreciate that you did a control account trial. (Not being sarcastic at all, just in case that doesn't come across.)

The flaw isn't in your methodology but the flaw is in the conclusion you draw. If in like 3 months a permaban comes down on account 2, then the problem with the system is that it is slow.

archerno18/7/2017, 6:07:56 PM3 votes

A) impossible

B) Easily abused

C) Preposterous: If its human to insult others, its human to troll too. Good people dont rage at teammates

FightORFlight8/7/2017, 7:11:11 PM2 votes

For next step in punishment system i was thinking along the lines of recorded games for people who believe encounter abnormal amount of "trolls and feeders" in their games.

People who would like could record every game they play using Fraps or even Nvidia experience has that option now, then when reporting a person simply uploading it to report form same way we attach files in a ticket. Or tribunal could be brought back for the task.. Similar to how boards are designed, we could have down and up votes, every violation is put on display for a few days then at the end votes are counted (more upvotes -> offender walks, more downvotes -> punishment), and participating players would get bonus key fragment or "case solver" summoner icon for 10, 25, 50, 100, 500 cases solved in form of achievments

l2 4 G 38/7/2017, 6:19:14 PM2 votes

I love you

JlackBesus8/8/2017, 8:29:53 PM1 votes

option E - give up this game and just troll

MsYorine8/9/2017, 2:15:54 AM1 votes

Riot is a company, Companys need money, easier to perma ban toxic players and get them to spend money. Hence the broken system that will never be fixed.

FightORFlight8/7/2017, 7:39:33 PM1 votes

Okay lol... no need to get so aggressive... it was just a suggestion... A simple "it wouldn't work" would suffice. I know they don't owe us anything... hell even if such system was put in place, i wouldn't use it because i don't need it, but happier community = healthier environment right?

All i did was thinking of a way to catch those trolls who int but fail to get banned because their score doesn't show it.. visual proof would confirm that.. and you respond with such hostility....

BenBaxter8/7/2017, 7:54:45 PM1 votes

running it down mid LOL

Beas7ie8/7/2017, 9:29:25 PM1 votes

The problem is that unless someone is making it obvious like running a down a lane to feed, or playing a champ like Anivia or Kench and just abusing those specific skills to outright screw over teammates it's hard to tell a troll from someone who is just having a bad game.

The JigSAW8/8/2017, 3:09:08 AM1 votes

Its more than the behavior that needs correction, its also the ranked system and punishing those who do well because their team lost a game that they honestly could do little to stop. A mastery system that grants points towards mastery if you win or lose, not according to the grade you get (when leveling to 5, in all fairness, as 6 and 7 require S rankings) so if you get an S at level 3, you get about 100 mastery points for a champ in a 30 minute loss while the guy who won and went 0-7 gets around 800 points and a D+

As far as behavior half the people from what I can tell don't even want to admit toxicity is a real serious problem, the 5% number is thrown around a lot and frankly, they have to admit there is a problem before it can be solved.

MUSHROOM MIDGET8/8/2017, 9:11:47 AM1 votes

i agree the war against toxicity was a very bad idea. now people realize they cant rage at their teammates so they just troll instead.