PSA: Why the tribunal is never coming back

Karunamon··11 votes·7,285 views

Around 81 million monthly players at the end of 2017. (You'll have to google this - the best source I can find is an account selling site, of all things)

Let's say that every single one of those players only played a single game all month. Thats 8,100,000 individual games (10 players per game - keep in mind, the real number is probably a lot higher)

Let's make an extremely conservative estimate that 1% of those games generated a single report. Flaming, inting, whatever.

That's 81,000 reports that have to be adjudicated. Let's split it and say that 80% of these reports are for flame, and 20% of these reports are for int. That's 65k flame, 16k int.

The flame reports are easy - someone just has to read the chat history. Let's say it takes someone about a minute to clear a single report (reading the chat and clicking punish or not punish, kinda like the old tribunal)

Int reports are a lot harder; someone needs to actually watch the replay of the game to make a decision on that. Just looking at KDA and build isn't enough, you have to actually watch how someone plays. Let's say these take about 10 minutes each.

With these numbers, each month, Riot staff would be spending, at minimum 1,350 hours handling flame reports, and 2,667 hours handling int reports. 4,017 hours each month, 48,204 hours per year. That's 25.11 MAN YEARS (40 hours/week, 1920/hours year), spent doing nothing but report handling!

(And what a shitty job that would be - imagine exposing yourself to the worst the player base has to offer, all day, every day.)

This doesn't scale.

Back in the old Tribunal days, the system was backlogged up to six months. This means people were receiving punishments for something they did _long _ after the incident had passed out of the memory of everyone concerned. This is completely worthless, both from a justice, and a reform standpoint, something Riot is on record as being large proponents of.

Some kind of automation is absolutely necessary to keep up.

The accuracy of the report system as it comes to flaming chat appears to be quite high - keep in mind, those of you demanding that a human handle your cases, that a human would be punishing you in the same way that the IFS would be. The rules didn't change.

If you want to play around with these numbers yourself, here's a google spreadsheet so you don't have to redo all the calculations.

60 Comments

Dr Rick Report4 votes

IFS is perfectly fine when it comes to toxicity. That should stay the way it is. The problem with this system is when it comes to anything else but the chat.

Haven't had anyone of my trolls get banned (or at least given low prio queue) for ruining multiple games since minute 1. Running it down mid, standing still, being AFK somewhere on the map, etc.

But don't worry, Riot, our savior for today is here with some new skins C:

PS: Yes, feel free to downvote me, I'm used to it.

Kaioko1 votes

Designing a skin is much easier than finding a way to automatically detect feeding/afk dodging. I mean instead of complaining about the problem why don't you design an actual implementable solution that Riot can then develop? All I ever see people is complain about an issue but they never offer a solution.

Dr Rick Report3 votes

The point is there. But I have 2 concerns:

  1. People at Riot do get paid a monthly salary to work on this stuff. I don't. If I were to get their salary, I'd be more than happy to design a solution. I'm trying to specialize towards AI and ML so it would be a good fit for me as well.
  2. Riot did find a decent solution. The OP said 16K reports for everything not toxic and 65K for toxic so I'll use the same numbers. Why would it be hard for Tribunal to come back for these types of reports? Replays are downloadable and watchable now, everyone can add features easily thanks to the new client, etc.

Toxicity is handled by IFS just fine. It works a bit more often than I want it to xP But all the others. The Tribunal would be fine for those with a Rioter (or more) checking maybe 10-20 games a week and their results to see if players are just voting for random stuff (and of course ban those players from using the system).

Kaioko1 votes

1.) Might have been my fault -- when I said design I didn't mean go out and do the coding that was what I mean by the "implementation phase." All I meant was to offer a viable solution that we could at least think through.

2.) OPs numbers are way off and speaking from personal experience -- as someone who was on the old tribunal -- it just didn't work both in terms of toxic detecting or trolling/feeding. The number of cases we were behind was so huge even when they implemented RP rewards for us. Worse even -- people use to almost solely vote "guilty" even when the party wasn't. They would remove the people who abused this obviously but that only made our backlog bigger.

Lastly Riot's P&L (as does most gaming industry ones) does not revolve on a high labor Opex. That just doesn't happen. Games might reach millions of end users but the core FTEs that actually handle the work is very few. Last I checked it was around 2,650. Majority of which have to be designers/coders/marketers/corporate supports and a whole bunch of other things you don't think of (have you done international tax for intellectual property -- spoiler alert it's fucking hard and takes a whole team of people!)

Dr Rick Report1 votes

Took my time to re-read the rules before posting this.

With the last paragraph, I have to agree. No doubt. But I think having more people hired for that job could have a really big ROI given the amount of people I know personally who quit the game or just refusing to buy stuff (because (s)he's really thinking of quitting too) because of trolls. NA is solely toxic, but on EU, it's a lot lot lot worse. On EUNE if they don't like a game (or the jungler), they will start trolling and mute everyone so the IFS won't do anything to them. A lot of lost money there too.

I think having a feature similar to Overwatch in CS:GO (with a group of honored (honor 5 only maybe?) and high/mid tier players) could be a viable solution to help both the community and also the IFS with some data it can use to work better.

With AI, the more data you have initially, the better your accuracy will be. Why not have this with the most loyal and not-toxic players? With the systems riot has today (including the honor system mainly), the trolls/cheap wh**es could be filtered out easily.

9 X1 votes

The solution is more likely to be found in assigning actual people to ensure everyone is given their just desserts. Yes, expensive considering the mess that is League now but that's the price for putting it off for so long.

But the price improves with time. If you take care of actual problem players consistently and fairly, people will be less likely to do troll things, making more players happy and less likely to do troll things because they aren't expected to be a perfect gentleman in every single match in the face of constant adversity. The need for manpower decreases, improved mood means more skins actually bought, more recommendations for other fresh faces to play the game...

Rito Games, do you remember what brought League of Legends to the dance in the first place?

Hell, I don't get paid enough for this.

Dr Rick Report3 votes

Hiring more and more people (even for a company in a better state like Riot) simply doesn't scale. You can't keep hiring more and more people who'll most likely leave after a few months given how bad a job like "Report Reviewer" would sound like.

What they need is something similar to what I've said in the other thread. Make a system that allows both IFS to train itself for inting/trolling behavior while maintaining this general idea of having a group of players (honor 5, g3+ or maybe p5 and above) who can judge by watching their replays (since replays are supported now, in the tribunal ways it wasn't) and tell the system if they were griefing or inting or doing what they were reported for.

This worked out quite well for CS:GO, why wouldn't it work out for League as well? A scalable, human controlled solution. Plus you get to train IFS so maybe it'll learn why it's wrong in some cases via the tons of new data it would be able to gain.

Timethief493 votes

If everyone plays a game its 8.100.000 since its 10 players a game (it would be less concidering TwTL but i doubt those are many).

Karunamon1 votes

That's probably fair given the amount of rough estimating we're doing - you can reduce every number I gave to 10%.. but this doesn't really change the overall point much. One, since the overall report numbers probably wouldn't change all that much (since you're having all 81M players having one game, individual games would generate more than a single report).

The other thing is that we know all of these players are probably playing more than a single game. Updated the numbers anyways.

Player3Th0mas3 votes

Did you just??

https://boards.na.leagueoflegends.com/en/c/player-behavior-moderation/2E8dqxs9-an-automated-system-banning-on-the-spot-without-context-is-the-only-way-to-have-a-ban-system-at-al

XD, what a coincidence

Totally agree btw. IFS is perfectly fine as it it, and it would be impossible to tribunal everything.

TempName748142 votes

i used to just spam the punish button because most of the time it was what the verdict ended up being. it was just a source of free IP (was it RP too?)

9 X1 votes

Toxicity is a problem, and of course it's going to require a larger allocation of resources to combat seeing as it was allowed to grow as big as it has.

The way I see it, the actual investment of human resources and subsequent results will create a snowball effect. If Rito Games actually punishes 95%+ of the evil-doers and the player base knows they're actually doing something, they'll be more inclined to be civil themselves. Maybe you know, actually happy to be playing the game League of Legends rather than "Whose Troll Is It Anyway?"

The need for that human manpower dwindles as the number of actionable reports goes down and you can gradually cut back. Naturally there will be a point where you want to stop cutting back and instead maintain, which somewhere along the line someone higher up in Rito Games forgot and decided fuck it, activate automatic ban bot.

Dragfin2 votes

[{quoted}](name=9 X,realm=NA,application-id=ZGEFLEUQ,discussion-id=L1PPeqqi,comment-id=0002,timestamp=2018-04-24T18:51:24.035+0000)

Toxicity is a problem, and of course it's going to require a larger allocation of resources to combat seeing as it was allowed to grow as big as it has.

The way I see it, the actual investment of human resources and subsequent results will create a snowball effect. If Rito Games actually punishes 95%+ of the evil-doers and the player base knows they're actually doing something, they'll be more inclined to be civil themselves. Maybe you know, actually happy to be playing the game League of Legends rather than "Whose Troll Is It Anyway?"

The need for that human manpower dwindles as the number of actionable reports goes down and you can gradually cut back. Naturally there will be a point where you want to stop cutting back and instead maintain, which somewhere along the line someone higher up in Rito Games forgot and decided fuck it, activate automatic ban bot.

All they would need to do is review a percentage of the reports. Let the player base know they will be manually reviewing reports and punishing accordingly. Eventually the second part of your post would happen. The OP states full reports on every damn game played which realistically that is not possible. For them to even go that route makes them delusional as well.

All one needs to say is that RIOT is lazy and this is why the problem has grown in numbers. They need to take a proactive approach and not the lazy one they have taken with this god awful auto ban shit that basically only catches chat toxicity. The rest of the toxicity goes pretty much unpunished.

Chermorg1 votes

The problem is not that the system cannot detect it. It is that Riot has a high bar of proof for someone intentionally trolling before they will be punished. Riot does not want to just ban everyone and their mother because they might have been intentionally feeding. It must be proven beyond reasonable doubt that the player was doing what they were doing to intentionally troll if they are to be punished.

9 X1 votes

Which is a big part of why actual people, not an automatic ban bot, would be involved to investigate such reports before action is taken.

Clearly frivolous reports would be punishable under this model as well as they'd be depleting actual resources. Obviously players should be given a disclaimer in advance that they shouldn't take the action of reporting another player lightly.

Chermorg3 votes

Even humans can't determine intent without someone admitting it very easily. What you think is obvious is likely not so from a third person perspective.

9 X2 votes

If it seems obvious to the players witnessing it, odds are it is that clear cut. If people are being actioned for doing something silly like running up and getting solo killed three times in two minutes, smart money says it'll happen a lot less above Bronze elo. The mindset of having to play safer in Gold elo to avoid being banned isn't ideal, but it's still better than the current trend of having no consequences for your actions unless it's as flagrant as repeatedly diving someone under tower you have no hope of killing.

Or you know, other questions would be boosted accounts which could be easily investigated through match history, and by the way, I haven't seen Rito Games do much about the clear offenders I've reported in this case either. A clear track record of picking a particular champion to lose? It's actually very easy, with actual people doing the looking.

Chermorg4 votes

Or you could, you know, not punish players for being bad at the game.

9 X2 votes

????

Chermorg2 votes

You want someone to be punished just because they try to fight the enemy 3 times. That's not intentional feeding. It's making poor choices.

9 X1 votes

If a player is in Platinum+, and they've given up three solo kills in two minutes, that's intentional feeding or cheating (boosted) in most cases.

Maybe not in Bronze or Silver, but in Diamond and Platinum there's little excuse for it, especially with the amount of players who outright declare they're intentionally feeding and/or don't give a shit and continue to ruin games long after they've been reported. It is a cesspool, they derive their amusement from ruining the game experience for others and Rito Games does nothing about it.

Chermorg3 votes

No, it is not. You cannot equate simple numbers to obtain information as to someone's intent.

This is why Riot is the ones punishing players, and the exact reason they wouldn't be able to use player report count as a justification - even if they wanted to do that.

9 X1 votes

You're absolutely right, you cannot equate simple numbers to obtain information as to someone's intent. That's why I laid out a very specific situation, of which I'm positive any rational person in medium elo can agree is definitely in violation of the terms of use and our game experience.

Rito Games isn't punishing anyone of consequence. I actually just remembered that time I put in a support ticket for Tyler1 hard-inting, even declaring his intention in the chat log and sure enough one of his primary streaming accounts four months later is using that very exact name. No, I don't buy your bullshit.

As an organization, Rito Games is legitimately doing nothing about toxicity.

Karunamon3 votes

That's why I laid out a very specific situation, of which I'm positive any rational person in medium elo can agree is definitely in violation of the terms of use and our game experience.

This, right here, is why players are not in charge of what is and is not inting. Smurfs happen, bad matchups happen, absolute fucking curbstomps happen even in high elo.

You ever acknowledge the imperfection of this system:

that's intentional feeding or cheating (boosted) in most cases.

Most != All.

Well sorry, I'm not willing to start banning innocent people just because they got a bad matchup. Better 100 guilty men go free than one innocent man be wrongly punished.

Karunamon3 votes

Show me a "wrong punishment" that wasn't quickly resolved.

Difficulty: Your definition of "innocent" appears to include people who flame others in the chatbox.

9 X1 votes

Nah, the difficulty is there are a couple people on the boards who seem to confuse what's right and wrong with the rules Rito Games has set for a video game, when my objective is to change those rules for the better.

Karunamon2 votes

[{quoted}](name=9 X,realm=NA,application-id=ZGEFLEUQ,discussion-id=L1PPeqqi,comment-id=00020000000000000000000000000000000000000000000000000002,timestamp=2018-04-24T23:01:12.196+0000)

Nah, the difficulty is there are a couple people on the boards who seem to confuse what's right and wrong with the rules Rito Games has set for a video game, when my objective is to change those rules for the better.

If I understand you right, the general gist of those rule changes appears to be "smack feeders and trolls harder and faster" (which I am perfectly okay with) and "stop smacking people who participate in troll slapfights rather than using /mute" (which I am in vehement disagreement with), and "people who have extraordinarily bad games at high elo are necessarily intentionally feeding" (which.. no. just no)

Am I on track?

Chermorg4 votes

Making poor choices and thinking you're doing better than you are are NOT against the terms of use, and they are NOT intentional feeding.

Chermorg3 votes

Diamond players still die in lane, go back, buy their next item, and overestimate the power spike they get.

Everyone makes mistakes. You are motioning for innocent players to be punished just because they had a bad game.

Baka Red2 votes

What do you believe the verdict from the human judges would be for the 4/24/21 KDA person ( **that would be me myself btw, so I am not naming or shaming anyone here, or can one do that to himself? :) ** ) in the match below? Was he (me) inting or not? That is almost a death per minute. If you believe that the human judges would judge "not guilty", could you describe in more detail what kind of beings these judges would be?

https://matchhistory.eune.leagueoflegends.com/en/#match-details/EUN1/1863161445/205073558?tab=stats

We lost, but I most certainly wasn't inting. Would the people doing the judging be able to tell that I wasn't? I am not made of any metal btw (ARAM has no ranked like you know), so the judges might think "This is just some unranked garbage and clearly he deserves a punishment." Automated system does know that I tend to have over 80% winrate with Karthus but would the biological judges bother to check that before they swing the ban hammer?

On the other hand since the automated system knows my winrate with Kart, it could falsely believe that "Since he usually does that well, it might actually mean he intentionally threw the game this time! Ban him!" However, since I wasn't banned/punished because of that game, I know that Riot's current system didn't consider that enough of proof to punish me. Because of these kind matches happen I can understand quite well why Riot has trouble in developing a system that can detect inting and other gameplay griefing and I am happy that they take their time with that.

I haven't seen more than a handful (or two)actual inters in my 4k+ matches (ie: just a small fraction of my total games), so imo the threat of being falsely punished would be much worse than Riot taking their time in getting the griefers punished. Anyone would hate being punished falsely - after all, people hate being punished, so being falsely punished must be worse.

Dragfin1 votes

{quoted}

The problem is not that the system cannot detect it. It is that Riot has a high bar of proof for someone intentionally trolling before they will be punished. Riot does not want to just ban everyone and their mother because they might have been intentionally feeding. It must be proven beyond reasonable doubt that the player was doing what they were doing to intentionally troll if they are to be punished.

Exactly why they should be manually reviewing a percentage of the reports they can't prove with the auto system and let the play base know they are doing it. It's called deterring and preventing and could have a real effect by possibly making the player base more aware and maybe think twice about doing things like that in game.

Chermorg1 votes

They already do this.

The problem is not they aren’t reviewing the reports. The problem is that players report anyone and everyone and expect them to get punished when there isn’t enough proof to justify a punishment.

Dragfin1 votes

{quoted}

They already do this.

The problem is not they aren’t reviewing the reports. The problem is that players report anyone and everyone and expect them to get punished when there isn’t enough proof to justify a punishment.

Well IMO if they were doing this effectively there would be a decline in offenses. If they made it a point to show the player base that there were actual results in what they were doing there would be a decline in offenses. They need to make sure that those playing the game know that they are doing this in an effective way. Simple putting this information into the EULA and having us opt in is not an effective way. It's a "cover my own ass" way to make excuses when there's an offense that someone bitches about.

I would compare this to fast food service in that everywhere you go they screw up orders more than half the time. Management isn't doing enough to fix the problem because it can't possibly be only happening to me. It's happening to many people.

Bottom line is, what RIOT is doing is not effective in even the slightest way. I don't care what the numbers they pulled out of their asses say. To even find those numbers you have to do a month of research because it's not readily available like it should be.

You talk about proof before punishment well where is the proof that they are actually doing something other than saying words that you have to search for to the end of the world to find.

Chermorg1 votes

The issue is not that there is more offenses or no decline. The issue is the community says “omg they’re inting” when the game itself doesn’t show that they’re intentionally doing it.

Dragfin1 votes

{quoted}

The issue is not that there is more offenses or no decline. The issue is the community says “omg they’re inting” when the game itself doesn’t show that they’re intentionally doing it.

A manual review would show that. This should go without saying. Does what you say mean that the review is actually happening or am I just supposed to take your word for it that it is? Show us that it's happening. Maybe show us the numbers. False report numbers included. Lets see the numbers for all of it. Be transparent and maybe players will understand better and change their ways. Frustration breeds toxicity as well. The player base is frustrated with this horrible report system and RIOT thinks it's the greatest thing since sliced bread. Companies are all about surveys. Ask better questions and get better numbers because the questions I always get in surveys from RIOT never ask the right questions.

My take is they don't care about this enough to actually do something effective so they feed us this bullshit report system and bullshit numbers they say back up their bullshit excuses.

Chermorg1 votes

Do you actually think they don’t ever review reports and their system? Would them coming and saying “we do actually review the system” appease you?

My belief is it wouldn’t - maybe it would appease you, but others would largely get mad that “well if you’re reviewing why aren’t these players who did bad in my game getting punished”. The bottom line is there is a small but vocal minority on the boards, many but nowhere near all of whom have been punished for chat offenses, who refuse to accept players can be bad at the game and that they aren’t the boss.

Dragfin1 votes

{quoted}

Do you actually think they don’t ever review reports and their system? Would them coming and saying “we do actually review the system” appease you?

My belief is it wouldn’t - maybe it would appease you, but others would largely get mad that “well if you’re reviewing why aren’t these players who did bad in my game getting punished”. The bottom line is there is a small but vocal minority on the boards, many but nowhere near all of whom have been punished for chat offenses, who refuse to accept players can be bad at the game and that they aren’t the boss.

They always want to report the positive but they never want to report the negative. That is why it's not effective.

I have already answered your question. I want to see it. I am sure many others want to see it. "Get mad", as opposed to what they are getting mad at right now? What does that even mean? Show them facts instead of talking about it and maybe they change their view and the anger will subside.

How do you know it won't be effective if you don't do it. RIOT has the numbers. You know they do. Show us and lets find out what happens. What would it hurt? It could possibly have a positive affect on the player base.

Karunamon1 votes

I'd go a step further and say the problem is that the process is entirely opaque, has little-to-no feedback, and, as a result, there's a strong sentiment that Riot isn't doing anything with the reports.

Chermorg1 votes

Heya, I know the link was kinda important to your post, but although it appeared innocent it's actually to a website that violates Riot's Terms of Use. I've removed it for this reason. If you have any questions about this please do not hesitate to contact us in the Boards Moderation Discord server.

50Shades0fTrynda1 votes

...they are still reading those from season 5

Karunamon1 votes

[deleted]

Xidphel1 votes

Who are you talking to?

Karunamon1 votes

No idea. Think I clicked reply on the wrong thing.

T4underbolt1 votes

Tribunal for inters would good. We wouldn't get that many of them due to time limitation and not enough people to work at e.g. 16k troll games. But we could eliminate the most toxic ones right away.current system bans close to no trolls. Maybe like 0.01% of reported trolls get banned.even a little more due to tribunal would make a difference. Unfortunately riot wont so it because you can't excuse an obvious flame because there chat logs proving you called someone e.g. an idiot. While troll can be excused by riot with bad game 50 times in a row so they don't have to ban potential to buyers.

remakoro1 votes

So you are saying... That people should not be punished at all?

Karunamon2 votes

No, I'm saying that automation is required to keep up with the number of games played and reported.

remakoro1 votes

Why cant we have both? Automation are just as useless in some cases as handpicking every damn case that can be resolved by checking keywords. For example inting and other somilar behavior. There are people that do it systematically for YEARS and they still go unpunished. AI proved to be quite useless there.

Karunamon1 votes

The chat filter isn't just checking keywords.. that's only for the zero tolerance stuff. It's a lot more sophisticated than that. It has to be to show the behavior we're seeing.

Inting is another matter entirely.. and I'm not convinced there is any programmatic solution for it.

remakoro1 votes

Inting is another matter entirely.. and I'm not convinced there is any programmatic solution for it.

Which fucking brings us to my first reply. I wonder why people had to downvote it. If tribunal does not work according to you and robots cant do shit here. Whats then?

More from Player Behavior

More by Karunamon