Hey Riot, I'm a statistician/data junkie

Z Statistic·Dec 13, 2016·5 votes·720 views

Could you guys publish a quarterly/semi-annual .csv that details all the player behaviors, games, builds, flags, triggers, punishments, etc.? It can be anonymous of course. I would expect it all to have non-identifiable unique case IDs so users would have the privacy they deserve.

I ask because it would definitely be a big help in dispersing the suspicion players have of false banning or your methodology or simply just increasing goodwill towards your playerbase because third party sources can now verify the statistics you guys publish or even produce some quality control.

One of the things I've learn from Open Source systems is that having a lot of people on board acting independently can provide a LOT of quality control and some fresh perspective that diverse opinions and philosophies in a dev team simply cannot match. You guys already do this to an extent with your API support for LoLking and whatnot, so maybe extending that support to the behavior system would be worth the extra work.

Cheers.

17 Comments

Sir PoodleDec 13, 20164 votes

I admire your intent to dispell myths about the behavior system, but I don't think they can be dispelled. Everyone who is sane and has actively looked into it acknowledges that the system is very fair, surprisingly accurate, and gives benefit of the doubt (favors possible false negatives over possible false positives for toxic players).

Seriously, people have these insane conspiracy theories about Riot.

Z StatisticDec 13, 20161 votes

Well, no, there's nothing to look into that isn't filtered through Riot. That's the point. Everything that we know is basically just trial and error anecdotes.

There's also the fundamental fact that alpha and beta are inversely related to each other and to n. I'm sorry. If you want both superb Power and Specificity, you need a lot of evidence. You cannot claim to have very high and balanced accuracy with low sample sizes without documentation proving it.

Sir PoodleDec 13, 20161 votes

I guess I sorta get it- not too familiar with the concepts. Still, from what I can see, even accounting for the possibility that the filter is being used to distort data, it feels (not strong evidence :P) like it's pretty good.

There are still insane conspiracy theories though.

Z StatisticDec 13, 20162 votes

With a given amount of evidence, we can either be very sensitive or very discerning. If we're not sensitive enough, we miss things, but if we're not discerning enough, we false flag stuff. The only way to hone in on a more precise judgment is by more evidence to go by.

I don't think there's a conspiracy. I think it's almost certainly something more mundane. The most obvious example is that Lyte reported how the best behaved 75% of the playerbase was responsible for 75% of the toxicity in League. He came to the conclusion that toxicity is a one-off event that can be spurned on by temporary environments; basically the phenomenon that everyone's a shitty driver when there's more people. No, it's the same proportion, but more people means a higher probability of getting a hit.

However, this could also mean that the entire idea of toxicity is more or less inaccurate and instead it's just normal elbow-bumping between people like normal with the added wrench that people are generally more direct over the internet.

The first interpretation means that there is a problem and that Riot should fix it by adding preventative measures. The second interpretation means that Riot should do nothing and instead delegate response on a case-by-case basis based on the individual and promote self-responsibility and ownership of the situation (i.e. mute dat fucker).

These are questions that can only be answered by examining raw data because I do not consider 'ggez' to be toxic or really many things Riot says is toxic.

Friend of FalcorDec 14, 20161 votes

The community determined 'ggez' was unsportsmanlike, not Riot.

Is that what this is about?

Z StatisticDec 14, 20162 votes

A trend I see is that unless someone is clearly innocent or clearly and egregiously a fucktard, people will rationalize the gray cases post hoc.

I guarantee you that I could find two cases with identical 'toxicity', ban one and pardon the other, and then present them independently to the community after telling them that this person was banned/not banned, and the community would not be consistent on the content but on the outcome. This is the fundamental problem I had with report cards even when they first came out. In some ways, it was useful for people to reform by having in put from others, but I think it also may have engineered a sort of group think.

This can be especially plausible if a bunch of people banned someone for X but the person also said 'ggez'. If the case is gray, then some will correctly identify X and some will identify 'ggez'. Regardless of which side of the free speech argument you fall on for League, here is scientific reality:

Humans are superstitious creatures
Humans are very susceptible to conformity, especially in the presence of social censure
Humans tend to rationalize outcomes as being right instead of being consistent

Mix all of these together, and it is very likely that something non-toxic would be labelled as toxic and then never explored again.

In a short form though, people don't get to claim to promote sportsman-like conduct while saying there are many forms of it. TF2 is wrought with 'ggez', obscenities, and even racial slurs. They also have more fun with it. Now, if the TF2 community is unsportsman-like, why are they having so much fun? If there are multiple equal forms of sportsman-like conduct, a) why should we stick with the one League has and b) how can Riot claim to promote sportsman-like conduct as a standard if there are multiple standards?

This is not all to say I crave to be able to say 'ggez'. No, I want to be secure that if I leave the game for an extended period, like I did earlier this year, I don't return only to find that 'glhf' is now toxic because it's apparently demeaning and sarcastic.

ModUlanopoDec 14, 20162 votes

In a short form though, people don't get to claim to promote sportsman-like conduct while saying there are many forms of it. TF2 is wrought with 'ggez', obscenities, and even racial slurs. They also have more fun with it. Now, if the TF2 community is unsportsman-like, why are they having so much fun? If there are multiple equal forms of sportsman-like conduct, a) why should we stick with the one League has and b) how can Riot claim to promote sportsman-like conduct as a standard if there are multiple standards?

Riot promotes the form of sportsmanship desired by the League community, as delineated by the way people report and their associated behaviors. More specifically, they can tell if you're actually upset by something by how it affects how you play, how often you play, when you play, etc.

As for the rest of it, people like what they like. I don't have any appreciation for professional wrestling or NASCAR and I have absolutely no desire to modify those pastimes into something more palatable to me. Riot offers a specific experience and hopes that enough people choose to entertain themselves with League to remain profitable. Let other games tend their own gardens.

Kartac MagewrighDec 14, 20161 votes

RIot claims they do promote sportsmanship, but most of the flamers get away with it, as the system requires more than one player to report them. And RIot honor system makes it even worse, as honors are totally ignored if player is reported. Also the chat is totally ignored as only the lines of offender are processed.

Riot says they are promoting sportsmanship, but their own actions and stance does the opposite. Flamer and toxic people get away with it if they get everyone to report person who they attack. The school-yard diplomacy. This is due the fact people rarely report more than one player especially when you have no chat logs of the game on the moment you do reporting. Thus they report the player they recall being toxic, and that is what those toxic people count on. They use demagogue to ensure people recall wrong.

The punishment system is thus arbitrary and de-facto random having quite likely way more false negatives (cases of trolling not getting punished) and false-positives (cases of non-toxicity punished) than true positives and negatives.

The statistics would at least give players feeling the punishment system works, and for that the x statistics task is important. Very important. But due nature of false positives and negatives, the system should also give report of number of reports not causing punishment grouped by number of reports per game. And I have strong belief reason why Riot does not do this, is the fact they know the games are more toxic they claim in PR, and the punishment system is dysfunctional, and such reports would be bad PR.

ModUlanopoDec 14, 20162 votes

[{quoted}](name=Kartac Magewrigh,realm=EUW,application-id=ZGEFLEUQ,discussion-id=qzl17EJ7,comment-id=00010000000000000000000000000000,timestamp=2016-12-14T03:17:07.324+0000)

RIot claims they do promote sportsmanship, but most of the flamers get away with it, as the system requires more than one player to report them.

Not true. A single report or support ticket is often sufficient to trigger an escalated ban.

And RIot honor system makes it even worse, as honors are totally ignored if player is reported.

Also not true. I don't know where you got this idea. The only time honor is changed - other than when people award it - is when honor is reset to zero when an account is sanctioned.

Also the chat is totally ignored as only the lines of offender are processed.

Very much not true. All of the chat is reviewed when a match is reviewed.

Riot says they are promoting sportsmanship, but their own actions and stance does the opposite. Flamer and toxic people get away with it if they get everyone to report person who they attack.

Not so. False reports lower report weight, which leads the system to ignore their reports.

The school-yard diplomacy. This is due the fact people rarely report more than one player especially when you have no chat logs of the game on the moment you do reporting.

I don't have any knowledge of the statistics on this. Neither do you.

Thus they report the player they recall being toxic, and that is what those toxic people count on. They use demagogue to ensure people recall wrong.

So... Your argument is that truly toxic people try to deflect by asking others to report? If that's the case, you should know that report begging is very high on the list of things people report.

The punishment system is thus arbitrary and de-facto random having quite likely way more false negatives (cases of trolling not getting punished) and false-positives (cases of non-toxicity punished) than true positives and negatives.

That's quite the leap. My question to you is this: if the system was that flawed, why would Riot continue to use it?

The statistics would at least give players feeling the punishment system works, and for that the x statistics task is important. Very important. But due nature of false positives and negatives, the system should also give report of number of reports not causing punishment grouped by number of reports per game. And I have strong belief reason why Riot does not do this, is the fact they know the games are more toxic they claim in PR, and the punishment system is dysfunctional, and such reports would be bad PR.

I think there are two primary reasons they don't release raw statistics.

First, it would provide them zero benefit. There will always be people with conspiracy theories about how flawed and random the punishment system is. Not only would no amount of data appease them, they would twist it around for their own propaganda purposes. Lord knows, every time Riot fesses up to making a mistake, there's a certain wedge of the community that always pipes up with "See, this means they make mistakes all the time!!!"

Second, it would provide no benefit to the average player. That person doesn't need a bunch of statistics to tell them they're having fun with League.

Kartac MagewrighDec 14, 20161 votes

{quoted}

Not true. A single report or support ticket is often sufficient to trigger an escalated ban.

Often is not good enough, and in my experience of reporting toxicity for good reason for seasons, it is not often, but sometimes. Due my play of support zyra with relic shield, I get flamed on EUNE every single game. I haven't got for months any indication that my reports have got single player banned. And I play lots. Thus it seems single report is not enough, but you should do the report in hope someone else in the game does this.

For me, single report has only impact if it always triggers investigation.

Very much not true. All of the chat is reviewed when a match is reviewed.

Honors has no effect on system. You might get ribbon, but number of games you are not reported or number of games you are honored has no effect for punishment. Just number of reports. And this has been confirmed by RIot support. Only number of reports matter, thus honors are ignored by the system.

Not so. False reports lower report weight, which leads the system to ignore their reports.

And RIot has given us no indication what is false report. If false report is report which does not cause punishment, and single player reporting the team mates together bullying him are ignored, his score thus automatically gets lower. If I get constant flaming, cancer, and kill yourself, and my score of reprots still tank, your statement that single player reports matter is true. It matters by removng your ability to report others.

So... Your argument is that truly toxic people try to deflect by asking others to report? If that's the case, you should know that report begging is very high on the list of things people report.

And as only I do report it, it does not matter. The abusers know that, and use it. Easiest way to get away with flaming is getting people report your victim. IT does not matter how high it is in ladder, if it is not investigated. You know, killing someone is high felony, but if nobody is doing murder investigation, and victim is assumed to be missing person, it does not matter.

That's quite the leap. My question to you is this: if the system was that flawed, why would Riot continue to use it?

Yes, they would, as they deny it is flawed. Riot has continued to use flawed MMR based matchmaking for 7 seasons now. IT was proven to them on season 2 it does not work, which they reacted by hiding MMR and creating League system as PR stunt. Thus we have very good case show how RIot would react.

First, it would provide them zero benefit. There will always be people with conspiracy theories about how flawed and random the punishment system is. Not only would no amount of data appease them, they would twist it around for their own propaganda purposes. Lord knows, every time Riot fesses up to making a mistake, there's a certain wedge of the community that always pipes up with "See, this means they make mistakes all the time!!!"

You are wrong. The common mentality is that Tribunal system is joke. Rules violators laugh at it even in game talk. There is benefit for Riot, as it would prove them to be honest and really care for sportsmanship and summoner's code. Instead they make decisions eroding this like Support Singed being punished even if Riot states in Summoner's Code spells and champion is not reason for report. It is now.

If you did not get it, the good and mature players have been leaving the game. LOL has very bad reputation in MMO players mind due toxicity of community. That toxicity is main reason I tell others not to try LOL even if I do love the game mechanics, and well implemented champions. The number of op champions is quite low, and item descriptions, tool tips and understandable stat names are big thing.

Second, it would provide no benefit to the average player. That person doesn't need a bunch of statistics to tell them they're having fun with League.

Yes, it would. If average player would see the number of reports and punishments, they would get feeling RIot is doing something. Instead you get feeling Riot is doing all they can to hide all details. It makes you feel they do not want you to see them. And as company, only reasonable outcome is that they know it would be bad PR as they are NOT doing their job to fight toxicity.

Z StatisticDec 14, 20161 votes

And now we're going around in circles

Me: I want to be able look at Riot's raw data Opponents: why? Me: So I can verify Riot's conclusions and methodology Opponents: Why don't you trust Riot is responding to player desires? Me: Because TF2 and League are both set in the US with very similar demographics yet have vastly different culture of behavior and punishment Opponents: Are you doing this just so you can say 'ggez'? Me: No, I'm just skeptical because the Tribunal was based on self-selected participants and I can think of a few examples from Riot themselves that are not clear cut one way or the other Opponents: But Riot has all the metrics and has solid methodology that accurately represents reality in the League community Me: Ok, then let me see the raw data Opponents: But why do you need to see it? Me: [slayer-jinx-unamused]

Ultimately, banning players is a form of social engineering. While I think standards are in order and perhaps banning is reasonable for very egregious cases, the entire point of the post you're replying to is that Riot has the power to engineer the community according to superstitious behaviors from the community.

The most telling experience I had was when I asked my friend about her reporting habits. She said she reports rude or obnoxious people. "Ok, I can get on board with that, but I don't see how what the person said last game was offensive." "It wasn't offensive." "Then how was it rude?" "Because it is rude." "But if you're not offended by it, how is it rude?" "Because it's offensive to other people." "But if they don't care, and you don't care, why should you report them?" "Look, it's just rude."

I personally don't give a damn what people say in chat so I hardly ever report. I don't think I've reported anyone this season. And while I fully accept that people will report things I find frivolous, I have to wonder if most people are reporting things not because they find it rude or offensive themselves but because they think other people find it rude and offensive. And if we've come to that, the current system and even the fundamental Tribunal it was built upon suddenly comes crashing down because now the voters are not independent inputs but hugely inflated by massive amounts of multicollinearity (bad stuff when making predictions).

ModUlanopoDec 14, 20162 votes

If you're replying to me, that's an awful lot of straw and I'm going to spend exactly zero time rebutting arguments I never made.

For the record, my argument was that it's quite possible for different online environments to have widely varying social contracts and systems for dealing with people who break that contract. It's also possible for different companies to have divergent business models for dealing with toxicity. A social contract or punishment system in one environment does not invalidate the same in other environments (i.e. the NHL and NFL have very different approaches to fighting during the game).

Look, this basically comes down to the fact that you don't believe that crowd sourcing works and I do. "GGEZ" being punished isn't some "superstitious" behavior in my book, nor is anything else the system deems punishable. That's because - outside of some specific, clearly understood behaviors - punishment is based on its impact on the community. Riot wants people to feel good about League, so they target the things that interfere with that.

I also feel that there's not enough sociology or anthropology in your statistics. Being offended is a cultural response to a stimulus. Your friend translated the offense taken by others because that's how civilization works. A culture either learns to reduce friction or eventually you get fire. It's a form of erasure to tell people their feelings and opinions are wrong. It's also a form of privilege to not have to worry about people being insensitive or offensive.

Z StatisticDec 14, 20162 votes

I wasn't responding to you specifically. I was laying out what this whole comment chain has come to; you're not contributing anything new. I want a .csv to verify Riot's claims.
My entire point is that crowd sourcing is a huge success and that restriction of raw data is an impediment to it. How you concluded that I believe crowd sourcing is a poor idea is beyond me.
I don't include non-sciences in statistical arguments. If you want to be based in hard science, then you argue human behavior through behavioral psychology from a bottom up approach, which is what I have done. If you want to use the unit of discussion as 'arbitrarily large groups' such as a playerbase, then you have left the realm of science and now have entered into the realm of philosophy

If you want to talk philosophy of behavior and human interaction, I'm cool with that. It just so happens that I subscribe to an individualist philosophy since it has a very low failure rate for civilization. I think players should be focused on their own behavior and their personal, case-by-case, direct interactions. Generally speaking, if you have a bunch of independent individuals saying that a person is being rude to them in a given interaction with that person, that person is probably being rude.

If however these individuals attempt to judge rudeness or offense based on how they think other people will take it, the accuracy gets destroyed for two reasons. First, empathy is just projection. That's not just me saying it. That's reproducible fact. Second, ignoring your own reaction of being offended or lack thereof in lieu of assuming someone else is offended or could be offended if they were in this game just adds one or two layers of confounds to the decision-making.

There are reasonable expectations of how people will act or take certain actions, but strict adherence to those expectations is not going to work. Friction does breed fire. What's wrong with fire? Fire can burn you, but it can also keep you warm. We're not going to go to war over League. We're not going to punch each other in the face (though that does work in NHL). Nothing bad happens because of League. Losing a game or even losing a promo series means nothing. It shouldn't mean much. People do have a right to their emotions, but that doesn't mean every emotional reaction is reasonable or warranted. Whatever actions someone takes outside of League is their own choice

My lack of worrying is not a privilege; I had to work hard to overcome my OCPD and social anxiety. It took months of therapy and work to get this far when only 18 months ago I couldn't order food at a drive-in because I was afraid of messing up the order.

Kartac MagewrighDec 14, 20161 votes

Very good post. Unfortunately Riot fanboys never use facts nor never answer to your questions, but follow dogmatic fanatic faith in Riot. If Riot is not doing something, we should not discuss it. I think this is part of post-Nixon self-censorship of USA. They try not to talk of things which is not beneficiary to rulers.

Your points why to get data were really good. And I bet it hit nerve of those who preach in forums how single report matters even if they in next post give us logical reasoning why it does not. But this nation voted Trump as president. THey believed member of elite would fight elite.... I rest my case of American "intelligence".

Z StatisticDec 14, 20161 votes

There is some merit to their point of view. While I don't think people should be outright banned for using racial slurs, verbally abusing people, or stream of consciousness typing because a mute button exists (and this demonization of the 'band-aid fix' by Riot is just ludicrous), the fact of the matter is that calling people fucking cykes or telling them 'kys' is honestly disgusting, obscene, and any reasonable person would find it offensive in most circumstances.

You have the right to be a jerk and an obligation to be kind.

Just because standards are subjective and the line varies from person to person doesn't mean there is no general standard. There is no line in which red definitely becomes orange but to then say that red and orange therefore do not exist is laughable. People do not get banned for being somewhat of a jerk or jerkish. They are generally banned for being an insufferable prick. While the incredibly slippery definition of 'being a jerk' has a chilling effect on player interactions in my experience, it's hard to argue against the simply definition because, as I said, Riot really only bans people who are clearly past the breaking point, meaning that playing safe with behavior is honestly unwarranted.

I actually suspect one of two things is the case:

The players on the whole are Hipster %%% Beta Fucks (HFBF tm) since 90% of them are in college
Players have been superstitiously taught that certain actions are 'toxic'

If it's the former, then you change the community by explaining to people very reasonably that someone calling you any number of slurs or even exploding about your skills or rather lack thereof is not only not the end of the world but shouldn't even register on your radar. If people can depersonalize others so that they can be more aggressive or assertive online, why can't people depersonalize to shrug off text? You don't get offended if an NPC calls your character names, do you? What does it matter if it's text generated by a computer or a human?

If it's the latter, then it's just a matter of pulling back the curtain statistically speaking and explaining to the community that they and Riot have let this ideological basis for toxicity gone on far too long and it's having a negative impact on people and their behavior. The most ironic thing in the past few years for me is that the largest source of my anxiety and unhappiness was League while the strongest form of exposure therapy was...League. Before, I was anxious because I was afraid of offending and getting reported. Now, I have a charitable view of people's standards and don't worry about it. Granted, I haven't played ranked in 8 months, but I am significantly better at dealing with negativity in ARAM than I was two years ago.