Explaining the Echo Fox vs. Clutch Gaming Ruling

Riot·Jun 21, 2018·0 votes·11,936 views

Chris “Riot Commish” Greeley here. I’m the Commissioner of the NA LCS. This past weekend we used a rule, the Awarded Game Victory rule, for the first time. In this Quick Notes, I want to talk about how the decision was made and continue to provide greater transparency about how the NA LCS functions behind the scenes.

The Circumstances

In the Week 1 match between Clutch Gaming and Echo Fox on Sunday, June 17, the game server hosting the match crashed at 32:26 into the game. If you were watching the stream, the game simply froze on screen, and the players weren’t able to take any further action.

Our Live Operations team repeatedly attempted to Chronobreak the game during the stoppage of play, but due to the server crash, the Chronobreak tool was not able to restore the game. After approximately 20-25 minutes of work to restore the game to its current state, we acknowledged that Chronobreak was not going to solve the issue.

Game Breaking Bugs & Awarding Game Victories

The advent of Chronobreak has allowed us to avoid a number of situations in which a game impacting bug may otherwise have required a complete remake. Despite that safety net, we still know that there are going to be rare situations where Chronobreak is not available (for example, in the event of a game server crash or where a game breaking bug is discovered that simply going back in time will not correct or avoid).

In these cases where a critical bug is not the fault of either team (i.e., not a situation where a player knows of a bug that they, in our opinion, intentionally trigger to cause some kind of issue), and Chronobreak is not available, the inclination should always be in favor of a remake, except in situations where it would be more fair to award a winner based on the current game state than playing an entirely new game. In those cases, punishing the team that has the game in hand by remaking has to be carefully considered against depriving the other team of a chance to continue playing through an awarded game victory.

During the biannual review of the rules, we spent a lot of time with our statistics team pouring over game data looking for some set of criteria that were objectively indicative of victory. What we learned is that they don’t exist. Short of one team attacking the open Nexus with the enemy team dead, it is exceedingly difficult to articulate objective standards around when victory should be awarded in the case of some kind of terminal failure. Even the guidepost examples provided in the rulebook are sensitive to changes in the game, objectives, gold and experience values, and the meta.

Ultimately, the awarded game victory rule vests discretion in the Head Referee monitoring the game and entrusts them to determine whether, in extreme situations, an awarded game victory or remake are the most appropriate outcomes.

The Decision Making Process

Our first priority in any situation where there is a bug that affects gameplay is to quickly ascertain what happened, whether we need to take some action (rather than just unpause and continue), and what options we have if some action is required (both practically and philosophically).

In this case, when the Head Referee’s view of the game froze, he asked the stage referees what they saw on the pro screens on stage. He quickly learned that all ten players had frozen as well, and each was showing a window that the game was attempting to reconnect. The broadcast team and the observers were asking about a pause, but the referee team concluded that no one had actually paused the game. This led our Live Operations team to believe that there was an issue with the game server.

The Live Operations team engaged the Chronobreak tool to try to recreate the game and get the players back into game as fast as possible. At the same time, the Head Referee (who was able to rewind the game through the observation tool) began to look for an appropriate point to Chronobreak the game to. However, after several minutes of working with the Chronobreak tool, the Live Operations team reported that they were unable to Chronobreak the game through standard methods.

The Live Operations team continued to work on the Chronobreak tool in an attempt to restore the game. In parallel, the Head Referee began planning for a situation in which he would have to choose between issuing an instruction to remake the game from scratch (including a new champion select phase) or awarding the game victory to Echo Fox. As part of our usual backstage procedures, the Head Referee pulled up the rulebook and reviewed the relevant portions to make sure he had all of the information at hand necessary to make a decision.

The awarded game victory rule turns on a determination that the team losing the game “cannot avoid defeat to a degree of reasonable certainty.” Put simply – what is the likelihood that the losing team can come back and win?

At the time of the crash, Echo Fox was up 7 turrets to 0. There was 1 inhibitor down on Clutch Gaming’s side, a 4 dragon lead and a 10k gold lead for Echo Fox. After reviewing the current state of the game (including towers, objectives, gold differential and objective timers), the Head Referee determined that in the event that the game could not be recovered through Chronobreak, that the victory would be awarded to Echo Fox because it was sufficiently highly unlikely, in his opinion, that Clutch Gaming could avoid defeat in the game. That doesn’t mean it was impossible – there are always situations in League where a team that is down by a large margin mounts a comeback, but that in this situation, it was highly unlikely.

After the Head Referee made his decision, we gathered the management from Echo Fox and Clutch Gaming backstage and informed them of the decision. Subsequently, we asked the players to take their gear, leave the stage, informed them of the decision backstage, and announced the decision on broadcast. At roughly the same time, we learned from our gameplay team that the likely cause of the crash was a bug which can manifest when a player selects Taliyah with smite in a game and disabled Taliyah for the remaining game of the day.

The Aftermath

We have had several conversations with Clutch Gaming organization at all levels explaining our decision and rationale. Separately, we have reached out to all the NA LCS teams asking for feedback on the rule, its application in this game, and application to situations in the future. This is the first time we have used the awarded game victory rule, and we want to ensure that in the future the League and teams are aligned on the use of the rule in the rare circumstances where it may be needed.

While we are confident in the decision based on the information we had on hand at the time it was made, we have subsequently taken a look through patch 8.11 statistics from the NA LCS and other leagues to get some sense of whether or not our conclusion on-the-spot makes sense. Across 131 games in NA LCS, EU LCS, NA Academy, LCK, CBLoL, CLS, OPL, and TCL:

No team has lost when they had a gold lead as large as Echo Fox’ at thirty minutes;
No team has lost when they had a 7 tower advantage;
In 74 games where a team has had an advantage of at least 10,079 gold (Echo Fox's exact lead at the time of the crash), the team with the gold advantage has won every single time.

Obviously, none of these statistics are dispositive and comebacks can happen, but the point is that such comebacks are few and far between, and when weighing between two unpalatable options (complete remake and awarded game victory), the Head Referee was and remains reasonably certain that Echo Fox would have been able to close this game out had the crash not rendered the game unplayable.

We have confirmed that the Clutch Gaming and Echo Fox game was derailed by a bug that can be triggered when Taliyah (or Karthus) spams abilities in fog of war. The bug drives CPU usage towards 100%, at which point the game server terminates the game. Due to the nature of this particular bug, even if we were able to Chronobreak the game, it is highly likely that the game would have continued to trigger this bug, rendering it unplayable. This bug was diagnosed earlier this week and a micropatch was applied to prevent it from occurring in the future.

Quick Notes is a NA esports series similar to Meddler’s where we’ll deep dive into topics that the community wants to know more about, discuss work in progress, and share our decision-making process.

Chris “Riot Commish” Greeley is the Commissioner of the NA LCS and ran League Operations for MSI and Worlds in 2017. He is a recovering lawyer from New York City who can’t write a rulebook in less than 50 pages. He channeled years of raiding on his Resto Druid into life in the bottom lane dropping wards and shooting glitterlances. You can find him on Instagram, Twitter and Reddit as @IAmGrza.

27 Comments

SulfuricJun 22, 20188 votes

Thanks for laying out the decision with some data to back it up. Tough decisions rarely feel correct but they have to be made one way or the other.

Ahri Carries AllJun 22, 20185 votes

I saw a high elo YouTuber attempting to recreate the bug that crashed the game server and over numerous tries, he only got it to happen once. This is the first time I've seen a bug like this and I think with it being hard to trigger, it isn't on Riot for not seeing this coming. No one could. As for the ruling on the game and the existence of the rule which only goes into effect if the game cannot be Chronobreak'd... It's remarkable that this rule Riot put in place years ago hasn't seen use until this match and I think that when you look at how the game was panning out, CG really didn't have much of a chance. I'm not saying FOX was definitely going to win, but in a majority of games, with this kind of lead, the team in the lead is slated to win 99% of the time. Good ruling in my opinion but a shame that it came to this.

kill3rlanceJun 22, 20185 votes

FORGET RIOT, THIS COMPANY SUCKS. There's always a small chance, there has to be some sort of alternative or mending of the situation.

"In 74 games where a team has had an advantage of at least 10,079 gold (Echo Fox's exact lead at the time of the crash), the team with the gold advantage has won every single time."

Oh.

ChompyWulfJun 22, 20183 votes

I'm just here to laugh because with all the massive profits, with game breaking bugs forcing restarts in the past, and a program supposedly designed to recover games that crashed, it happened. After 7 seasons and new Esports games/genres on the rise, League looked bad. Really bad. It's embarrassing and Riot should not only be explaining the decision, but also issuing an apology to both teams for an unsatisfactory ending, offer some small reimbursement to fans who bought tickets to watch live, and given the clear problem this presents in Best of 1 play, drop it and go back to Best of 3. At least in a series there is far less chance of a single bug deciding which team has the W at the end. Maybe revise your rules so in an awarded game scenario it instantly puts the teams into a Best of 3.

Nog Nog NogJun 22, 20189 votes

Going this long without instigating the rule is actually pretty impressive and don't pretend crazy things don't happen in other sports. Basketball has had riots in the stands. Superb Owl 47 was delayed by over half an hour due to a power outage. Baseball, though, is the greatest culprit and can call a game official after the top of the 5th! The odds of a team coming back in four innings is far greater than the chances CG had (ie: given statistical history... 0).

I appreciate Riot being open and frankly their refs have been far more forthcoming as to the reasoning behind their decision than the officials of the other sports I just listed.

ChompyWulfJun 22, 20182 votes

I don't see how fights in the viewer stands of a ball game relates at all. Your example of the Super Bowl power loss is close, but given that was also a city infrastructure problem, not quite the same. Your best counter was baseball, but a called game is uncommon and rain delays or continuations are more common. Furthermore, weather is beyond control and not an issue for an Esports problem like what happened where coding issues and server stability are all faults of Riot.

ItsFreakinHarryJun 24, 20182 votes

Server issues are bound to happen, bugs are unavoidable in literally any game. In this case, Karthus with smite and Taliyah with smite was uncommon until just recently, which is why the bug is just now being discovered. You cannot expect a game dev, even a very large company such as Riot, to be able to fix every single issue that the game has. As long as the game is constantly updated, bugs will constantly arise, and unforeseen events such as this will happen. Just because 1 game in over thousands across multiple years had the server crash and results in an AGV, that does not mean the whole rules have to be changed to accommodate this rare event. If this kind of problem happened every split then yes it would be quite bad, but that is exactly why Chronobreak exists, and besides Bo3 has its own problems and, if anything, would make it even more likely that these issues arise because more games are played. The way Riot handled this was the best they could: They made the fairest decision, was completely transparent with their process and reasoning, and found the source of the bug, then disabled the source to avoid the problem again until they could gather more information.

tl;dr, Riot cannot foresee every bug that happens in the game, do not get all pissy because some things in coding and computer engineering are unavoidable. Riot did what they could, and it is not fair to them to say that Riot is completely at fault for what happened.

Also the power can still get knocked out in the battle arena where they play, not just a problem for traditional sports (although I am yet to see any instances where that has actually happened.)

KloiperJun 22, 20184 votes

I'm sure that Riot did issue an apology to both teams, as well as give some consolation to fans who had their experience stopped short. This has no relevance, however, to best of 1 or 3 - this could have happened in game 3 of 3 and it would have been just as unsatisfactory for all parties involved.

You're right, it's embarrassing, but it happens incredibly rarely and Riot handled the issue immediately and professionally. It also happens in other esports as well as traditional sports, albeit less so in traditional sports because they've been around for up to hundreds of years and have had time to work out kinks in the game (bats breaking, balls deflating, etc).

ChompyWulfJun 22, 20181 votes

Even if the bug happened in a Game 3 of a Bo3, each team already had two other games in the match played. It would be unsatisfying, but nowhere as sour an end as a called single game.

Based YukiJun 26, 20181 votes

You're absolutely right. This is not a satisfactory defeat for Clutch, and not a satisfactory victory for Echo Fox, and especially unsatisfactory for fans of both teams watching. Serious letdown that seems to have been handled in a disappointing way. Hopefully this experience will lead to a better reaction/decision in the event that it happens again. Props to Riot for going this long without such an issue

ShadowZPhoenixJun 22, 20183 votes

I remember reading a post a couple of years ago about LCS rulings, and none of which had been applied because they allow games to play out 99.99% of the time. These Rulings include when it is a referees decision to end the match if one of these conditions are met, meaning that a team has no chance to recover. a 10k gold lead, 5 tower or double inhibitor advantage when 1-2 towers are taken on the other team are among those that fulfill the ruling where a Ref can award a win to a team.

These are RULES, stating that when a team has enough of a gold lead Refs say game is over, you win. The other team has a forced surrender. Has that happened once in the history of league? I don't think so, at least not in LCS competitive play, yet there is that rule. Cut Riot a little slack, yea it is a terrible thing to have happen, but things like this do happen

The Bad TouchJun 21, 20181 votes

No team has lost when they had a gold lead as large as Echo Fox’ at thirty minutes; No team has lost when they had a 7 tower advantage; In 74 games where a team has had an advantage of at least 10,079 gold (Echo Fox's exact lead at the time of the crash), the team with the gold advantage has won every single time.

So can we just have games fucking autoend at 30 minutes if one team is 10k ahead?

Idiots that wont FF get old.

ItsFreakinHarryJun 22, 20185 votes

Fair point, but remember this is just on patch 8.11 in the pro scene, not nearly large enough of a sample size to say for certain that the game should just auto-end when one team has a massive advantage over another.

Comebacks are still possible, it's just in this case they had to award the victory to FOX because the game crashed, not because teams wanted to ff.

The Bad TouchJul 1, 20181 votes

{quoted}

Fair point, but remember this is just on patch 8.11 in the pro scene, not nearly large enough of a sample size to say for certain that the game should just auto-end when one team has a massive advantage over another.

Comebacks are still possible, it's just in this case they had to award the victory to FOX because the game crashed, not because teams wanted to ff.

Nah it's the same on every recent patch and in every elo.

First inhib alone is a 92% win chance. Games with that much of a gold lead AND first inhib is more like a 99% chance. I've literally seen ONE comeback game this season. That game was the fed ADC disconnecting at elder. If nothing else this would have prevented the team that should have won from losing just because someones net crapped out.

UberChayChayJun 22, 20183 votes

Apparently you can't even read your own post. It isn't just the 10k gold advantage (even if an argument can be made that that could validate the ruling by itself), it's the 10k gold advantage PLUS the 7-0 turret lead PLUS the 4-0 dragon lead PLUS the fact that Clutch lost an INHIBITOR (if you're gunna list things, please list all of them or none at all, like really). Every single one of those things are a huge advantage even in just soloQ, let alone a level of play where everyone is communicating and working together to put a plan into motion that they've run through with their comps thousands of times, and even with all those things working against Clutch, Riot still contacted every team in this region to weigh in on this ruling. Also, you know as well as I do in soloQ that people refuse to ff in even standard ranked games because they refuse to lose LP until they're forced to (or they're trolling), but in this situation, they're paid to do this. Like a yearly, 75k minimum salary. I don't care who you are, you're playing that game out, because you don't want to have your team's roster shaken like a goddamn etch a sketch because the owner thinks that you aren't motivated to play. We've seen it in C9 already, and those are 3 tried and tested players, who've played together and succeeded more than anyone else on the World's stage (when competing for NA).

Idiots that don't actually think about what they're talking about are annoying.

The Bad TouchJul 1, 20181 votes

{quoted}

Apparently you can't even read your own post. It isn't just the 10k gold advantage (even if an argument can be made that that could validate the ruling by itself), it's the 10k gold advantage PLUS the 7-0 turret lead PLUS the 4-0 dragon lead PLUS the fact that Clutch lost an INHIBITOR (if you're gunna list things, please list all of them or none at all, like really). Every single one of those things are a huge advantage even in just soloQ, let alone a level of play where everyone is communicating and working together to put a plan into motion that they've run through with their comps thousands of times, and even with all those things working against Clutch, Riot still contacted every team in this region to weigh in on this ruling. Also, you know as well as I do in soloQ that people refuse to ff in even standard ranked games because they refuse to lose LP until they're forced to (or they're trolling), but in this situation, they're paid to do this. Like a yearly, 75k minimum salary. I don't care who you are, you're playing that game out, because you don't want to have your team's roster shaken like a goddamn etch a sketch because the owner thinks that you aren't motivated to play. We've seen it in C9 already, and those are 3 tried and tested players, who've played together and succeeded more than anyone else on the World's stage (when competing for NA).

Idiots that don't actually think about what they're talking about are annoying.

Idiots like you don't understand where that 10k gold advantage came from. LOL

Oh look the 7 towers blah blah blah added up to a 10k gold advantage. Well no shit. That's every single game under 50 minutes long where one team is up 10k gold.

Hun the groceries were $186.34 Should I list every single item we bought?

/facepalm.

RHITJun 22, 20181 votes

I understand the game server crashed, but can you explain (here or in a technical blog) why the game data isn't running on one server and being copied to another during the match? This way if the server the game is being played on spontaneously combusts (or goes does to something more reasonable like harddrive malfunctioning), you could switch to the 2nd server and chronobreak to a reasonable time to resume play. I know in this case there was a high likelyhood of the problem reoccuring so it wouldn't have helped, but there is the chance of this being needed in the future.

ItsFreakinHarryJun 23, 20181 votes

Thats exactly what Chronobreak does, and they stated how they could not Chronobreak it. Here is the article with more details about Chronobreak: https://www.lolesports.com/en_US/articles/deconstructing-chronobreak

AbysmalDrunkJun 22, 20181 votes

You still broke your own rules. Read your rule book again, no conditions were met. Rito fails again.

AbysmalDrunkJun 22, 20181 votes

Also, don't compare previous LCS statistics in a decision like this. This meta is ridiculously easy to flip flop. That's the worst argument you could use.

BeholdGreatnessJun 22, 20183 votes

Thanks for your advice on the Meta, Silver 4 player AbysmalDrunk.

DrCyanideJun 22, 20185 votes

The match was played on 8.11, they compared it to other games on 8.11. Considering they had a sample size of 131 games (74 with leads like the one in the game in question) to compare to, I think it's safe to say in this instance the ruling is fair.

ItsFreakinHarryJun 22, 20188 votes

Quotation from 13.1 on Awarded Game Victory: "League Officials may, but are not required to, use any or all of the following criteria in the determination that one team cannot avoid defeat to a degree of reasonable certainty"

They do not have to have those conditions met, they are just guidelines.

Source: http://esports-assets.s3.amazonaws.com/production/files/rules/2018-LCS-Rule-Set-v18.8.pdf

Sweet TaroJun 22, 20182 votes

They did not break any rules, the conditions served as guidelines. A great point that I heard on The Dive was that different metas and patches affect the metrics of how you measure these conditions (towers, inhibs, gold leads). One could argue how a 5k gold lead in this patch could mean something way different from a 5k gold lead from last season.

BeholdGreatnessJun 22, 20182 votes

Attempts to quote the rulebook to fire back on Riot, but didn't even read the rulebook. The reddit special

SquigglyLineJun 26, 20181 votes

No team has lost when they had a gold lead as large as Echo Fox’ at thirty minutes;

Cloud9: Hold my beer

.#NotMyC9

UkhoziJun 30, 20181 votes

I would like to point out something unrelated to most of the discussion (that part about working around bugs and the format of the LCS rules) but about what to do if there must be a decision to remake or use AGV.

Suppose hypothetically, that Riot did have some way to precisely determine the odds of a team making a comeback. What should be the thresh-hold for deciding to remake or AGV? It should be the odds that team had of winning in the first place (i.e. from start of draft phase). e.g. If Team A has a 20% chance to beat Team B, but their chances have fallen to 10% when the problem occurs, AGV should be used because the margin is in favor of Team A losing.

In a situation where you cannot use Chronobreak to be fair to the actual probability, use whatever option brings you closest.