Deconstructing Chronobreak

Riot·Feb 16, 2017·3 votes·38,310 views

As you might have seen on the NA LCS, a critical bug during the FlyQuest and Cloud9 series (spoilers) on Sunday, February 12, caused a long pause. Previously, we would have offered a remake since the bug was verifiable and had a critical impact on the game. However, we were able to use our new tool, the Deterministic Disaster Recovery Tool aka “Chronobreak”, to recover the game to a point just before the bug occured.

Since the series, we’ve seen some fan questions about Chronobreak and how it’ll impact competitive play. In this post, we’ll be sharing more information about Chronobreak, how it works, how it’ll be used, and our plans around global rollout.

What is Chronobreak?

Since the start of Spring Split, we’ve had Chronobreak running in the background at the NA LCS. As the name suggests, this new tool allows us to recover from disasters such as bugs during live events to maintain competitive integrity.

How does it work?

Chronobreak saves a Server Replay of a game while it’s being played (a recording of all the inputs received by the server). If needed, the Server Replay can be used to create a new game and play it up until any point in the timeline. Players and Spectators can then connect to the new game and resume play.

Our engineering team will be releasing more information about Chronobreak and Determinism for those interested in a more detailed, technical look at how this tool was engineered and concepted in the coming weeks.

When will it be used?

Right now, we’ll only use Chronobreak in a situation that would warrant remaking the game. NA LCS officials evaluate bugs on a case by case basis, making sure that they are both:

Verifiable: The bug is proven to be a bug and not caused by player error.
Critical: The bug had a significant gameplay impacting effect. The effect of the bug would have to greatly influence the competitive game state in a negative way for at least one team. If a bug occurs, but the outcome is not altered by the bug, then it will not be considered critical. For example, a player dying from a bug when they would not have died otherwise is critical.

Once NA LCS officials determine the bug meets the above criteria and a remake is accepted by the impacted team, NA LCS officials will attempt to use Chronobreak to recover the game prior to when the bug occured. If Chronobreak doesn’t work, a full remake will be implemented instead.

While Chronobreak addresses the types of bugs that cause more than 90% of remakes in competitive play, it does not address all of them such as the Aurelion Sol bug at Worlds 2016, which would have recurred even in a recovered game.

Did the NA LCS teams know?

Yes, NA LCS teams were informed about our new tool at the start of Spring Split.

What’s next?

Our goal has been to get Chronobreak on NA LCS so that we could trial it out and test it for global rollout. Testing on NA allows us to iterate quickly on the tool, make it faster, and test it fully before we introduce a global rollout. We’re working on v2 now, which will include getting Chronobreak to every region (including configuring tournament realms, training local operations on how to use the tool etc). We hope to roll it out to other regions starting in the Summer Split.

Questions? We’ll be hanging out in the comments below, so let us know.

Emil “RiotEmil” Jannesson is a Live Producer for Esports Operations, a team that has been running the technical framework of tournament realms and events across the globe since 2013. RiotEmil works on a range of events, from NA LCS to the World Championship, and his team is involved in tournament realm support and esports bug triaging. When he's not spending time with his dogs Kayle and Nasus, you can frequently find him ganking unsuspecting foes in the latest PvP games. If you have any questions or want to challenge him to a duel in any competitive game, hit him up on Twitter at @RiotEmil.

45 Comments

TurquoiseYoshiFeb 16, 20178 votes

Question: Why did you not mention Chronobreak until a use of it occurred? Was it because you wanted to look really cool when it was shown off?

Because it is really cool.

RiotRiot EmilFeb 16, 20177 votes

As it's still in "trial" or testing period, there was always the chance of a recovery not being possible, so we felt that it would be better if we revealed it after a successful recovery and then communicated from there.

TurquoiseYoshiFeb 16, 20175 votes

But were you secretly really happy that it showed up in Game 3 of the biggest set of the split so far?

RiotIzeoFeb 16, 201714 votes

Secretly having a heart attack imagining it failing and having to remake anyway. Or worse, it'd initially work but we'd end up remaking later due to bug recurring.

Woo buddy that pause immediately after the resume, o lawdeh my heart

RiotRiot EmilFeb 16, 20173 votes

We're obviously never happy about running into issues/bugs, but I was extremely happy that we were able to use the tool to help save the game from needing a full remake (especially in such an awesome series).

SpideraxeFeb 16, 20174 votes

Do you think Chronobreak could've been used to salvage that S5 worlds game when Gragas got disabled

RiotRiot EmilFeb 16, 20173 votes

Yes, as that bug was extremely rare.

If something like this happens now, we will look at the likelihood of the bug happening again and make a decision on whether or not it was something that would immediately happen again, and then do a recovery or a remake (with a disable) based on that decision.

SpideraxeFeb 16, 20171 votes

What about the A.Sol disable of S6 worlds, did you guys have the tech then

RiotRiot EmilFeb 16, 20173 votes

Unfortunately, due to the way that bug functioned, we would not have been able to recover that game as the bug would have happened again; you can check out the explanation of the Aurelion Sol bug here.

LankPantsFeb 16, 20172 votes

Will this mean that the second option will be less common into the future since you know the bug is rare and even if you do get very unlucky and it happens again you can remake from within the game?

Will it take a bigger, more common bug for you to consider disabling a champion is what I'm asking I guess.

RiotAlwaysBeClosingFeb 16, 20173 votes

Disabling champions is fairly bug specific. If it is something persistent that we are unable to control for then that champion would still remain disabled (like if we did not determine that the accounts were causing the Aurelion Sol bug). But, the hope is that rare or one off bugs would now be controlled for by the Chronobreak tool and so disabling in those circumstances would be lower.

AMYS GRAVEFeb 16, 20171 votes

Was it attempted to be used but failed at any point before that we, the community, weren't aware of? If so in what way did it fail?

Cool stuff though.

RiotRiot EmilFeb 17, 20173 votes

We've never attempted a recovery in an actual competitive match before, but we did quite a bit of playtesting with Riot volunteers leading up to the start of the spring split and did not have any failures.

AMYS GRAVEFeb 17, 20171 votes

impressive work.

DavìdFeb 16, 20173 votes

"Our goal has been to get Chronobreak on NA LCS so that we could trial it out and test it for global rollout." The double space between "and" and "test" is really bothering me. Please fix it item 3090

ScotFeb 16, 20172 votes

Why does the rest of the game matter cant you just save the state the game was in and revert to that?

RiotIzeoFeb 16, 20176 votes

We'll probably go into more detail on the options we considered in a tech blog, but a boiled-down response to why we didn't choose this one --

Basically, we'd have to constantly save our gamestate and have a method to restore it. To make this possible without causing hitches and requiring obscene amounts of disk space and a bunch of terrifying code to save/restore all the saved state in the various systems and restore timers and whatnot (deep breath), we'd have to significantly rearchitect League around that idea, especially to keep supporting it going forward. It'd be a massive massive endeavor. Making the server deterministic was, while still crazy, a more realistic endeavor for saving games any time soon (such as Sunday's!).

Aside: While the list is small, there are a few game engines built in a way that easily supports this method. Jonathan Blow gave a TED talk (up on YouTube) about how he achieved rewinding in Braid -- a solid example of how you can architect a game to support this (but also goes to show how you have to be putting a ton of effort into it from the onset).

Hope that answers your question!

Mysticman89Feb 16, 20171 votes

Are the LCS servers actually different from the live servers in being deterministic then?

I would have thought the only sources of non-deterministic gameplay off the top of my head to be crits, dragon spawns (and maybe plant spawns), and even those are probably just pseudo-random and thus would behave deterministicly if the RNG was given the same seed (depending on the particulars of the RNG I suppose, but non-deterministic random strikes me as harder than pseudorandom).

RiotIzeoFeb 16, 20175 votes

Nope! The live servers are just as deterministic as the LCS servers -- the primary difference is that the LCS servers run very few games at once, so they can afford the extra CPU cycles and disk space to store some extra stuff (server inputs, some bits of info for verification) without causing hitches and other tragedies.

Note: The determinism we've achieved is only for our re-runs of the same game -- e.g. we don't use the same RNG seed for different games of League. (So, it's only really applicable for Chronobreak and dev things like reproducing bugs or making sure a refactor didn't change the game. Our goal was not to remove randomness from League, it still has its purpose, even if one day it's just getting different voice lines).

As for causes of non-determinism (in this context, at least) -- we'll really get into this in the tech blog(s), but some examples: Some RNG's choose numbers partly based on OS time (they'd choose a different number during recovery because different time); Logic often tries to use precise/"instantaneous" time (so now if the OS schedules your process differently, you get different results). At a high-level -- it's any form of input we don't capture -> control and replicate during recovery.

Occasionally, I write not-text-walls, I swear.

RottedApplesFeb 17, 20171 votes

[{quoted}](name=Riot Izeo,realm=NA,application-id=9hBQwnEU,discussion-id=dBH0htgy,comment-id=0002000000000000,timestamp=2017-02-16T21:00:35.260+0000) Occasionally, I write not-text-walls, I swear.

It was an interesting text wall don't worry.

MordepoolFeb 16, 20172 votes

Have you considered giving it a cooldown of 1 zilean death after the ability has prevented 2 deaths on the same ally champion

Mr GoobieFeb 16, 20172 votes

Why didn't the third game count toward Fantasy LCS? It costed me my matchup...

DarkestchimeFeb 16, 20171 votes

You were probably on the "first two games" scoring system. Even the "best game" scoring system may not take the third game if it wasn't the players/teams best game of the series.

RiotRiot EmilFeb 16, 20173 votes

This was an unfortunate side effect of the recovery but this should now be resolved, take a look and let me know if you won!

AugintFeb 16, 20171 votes

Why was the bug considered critical? The damage of MissFortune ult did go through even with the visual bug, and it was not enough to kill Zyra . The kill after the recovery was due to a difference in execution by Malzahar and is completely irrelevant to the bug. There was nothing could have done differently in the situation.

SilverSlayer425Feb 16, 20171 votes

imagine an invisible mf ult across the entire team. That is a very scary proposition since the way to counter mf ult is to position correctly.

RiotAlwaysBeClosingFeb 16, 20178 votes

I made this call for a few reasons.

In a situation where it is purely visual, an ultimate of the caliber and strength of MF being invisible creates a significant amount of confusion. FLY immediately petitioned the referee that something was wrong and expressed confusion to each other during the play. I believe their actions during and immediately after the bug was one of confusion that the ultimate was not working predicated on the faulty visual. C9 would also have been offered a remake on this play, since both teams were negatively impacted. But, since Fly already accepted theirs, we began the process.
I am not certain that it was purely visual. I have seen instances of this same bug in the past with no damage or some damage being applied (no question at least some damage was applied). The cannon minion during the last 2 waves of the MF ultimate has the splash particle from her skin but the hp bar is no longer moving. This was after Zyra flashes out, but does bring into question whether the damage was happening properly throughout the ability.

RottedApplesFeb 16, 20175 votes

It was an incredibly hard thing to call. As I'm sure you saw the casters thought the damage didn't go through while the analyst desk did.

I myself was having a hard time figuring out if the damage had gone through or not despite you guys showing the replay over and over again.

AMYS GRAVEFeb 16, 20171 votes

gj man.

The Tank ManFeb 16, 20171 votes

I guess a question about how the game saves information-why do you need to start a new game from scratch and play it all the way up to the point you want rather than just create the new game at that point?

RiotIzeoFeb 16, 20175 votes

The game mostly just doesn't save that information :) When a minion dies, it ceases to exist, no real reason for the game to hold onto that information (and related information like "What was it's health at 5 seconds before it died?" and that entire timeline). Some pieces of information are stored for the sake of spectator and downloading replays, but a lot of the server state isn't stored (e.g. what the AI was thinking) -- just the information needed by clients (e.g. the position it's at), which is several steps removed from what would be needed to restore a game.

As to "Why not change the game to start storing that server information as well?", see my response to Scot (OCE) :)

Screamin BansheeFeb 16, 20171 votes

Say for a moment, after the remake the play happens again in the exact same way with an invisible miss fortune ultimate. Would you try Chronobreak again, or proceed with a full remake?

RiotAlwaysBeClosingFeb 16, 20171 votes

If it were literally the exact same way, then full remake (champion disabled in this case). But, if we are able to ascertain reproduction steps we may be able to inform the team in question what is causing the bug and tell them not to do those steps otherwise remake is off the table (this is assuming it is negatively impacting only the team with the champion).

woodvsmurphFeb 16, 20171 votes

Seems like a great tool and one that puts League more in line with other professional sports where officials can review a play and reset the playing field. I hope this will be implemented by worlds of this year, if not sooner. I would suggest you emphasize how resource intense the tool is though, so you don't have a bunch of people asking why it isn't used in their solo queue games to rewind anytime there is a bug. Obviously that isn't possible, but you might want to make that clear and why to people.

Chimp JokerFeb 17, 20171 votes

What did you Rioters call it before giving it's nickname "Chronobreak"? Deterministic Disaster Recovery Tool? I got to say, chronobreak sounds easier to say, but I like the other name better just because it reminds me of DDR (Dance Dance Revolution)....T... (The game) : /

RiotRiot EmilFeb 18, 20172 votes

Pretty much been "Disaster Recovery" or "DDR Tool" since the start.

ModPrandineFeb 18, 20171 votes

[{quoted}](name=Riot Emil,realm=NA,application-id=9hBQwnEU,discussion-id=dBH0htgy,comment-id=000a0000,timestamp=2017-02-18T00:23:04.817+0000)

Pretty much been "Disaster Recovery" or "DDR Tool" since the start.

Is it weird that I automatically think Dance Dance Revolution whenever I see the initials DDR? :P

AzureDragoonFeb 18, 20171 votes

Good to see I wasn't the only one who had similar random thoughts....

ModAttysu The PoroFeb 17, 20171 votes

Thank you for creating this article so I could write about something I actually care about for my Video Game Design assignment. [slayer-jinx-wink]

RiotRiot EmilFeb 18, 20171 votes

Just wait until you see the tech blogs!

ModAttysu The PoroFeb 18, 20171 votes

[{quoted}](name=Riot Emil,realm=NA,application-id=9hBQwnEU,discussion-id=dBH0htgy,comment-id=000b0000,timestamp=2017-02-18T00:24:03.610+0000)

Just wait until you see the tech blogs!

Is super excited [slayer-pantheon-thumbs]

idkidc123Feb 19, 20171 votes

It's funny, when I was watching the game live, I was like "hmmm, what if they could make a tool that would rewind it a little bit without having to remake it" and then shazam, chronobreak was revealed. Well done Riot. I think this tool will help balance the game because sometimes a full blown remake can turn the tide of the series.

TeslynFeb 23, 20171 votes

So, now you have a tool Dota 2 had for awhile. better late than never to the party i guess.

Xi JinpingApr 20, 20171 votes

can you guys share this technology to other regions? China is experiencing a problem that teams are remaking because of their equipment problems and also some bug problems which ruining so many great games, If every region can use this kind of awesome things that will be really great to watch