Boards›Esports

Deconstructing Chronobreak

Riot·2/16/2017, 1:01:44 AM·3 votes·38,310 views

As you might have seen on the NA LCS, a critical bug during the FlyQuest and Cloud9 series (spoilers) on Sunday, February 12, caused a long pause. Previously, we would have offered a remake since the bug was verifiable and had a critical impact on the game. However, we were able to use our new tool, the Deterministic Disaster Recovery Tool aka “Chronobreak”, to recover the game to a point just before the bug occured.

Since the series, we’ve seen some fan questions about Chronobreak and how it’ll impact competitive play. In this post, we’ll be sharing more information about Chronobreak, how it works, how it’ll be used, and our plans around global rollout.

What is Chronobreak?

Since the start of Spring Split, we’ve had Chronobreak running in the background at the NA LCS. As the name suggests, this new tool allows us to recover from disasters such as bugs during live events to maintain competitive integrity.

How does it work?

Chronobreak saves a Server Replay of a game while it’s being played (a recording of all the inputs received by the server). If needed, the Server Replay can be used to create a new game and play it up until any point in the timeline. Players and Spectators can then connect to the new game and resume play.

Our engineering team will be releasing more information about Chronobreak and Determinism for those interested in a more detailed, technical look at how this tool was engineered and concepted in the coming weeks.

When will it be used?

Right now, we’ll only use Chronobreak in a situation that would warrant remaking the game.  NA LCS officials evaluate bugs on a case by case basis, making sure that they are both:

  • Verifiable: The bug is proven to be a bug and not caused by player error.
  • Critical: The bug had a significant gameplay impacting effect. The effect of the bug would have to greatly influence the competitive game state in a negative way for at least one team. If a bug occurs, but the outcome is not altered by the bug, then it will not be considered critical. For example, a player dying from a bug when they would not have died otherwise is critical.

Once NA LCS officials determine the bug meets the above criteria and a remake is accepted by the impacted team, NA LCS officials will attempt to use Chronobreak to recover the game prior to when the bug occured. If Chronobreak doesn’t work, a full remake will be implemented instead.

While Chronobreak addresses the types of bugs that cause more than 90% of remakes in competitive play, it does not address all of them such as the Aurelion Sol bug at Worlds 2016, which would have recurred even in a recovered game.

Did the NA LCS teams know?

Yes, NA LCS teams were informed about our new tool at the start of Spring Split.

What’s next?

Our goal has been to get Chronobreak on NA LCS so that we could trial it out and test it for global rollout. Testing on NA allows us to iterate quickly on the tool, make it faster, and test it fully before we introduce a global rollout. We’re working on v2 now, which will include getting Chronobreak to every region (including configuring tournament realms, training local operations on how to use the tool etc). We hope to roll it out to other regions starting in the Summer Split.

Questions? We’ll be hanging out in the comments below, so let us know.

 

Emil “RiotEmil” Jannesson is a Live Producer for Esports Operations, a team that has been running the technical framework of tournament realms and events across the globe since 2013. RiotEmil works on a range of events, from NA LCS to the World Championship, and his team is involved in tournament realm support and esports bug triaging. When he's not spending time with his dogs Kayle and Nasus, you can frequently find him ganking unsuspecting foes in the latest PvP games. If you have any questions or want to challenge him to a duel in any competitive game, hit him up on Twitter at @RiotEmil.

 

45 Comments

TurquoiseYoshi2/16/2017, 1:13:03 AM8 votes

Question: Why did you not mention Chronobreak until a use of it occurred? Was it because you wanted to look really cool when it was shown off?

Because it is really cool.

Davìd2/16/2017, 3:50:23 AM3 votes

"Our goal has been to get Chronobreak on NA LCS so that we could trial it out and  test it for global rollout." The double space between "and" and "test" is really bothering me. Please fix it item 3090

Mordepool2/16/2017, 2:10:02 AM2 votes

Have you considered giving it a cooldown of 1 zilean death after the ability has prevented 2 deaths on the same ally champion

Scot2/16/2017, 1:52:07 AM2 votes

Why does the rest of the game matter cant you just save the state the game was in and revert to that?

Mr Goobie2/16/2017, 3:12:25 AM2 votes

Why didn't the third game count toward Fantasy LCS? It costed me my matchup...

Chimp Joker2/17/2017, 3:49:53 AM1 votes

What did you Rioters call it before giving it's nickname "Chronobreak"? Deterministic Disaster Recovery Tool? I got to say, chronobreak sounds easier to say, but I like the other name better just because it reminds me of DDR (Dance Dance Revolution)....T... (The game) : /

ModAttysu The Poro2/17/2017, 6:38:08 AM1 votes

Thank you for creating this article so I could write about something I actually care about for my Video Game Design assignment. [slayer-jinx-wink]

idkidc1232/19/2017, 7:32:16 PM1 votes

It's funny, when I was watching the game live, I was like "hmmm, what if they could make a tool that would rewind it a little bit without having to remake it" and then shazam, chronobreak was revealed. Well done Riot. I think this tool will help balance the game because sometimes a full blown remake can turn the tide of the series.

Teslyn2/23/2017, 5:29:11 AM1 votes

So, now you have a tool Dota 2 had for awhile. better late than never to the party i guess.

Xi Jinping4/20/2017, 9:01:57 AM1 votes

can you guys share this technology to other regions? China is experiencing a problem that teams are remaking because of their equipment problems and also some bug problems which ruining so many great games, If every region can use this kind of awesome things that will be really great to watch

Augint2/16/2017, 2:52:53 AM1 votes

Why was the bug considered critical? The damage of MissFortune ult did go through even with the visual bug, and it was not enough to kill Zyra . The kill after the recovery was due to a difference in execution by Malzahar and is completely irrelevant to the bug. There was nothing MissFortune could have done differently in the situation.

The Tank Man2/16/2017, 4:48:50 AM1 votes

I guess a question about how the game saves information-why do you need to start a new game from scratch and play it all the way up to the point you want rather than just create the new game at that point?

Screamin Banshee2/16/2017, 2:20:23 PM1 votes

Say for a moment, after the remake the play happens again in the exact same way with an invisible miss fortune ultimate. Would you try Chronobreak again, or proceed with a full remake?

woodvsmurph2/16/2017, 6:16:37 PM1 votes

Seems like a great tool and one that puts League more in line with other professional sports where officials can review a play and reset the playing field. I hope this will be implemented by worlds of this year, if not sooner. I would suggest you emphasize how resource intense the tool is though, so you don't have a bunch of people asking why it isn't used in their solo queue games to rewind anytime there is a bug. Obviously that isn't possible, but you might want to make that clear and why to people.