Arena: Anatomy of a Pause

It's game three of a tight series at the NA LCS Battle Arena. The mid-laner surges down the river, roaming on a wild hunch. He smashes his ult, flashes into bot lane, and then -- a pause. The crowd groans, viewers wonder what’s going on, then, the shoutcasters explain the problem. After some duration of time, the game resumes.  

But what actually happens on stage and backstage during the pause?

Some of the team who help keep NA LCS running smoothly sat down to explain how pauses happen, how we prevent them, and what’s going on behind-the-scenes.

[[{"fid":"24309","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/01-pausearticle-fin.jpg%3FO8siqAeDkeyRq0.K6pJu9wZKnkIPVsds%26itok%3DXFewLDCf","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

Before we dive into pauses, what do we do to prevent a pause from happening in the first place? Can pauses come from players or only referees?

RIOT RAVES, HEAD REFEREE: The two biggest things we have to prevent pauses are standardized hardware and a checklist every pro has to complete before the game starts. Every pro computer uses the same base parts except for mouse and keyboard. Components like the graphics card or hard drive need to be on the market for 6-12 months with very stable driver releases. Also, our IT team is always checking out hardware and software to provide the most stable of venues, they’re why we’re so confident with the parts and software we use.

Also, before any LCS game starts, every pro has a list of steps they need to complete regarding their keyboard, mouse, computer, key bindings, settings, and so on. If a pro finishes this checklist correctly and signs off to their team referee, we shouldn’t see any issues related to an incorrect setup.

[[{"fid":"24301","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/02-pausearticle.jpg%3FCGgpYVuYp7wmrBTscQfKDT9osRpGOnCQ%26itok%3DAQBx7rwG","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

Both the head ref spectator account (operated backstage) and players can pause a game. If a player sees something wrong, the pro can pause the game with the command /pause into chat. If a stage ref sees something wrong, he or she can tell any pro to pause on voice comms as well. 

So, we’re in game at the NA LCS, everything’s going well and then...there’s a pause. What happened? What’s going on? Why?

RIOT RAVES: First, someone pauses the game and immediately talks to their team’s referee. Probably 80% of pauses occur around the start of the game and are something fixable within three minutes. The problem could be, “My runes are wrong,” or, “Keybindings aren’t working,” or something far worse. Often, these early issues are items on the checklist that a pro didn’t complete properly and can be quickly fixed.

[[{"fid":"24310","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/03-pausearticle-fin.jpg%3FIZKd3f9sEx2G.MWA0ZKUDnDhuzVyzKr3%26itok%3DWdCWKBKe","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

As Head Ref, how is this communicated to you? Who else needs to know?

RIOT RAVES: As soon as a player tells their ref what’s going on, that ref relays the information into the Live Operations (the people responsible for running the show) mic lines. At that time, the head referee, Live Producer, and other key people get basic information about the problem, like, “Keyboard problem. Graphical bug. Monitor issue.”

RIOT EMIL, LIVE PRODUCER: Since I’m informed along with refs in that first update, I can now track how severe the problem is and communicate to Broadcast Producers (the people responsible for the on-air presentation of the games) so our Shoutcasters know when we cut to show the player having issues, and how long we’re going to be out of game.

[[{"fid":"24303","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/04-pausearticle.jpg%3Ftn9NenwCkcuYnfYe5DRdqhw91wnnUzoU%26itok%3DkZm-yE3m","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

So once Live Operations and the refs know, what happens next?

PHREAK, SHOUTCASTER: Right after Riot Emil and Riot Raves knows, Broadcast is also relayed information about the problem, even if it’s a few words such as, “Keyboard issue, red team, two minutes.” The camera will cut over to a shot of the team and a ref around the problematic part, in this example a keyboard, and we can tell viewers exactly what’s going on. Hopefully we have something good to fill up the remaining 60 seconds with relevant information and then we’re back in game. It’s also important that what we talk about won’t give critical information to a team that they may not have known, like a large gold lead or an impending gank, if one of the pros has their headphones off to communicate more about a problem to the refs.

THEMAY0R, BROADCAST PRODUCER: From a broadcast standpoint, viewers always need to be informed about what’s going on if we’re not in game. When the broadcast is in a game, it’s easy to see that everything’s great and that the show is running smoothly. But when we’re out of game for whatever reason, fans will need to know what’s happening.

When there is a pause I can expect to hear from two different places that something’s going wrong: The Observers (team who controls the in-game camera) and from Riot Raves/Riot Emil (Head Ref and Live Ops). Now Production calls to modify the broadcast graphics on the fly with “Technical Pause” then, the director will call to cut to a player cam, so anyone tuned in or just tuning in will see, “Oh, something’s wrong with that pro’s computer.”

PHREAK: That’s why you’ll hear me or Riv go, “It looks like Sneaky’s keyboard is having problems” when the camera cuts to them, and we can talk about what led up to this point and then what we’re going to see right out of it. With Broadcast Production showing viewers directly what’s happening, and us talking about it, anyone who’s watching should know the who, what, when, where, and why we’re paused.

What about the times where it’s not so short and sweet? Say it’s a long pause, does anything change?

PHREAK: Well, the long pauses can be something simple, but time consuming, or…

RIOT RAVES: Bad. Really bad. Basically, there’s two different types of issues if it’s not a quick pause: more involved hardware problems and bugs. No one wants it to be a bug at all, triaging a bug is not easy and it’s never really the same every time there’s a bug. Problems that are bugs are really bad to work through, if it’s something known we probably have a work around, but if it’s completely new we’re in unknown waters.

RIOT EMIL: While swapping a keyboard only takes a minute or so, there are problems which are not so quick and easy to fix. Say a pro’s computer screen starts flickering. Or they blue screen. Or input is being delayed for whatever reason. Instead of diagnosing the problem right there and causing a really long pause, we’ll swap out the entire computer. Once we’re sure this is the problem we need to solve, we let players talk again.

[[{"fid":"24311","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/05-pausearticle-fin.jpg%3FMBN_LdC9k7LjjDuykljBFZ8u9kXqEqWj%26itok%3DovXzrGzq","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

Pros can talk during pauses?

RIOT RAVES: As soon as we get everything we need from them in order to start diagnosing or solving, we don’t really need their help anymore. So, instead of sitting in silence, Referees let both teams talk at the same time. It’s fair this way—they have the exact same amount of time to discuss whatever instead of one team getting a few extra minutes over the other.

RIOT EMIL: Swapping a PC takes six or seven minutes, and IT knows that they’ll be swapping PCs around a minute into the pause. Once we inform Broadcast we’re swapping it, they know there’s another six minutes that needs to be filled before pros get back in-game.

PHREAK: If we hear it’ll be six minutes before we can return to the show, we can go “Hey, isn’t there a piece around Team A that’s five minutes long that we played earlier? We can replay that right now, then cut back to the analyst desk and bring up the pause is almost over.” Camera will cut back to the teams afterwards and then we can jump right back into game with only a few moments of time not showing viewers entertainment.

How about an actual bug? If something’s wrong with the game itself, how bad can it get? Like that one MF ult Spring 2017.

RIOT RAVES: Not the MF ult one.
RIOT EMIL: Oh yeah, that one.
PHREAK: Yeah, that was a fun one.

THEMAY0R: We don’t talk about that one /s. Seriously though, we talked about that one a lot. There were a lot of things that went wrong across the board, as well as new tech we tried out.

RIOT EMIL: Yeah, a lot of things were bad on that bug.

For those who don’t know or remember, the MF bug was…

RIOT RAVES: FlyQuest vs. Cloud9. MF casts E, before flashing forward and casting her ultimate. After a few seconds a pause is called. The ref hears something like, “My ultimate did no damage.”

RIOT EMIL: So we pull up the separate head ref computer (what we use to monitor from backstage) and we can see MF ulting, but no bullets flying out of her guns. Our natural thought is, “Well if the bullets aren’t there, is damage being done as well?”

[[{"fid":"24312","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/06-pausearticle-fin.jpg%3FK5hMOTpag_2fwXJuanvQ2D7WexW1qPC.%26itok%3D5-AlKa3Z","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

THEMAY0R:  I’m overhearing all of this and thinking, shit. That’s a game breaking bug. I hate these kinds of situations since we never know how long a bug related pause will take. If we remake the game that can take upwards of 20 minutes to resolve. That’s a lot of time out of game.

RIOT RAVES: We’ve got to verify it’s a bug and dealing no damage, see if I can reproduce it under similar circumstances…we’re checking to see if this bug was anything more than visual.

How did you try to reproduce this MF bug?

RIOT RAVES: If we have a clear look at the bug from our spectator options, we have a clear idea what it was and can proceed on what to do next. If we don’t have a clear look, we attempt to reproduce the bug quickly (hopefully within a few minutes) to better understand what is going on with a custom instance of the game with the same champions. At a later point that day or in the week, we’ll use developer tools to try and recreate the exact timing and positioning of the actions that led to the bug.

Cooldowns, summoner spells, timing, we’ll use every bit of information we can get about what was going on from the players as well as the video from their screen to try and make it happen again reliably within 10-15 minutes.

With or without a quick repro to know what caused the bug to occur, we need to either let the game continue as is, or fix the bug. If we can prove the issue isn’t player error, we offer the disadvantaged team a remake, and now we can also offer Chronobreak to fix it. If the bug disadvantaged both teams, both are offered the same choice and if either team accepts we take it.

What do you mean, fix?

RIOT RAVES: When we’ve found gamebreaking bugs, we’ve historically had two options: Remake the game from scratch, or let the game continue. Obviously, this can cause some pretty crappy results for everyone if we’re twenty minutes in and we have to go ‘Just kidding, let’s restart.’ Chronobreak added a third option where we can restart the game and queue it up to just a few moments before the problematic time.

[[{"fid":"24313","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/07-pausearticle-fin.jpg%3F6o.EwpeWmArn94Hur_IEkE9CqgPWd5DK%26itok%3DkqadQ7rD","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

RIOT EMIL: We can’t rewind anything. The way the game is coded, servers are run, once a moment in time has happened, it’s happened. We can’t rewind any game back ten seconds. What we can do, though, is grab what the game was at ten seconds previous, or twenty, or whatever time is best for competitive integrity, and start up a new game from that precise moment. In effect it’s the same game, but it’s a whole new one as well.

RIOT RAVES: Except Chronobreak at that time was…

RIOT EMIL: Under development. For what it did, it was amazing. But for what it can do now compared to what it was then, it wasn’t as easy to use.

THEMAY0R: And it was a little cumbersome..

RIOT RAVES: So now we can do this thing, the question to the team becomes: Do you want to continue, or possibly start the game a little before the problem and if that doesn’t work then it’s a full remake?

Why wouldn’t every team just select the second option?

RIOT RAVES: Except this first test of Chronobreak on a live setting was a big unknown. If you pick Chronobreak and the new game state doesn’t work, you have to remake completely anyway because the active game is lost. Granted, even with the possible problem it caused, Chronobreak v1 was far better than a remake since you have a good chance at not needing to start a game fresh.

Damned if you do, damned if you don’t. Continuing left them in a bad spot, and if we had to remake the game, it wouldn’t be great either. We told FlyQuest and Cloud9 it was the first time we’d be using Chronobreak to start a game just before the problematic time, and they decided to try it.

RIOT EMIL: So we set up the tool and plugged everything in. The game looked set up right (a few moments before the MF ult), and everyone was in the right place.

RIOT RAVES: We told the players exactly where and when the game would start, which was but a few seconds before the original ultimate was cast. Everyone was readied up and got ready to start playing from that point…

THEMAY0R: Broadcast is ready to send us back into game…

RIOT EMIL: We’re ready to resume broadcasting with the match continuing…

PHREAK: And as soon as the game starts, we see the game be paused again immediately afterwards. I think Jatt shouted, “Why another pause?” and hung his head on camera.

RIOT RAVES: Now in the second pause, we told the players that the UI needs a few second to update for ability indicators. When the game unpaused everyone should wait a second or two before worrying an ability won’t come off cooldown. If it did last a few seconds after the pause, we were ready to perform a full remake at that point.

THEMAY0R: And when we finally managed to get back into game, we missed 10-15 seconds of action. With the quick pause and need to cut between the casters and the pros again, by the time we got into game we were slightly behind the action and joined the game mid-fight. So we had to run a replay of the repeat attempt at the kill bot lane in order to catch viewers up.

PHREAK: Lesson learned! We had to be really clear with teams when we used Chronobreak.

RIOT EMIL: There were always plans to include more functionality, but we needed something that worked ASAP, and that’s why 1.0 existed. After a lot of engineering work, we’re now on 2.0 (which has been used successfully multiple times across different regions) and that problem doesn’t exist anymore. We can pick a point before the problem time and ensure a new game is started properly, so Chronobreak is reliable if we have to use it.

PHREAK: Which we never want to.

THEMAY0R: Hell no. But it’s nice to have it there.

So next time a pro hits the pause command, remember that beneath that calm exterior there’s a team of people working furiously to resolve it. May all our pauses be short and few! 

[[{"fid":"24308","view_mode":"centered","fields":{"format":"centered","field_file_image_alt_text[und][0][value]":"","field_file_image_title_text[und][0][value]":""},"tagName":"IMG","src":"http%3A//riot-web-cdn.s3-us-west-1.amazonaws.com/lolesports/s3fs-public/styles/centered/public/08-pausearticle_0.jpg%3F0l9c6gNRkAmiDuRWdHPFmNxt_VkZETVY%26itok%3DC3pPo5Sp","type":"media","attributes":{"class":"media-element file-centered","height":"360","width":"990"}}]]

ARENA is a series about the whats, whys, and hows of the NA LCS, straight from the team behind the League. Over the next year, we'll be exploring topics that NA LCS fans are curious about such as decision-making, processes, and behind-the-scenes content. Check out our previous posts including Finals Caster Diary with Jatt and Pastrytime, Scouting Grounds, and Home of the NA LCS. 

 

GREG “MORGAGEDDON" LOVASIK works in NA Live Services to help improve player experiences for the various services NA provides. While not in Esports, Greg was an LCS Ref for several seasons and holds a handful of internal accolades such as being a Rumble Champion, Top 10 MMR within the company (at one point), and one of the highest totals of games of League played at Riot.