The Good, The Bad, and the League: 8/22 - 9/4

Your semi-weekly dose of server problem-os, NA League news, and other!

We’ve got the final team qualifying for Worlds from NA this weekend, Pantheon heroically dove into TFT, and there was a bunch of player impacting problems during this period :(

Follow-Ups:

• Mac voice still not working properly (Bug under development)

Ongoing

• ** None **

**Server Stuff: **

• High Error rates for players logging on(8/22, ~68 minutes) Automated alerting notifies the NOC that login errors are climbing. Investigation starts, but before a root cause can identify where or why the service is having problems, the errors return to normal values.

• Internal data tracking too high(8/22, ~25 minutes) An internal monitoring system spits out an alert that a particular data tracking thing is overloading with data (too much volume). As the NOC alerts the team that owns that thing, to start analyzing why the system is getting too much information, the levels return back to normal.

• Mission delays for players(8/22, ~10 hours) Internal monitoring notifies the NOC that mission processing is delayed from average values. Teams that deal with Missions are alerted, and investigation shows a few possible root causes. Continued investigation leads to the realization games are getting stuck in the processing DB and not moving. Rioters manually remove the one game that was stuck, and processing continues at normal values.

• LP gains/missions progress not working in TFT(8/27, ~24 hours) The main patch is deployed, however other regions notice TFT LP gains are not working properly, as well as missions aren’t progressing. TFT is disabled in all regions. Testing on PBE leads to a proper fix, but it will require platform restarting, so the fix is scheduled the following morning. The fix rolls out, the platform is force-restarted, and TFT starts working properly again.

**Game Stuff: **

• Ping increase for non-peered ISPs in Eastern US(8/24, ~33 hours) First reports on Boards about higher ping for players. Network engineers investigate, and confirm that routing looks suboptimal for a swath of non-peered ISPs. Network engineers uncover that backend internet connections (the big pipelines the entire Internet usually is routed through) have differences in settings that shouldn’t have happened. Communication starts with companies that run the backend, and the delay is fixed on their end.

• Reconnect spikes/ping spikes(8/25, ~210 minutes) Automated monitoring notices a few game servers having reconnect spikes outside of acceptable limits. Automated systems enable compensation mode for the affected servers. Total volume of compensation mode triggers Rioters to start investigating root causes, even as the spikes return to normal levels.

Morgageddon