The Good, The Bad, and the League: 10/19 - 11/1

Your semi-weekly dose of server problem-os, NA League news, and other! (Moved to Mondays for easier updates)

Worlds is down to the final two teams, TFT set 2 will be rolling out soon, patch 9.22 is close, and the end of split 3 is nigh.

Follow-Ups:

• Mac voice still not working properly Bug grouped with other Mac bugs. Something to do with .plists

Ongoing

• None

**Server Stuff: **

• Messaging to players impacted within Client(10/24, ~90 minutes) NOC tries to post messaging on ticket to players in EUNE, messaging fails to post. Investigation shows posting to any ticker fails. Assistance from other teams leads to discovering someone didn’t upload the proper key to master and clients, causing a desync.

• Incorrect SSL certificates(10/25, ~12 minutes) Reports come in that .leagueoflegends.com websites are showing with expired SSL certs. NOC escalates to an appropriate Riot group, who delete the old certs and update to the correct one.

• Match History failing to load(10/27, ~140 minutes) Automated alerting notifies the NOC that data volume for match history is too high. Escalation to the appropriate Riot team leads to disabling Match History while triaging what’s going on. A rolling restart of a critical service is required and started, and after completion Match History is re-enabled.

• Extended HW Maintenance(10/30, ~37 minutes) Network engineers notices that 2 nodes are in an unhealthy state after the scheduled hardware maintenance (and before the servers are opened for players). New instances to replace the unhealthy ones fall over as well, and investigation leads to a full restart of the services to fix the problem. The full restart ends up fixing the problem, but also exceeds the expected downtime, hence causing the unwanted extension.

• Issues with players purchasing content(10/30, ~38 minutes) Automated alerting sees a spike in errors but doesn’t trigger any flags. Player Support notifies the NOC about issues with purchases. NOC pages the appropriate team, which discovers a config change to store caused the issue, and the change is rolled back.

• Players having login issues(10/31, ~55 minutes) Log-in errors begin to spike across the US. Automated alerting notifies the NOC, and Player Support notifies the NOC around the same time. NOC contacts the proper team, who discovers an update to a service didn’t finish properly, which forced a rollback of the data.

• Games failing to upload, impacting Match History(11/1, ~125 minutes) Match History stops working as the data being transmitted/stored is overloading the allocated bandwidth. Adjustments are made to the service to fix the backlog of games waiting to be stored.

**Game Stuff: **

• Project Akali Rose Quartz Chroma disabled(10/25, ~1 hour) The NOC is notified via email that Project Akali’s Rose Quartz Chroma has missing assets, which cause invisible animations in game. Considering that has some game-breaking issues, the NOC disables that particular Chroma until the assets are fixed. This bug will be added to follow-up section.

Morgageddon