Boards›Live Incident Breakdown

The Good, The Bad, and the League: 1/4 - 1/21

RiotRiot Morgageddon·1/22/2020, 11:02:23 PM·5 votes·78,032 views

_Your semi-weekly dose of server problem-os, NA League news, and other! _

2020 has started! We’ve got LCS about to start, LoR is soon to enter open beta, Sett is punching an awful lot of things, and everyone loves giant Mechs.

Follow-Ups:

  • Mac voice still not working properly Bug grouped with other Mac bugs. More .plist fun (No progress since last check).

Ongoing

  • Server Transfers Disabled Transfers are disabled until the next platform downtime (likely 10.3), in order to install additional logging and diagnostic tools to improve transfers in the long run.

**Server Stuff: **

  • Aphelios’ Passive/R/E abilities in LCU video not loading(12/10, ~1 month) NOC is notified that Aphelios’ Passive/W/E videos in LCU don’t work. Needless to say, they failed to work because the videos were designed in LCU format to work with a champion with 1 passive and 4 abilities. Since the problem couldn’t be micropatched out, it needed to be fixed with 10.1.

  • PayOn payments not resolving properly(1/9, ~184 minutes) Reports filter in that there’s problems with PayOn. Engineers discover the issue isn’t on League’s side, and work with PayOn engineers to fix the problem.

  • Sign on problem increases(1/13, ~29 minutes) Automated alerting notifies the NOC that signing in problems are reaching problematic levels. Investigation leads to a dependent service (the backup) wasn’t properly resolving issues when the primary was having problems. Updates will need to be made to the dependent service to prevent this from happening in the future.

  • Sign on problem increases(1/13, ~29 minutes) Automated alerting notifies the NOC that signing in problems are reaching problematic levels. Investigation leads to a dependent service (the backup) wasn’t properly resolving issues when the primary was having problems. Updates will need to be made to the dependent service to prevent this from happening in the future.

  • Players unable to log into Riot services(1/14, ~61 minutes) Automated alerting notifies the NOC that sign on errors are reaching unacceptable levels. Engineers find that a core part of the sign in process isn’t responding to requests. Issues expand to impact other core services as well. Engineers need to spin up new instances and help offset the load as the systems start to recover.

  • High CPU Utilization across game servers, requiring mitigation steps(1/15, ~178 minutes) Due to warning levels of CPU utilization, NOC technicians set minimum numbers for custom games, as well as enabling queues for TFT and ARAM. As usage lowers, throttles are removed.

  • High CPU Utilization across game servers, requiring mitigation steps(1/17, ~220 minutes minutes) See above

  • High CPU Utilization across game servers, requiring mitigation steps(1/18, ~115 minutes minutes) See above above

  • High CPU Utilization across game servers, requiring mitigation steps(1/19, ~209 minutes minutes) See above above above

  • High CPU Utilization across game servers, requiring mitigation steps(1/20, ~112 minutes minutes) See above above above above

  • Social Panel missing from LoL Client(1/16, ~6 hours) A change is made to the backend servers during a change window. However, the change sees the social panel go down. Issue starts to impact all regions, and a triage team starts work on a hotfix. The hotfix is about to be deployed, only to find it’d cause another problem if deployed as is. Hotfix undergoes another QA pass and is approved for deployment, which solves the problem.

**Game Stuff: **

  • ARAM expanded F2P rotation list not updating(12/23, ~1 month) NOC is notified that changes fail to execute on the F2P rotation list. Investigation leads to the expanded F2P rotation as a starting point. Investigation ends noticing that it happened due to a manual config change that was incorrect.

Morgageddon

34 Comments

ed37059e-862c-4915-ac88-0d60551fec441/23/2020, 5:20:25 AM4 votes

Can you do something about dodging a match in rank. It's been a thing lately and has been so annoying to play. A waste of time and LP just so the game is on their favor

DiscworldDeath1/23/2020, 9:14:13 AM3 votes

Errr, aren't we missing a bunch of stuff in this post?

  1. EUW has had a ticker for "Account transfers currently unavailable" for about a week now.

  2. Mission and milestone progress, as well as tokens not being dispensed in EUW, multiple instances.

  3. End of match, sometimes mastery points, score, and honours are not distributed - not merely visual. This has been ongoing from as early as September and as late as last week, personally.

RiotRiot Morgageddon1/23/2020, 11:04:57 PM3 votes

{quoted}

Errr, aren't we missing a bunch of stuff in this post?

  1. EUW has had a ticker for "Account transfers currently unavailable" for about a week now.

  2. Mission and milestone progress, as well as tokens not being dispensed in EUW, multiple instances.

  3. End of match, sometimes mastery points, score, and honours are not distributed - not merely visual. This has been ongoing from as early as September and as late as last week, personally.

If we did EUW, I'm pretty sure my posts would be a bit longer. However, this is currently aimed at engagement with the NA playerbase, since my current role is regional specific (NA). If I was Central, then we might see stuff from EUW, EUNE, Russia, LATAM, etc.

Minarde1/23/2020, 2:22:21 AM2 votes

High CPU Utilization across game servers, requiring mitigation steps(1/15, ~178 minutes) Due to warning levels of CPU utilization, NOC technicians set minimum numbers for custom games, as well as enabling queues for TFT and ARAM. As usage lowers, throttles are removed.

Any particular reason to specifically hit TFT and ARAM? Why not enable queue timers for all modes?

Yukata1/26/2020, 8:36:45 PM1 votes

Ehhhh don’t wanna read I’m sure it’s just more toxicity from u guys probably:) lololololololololoooolooollll u

xXN1ghtXx1/29/2020, 3:56:57 PM1 votes

Thats cool and all but what about nerfing bot lane as lane? In case you havent noticed games in solo queue in literaly every elo are decided 5 mins in since the team with the feeding bot loses 100% of the time. Losing bot lane means losing drakes/jungle priority(jungler is now useless), enemy jungler doesnt need to gank half the map(guess what top and mid and now 2v1 maybe even 3v1's if the enemy adc is fed enough) and this happens every single game. No one likes it. BOT LANE META SUCKS. And you know it does, we have one every single season and everyone hates it: Its boring for 2/3 of the map(you can do anything without getting extremely behind), adc mains hate it because if they make 1 mistake they'll get flamed by their entire team. Just nerf the bot side, PLEASE.

Green Girl1/31/2020, 12:05:33 AM1 votes

[{quoted}](name=Riot Morgageddon,realm=NA,application-id=6WpmgL9O,discussion-id=lfz2Bvfb,comment-id=,timestamp=2020-01-22T23:02:23.052+0000)

_Your semi-weekly dose of server problem-os, NA League news, and other! _

2020 has started! We’ve got LCS about to start, LoR is soon to enter open beta, Sett is punching an awful lot of things, and everyone loves giant Mechs.

Follow-Ups:

  • Mac voice still not working properly Bug grouped with other Mac bugs. More .plist fun (No progress since last check).

Ongoing

  • None

**Server Stuff: **

  • Aphelios’ Passive/R/E abilities in LCU video not loading(12/10, ~1 month) NOC is notified that Aphelios’ Passive/W/E videos in LCU don’t work. Needless to say, they failed to work because the videos were designed in LCU format to work with a champion with 1 passive and 4 abilities. Since the problem couldn’t be micropatched out, it needed to be fixed with 10.1.

  • PayOn payments not resolving properly(1/9, ~184 minutes) Reports filter in that there’s problems with PayOn. Engineers discover the issue isn’t on League’s side, and work with PayOn engineers to fix the problem.

  • Sign on problem increases(1/13, ~29 minutes) Automated alerting notifies the NOC that signing in problems are reaching problematic levels. Investigation leads to a dependent service (the backup) wasn’t properly resolving issues when the primary was having problems. Updates will need to be made to the dependent service to prevent this from happening in the future.

  • Sign on problem increases(1/13, ~29 minutes) Automated alerting notifies the NOC that signing in problems are reaching problematic levels. Investigation leads to a dependent service (the backup) wasn’t properly resolving issues when the primary was having problems. Updates will need to be made to the dependent service to prevent this from happening in the future.

  • Players unable to log into Riot services(1/14, ~61 minutes) Automated alerting notifies the NOC that sign on errors are reaching unacceptable levels. Engineers find that a core part of the sign in process isn’t responding to requests. Issues expand to impact other core services as well. Engineers need to spin up new instances and help offset the load as the systems start to recover.

  • High CPU Utilization across game servers, requiring mitigation steps(1/15, ~178 minutes) Due to warning levels of CPU utilization, NOC technicians set minimum numbers for custom games, as well as enabling queues for TFT and ARAM. As usage lowers, throttles are removed.

  • High CPU Utilization across game servers, requiring mitigation steps(1/17, ~220 minutes minutes) See above

  • High CPU Utilization across game servers, requiring mitigation steps(1/18, ~115 minutes minutes) See above above

  • High CPU Utilization across game servers, requiring mitigation steps(1/19, ~209 minutes minutes) See above above above

  • High CPU Utilization across game servers, requiring mitigation steps(1/20, ~112 minutes minutes) See above above above above

  • Social Panel missing from LoL Client(1/16, ~6 hours) A change is made to the backend servers during a change window. However, the change sees the social panel go down. Issue starts to impact all regions, and a triage team starts work on a hotfix. The hotfix is about to be deployed, only to find it’d cause another problem if deployed as is. Hotfix undergoes another QA pass and is approved for deployment, which solves the problem.

**Game Stuff: **

  • ARAM expanded F2P rotation list not updating(12/23, ~1 month) NOC is notified that changes fail to execute on the F2P rotation list. Investigation leads to the expanded F2P rotation as a starting point. Investigation ends noticing that it happened due to a manual config change that was incorrect.

Morgageddon

what about this ? https://boards.na.leagueoflegends.com/en/c/general-discussion/EhQ42ldU-riot-why-is-server-transfer-down-again

RézZZ1/31/2020, 7:55:07 PM1 votes

Why are you ignoring Server transfers not working for the hundreth time on server stuff?

Padme7772/1/2020, 4:00:20 PM1 votes

lol 1/4 is my mid and 1/21 is bot lane

Nordic Fate1/23/2020, 12:04:16 AM1 votes

are you ever gonna fix voice chat on Mac

Generic Ninja2/1/2020, 4:22:01 PM1 votes

Do you perhaps know what the rationale behind the downgrade of League Accounts to Riot Accounts was? Because Riot Accounts are clearly just inferior League Accounts, with worse functionality but no benefit at all. And all the official reasons obviously dont hold up (such as the claim that it was needed for the other games Riot is going to release, which is disproven by the fact that Legends of Runeterra works fine with a league account).

the Anarchit3cht1/23/2020, 12:34:58 AM1 votes

Any plans to combat the rampant bots issue in SR now that TT is gone? It was easy to just ignore because no one really played TT, but its seriously ruining bots games. I play bots to get my fwotd or grind missions and the lack of human players is garbage. How have you guys not found a solution for this? At least bring TT back so they all go there.

Swiff Swish1/24/2020, 7:34:21 AM1 votes

Psst . . . .

any idea if Shurima is coming to Legends of Runeterra in the future, please tell me yes...[zombie-nunu-tears]

ritewhite1/26/2020, 8:17:00 AM1 votes

[slayer-jinx-unamused]