Boards›Live Incident Breakdown

[DONE!]6/3 - NA Backbone ISP Maintenance - Ping Increase and possible DC's in the South/Southwest US

RiotdArtagnan·6/3/2016, 7:03:13 PM·22 votes·14,721 views

#UPDATE: 11:30AM PDT

The maintenance was successfully completed by Level 3 and the new fiber optic line route is on. It looks like everything is coming up Milhouse!

We've gone ahead and re-routed players back onto the Riot Direct Network. Best of luck on the Rift everyone! ^.^

Hey all,

Recently we’ve been notified that Level 3, a major ISP backbone, is performing major maintenance near Chicago, IL starting on June 3rd 10:00PM PDT and completing sometime the following morning. Since they will be completely cutting one of their main fiber optic lines to alter its path, maintenance at this kind of scale is going to seriously screw up the connection paths we’ve set up with the Riot Direct Network. Because of this, we are going to proactively re-route player connections that go through impacted areas slightly before Level 3’s maintenance at 9:30PM PDT to offset their work.

Here’s what you need to know:

  • If you're in the Dallas area, your ping during this time may rise as you’ll be off the Riot Direct Network. We don’t expect it to increase drastically, but you may want to factor this in when you decide to play!
  • If you’re in Southern California, Arizona, or the general Southwest your connection will probably be taking a slightly longer path via Northern California during this window. You’ll also see a slightly higher ping time while we have these paths out of service.
  • When we re-route connections, there may be some brief disconnects. We’re going to have eyeballs glued to our monitoring during this time, and if we see any server-wide disconnects, we’ll enable Loss Forgiven ASAP.

Once the maintenance by Level 3 is complete, we’ll check everything out on our end to make sure it looks stable. If everything is glowing a bright green, we’ll re-route players back onto our Direct Network. Similarly to when we first did the initial cut over, there may be some disconnects and we’re prepared on our end if we see any server-wide issues.

We’ll provide updates as we get them!

42 Comments

Aeszarck6/3/2016, 7:09:38 PM14 votes

Thanks for letting us know dArtagnan. Good to see you guys are on top of things.

Leanus Crain6/3/2016, 9:15:40 PM3 votes

Still love the fact you guys have a direct network, hope L3 doesn't take long.

Akerasi6/3/2016, 8:07:13 PM3 votes

Although others have said it... from a fellow sysadmin/network admin, thank you for being so clear and up front about what is going on.

Frosty at Heart6/3/2016, 8:53:50 PM2 votes

When you say Dallas, does that include basically all of Texas or what? I'm like 4 hours south of Dallas so can I expect to be effected?

Olıve6/4/2016, 6:41:35 AM2 votes

God, I hate to be that guy, but everyone in SoCal got the absolute worst end of this delay. I absolute couldn't even move, let alone play. Good to see you guys are giving us a heads up, though.

SaltyKracka6/3/2016, 7:18:38 PM2 votes

Well, it's nice of you to keep us in the loop. Thanks for the heads up.

Bonnibel6/4/2016, 1:09:10 AM1 votes

Does this include players that play on the NA server but are not from North America?

nigel shrekberry6/4/2016, 2:48:53 AM1 votes

im in ohio with a constant 38 MS ping, but god damn feels like im at 100 or so, packet loss or something, even random DC, its crazy !

Counterflya6/4/2016, 3:02:00 AM1 votes

I can't even start the client after this update! I live in Germany. Any help? It says "There was an error connecting with Maestro. League of Legends will now exit. Please try restarting the client."summoner 11

Lt Aldo Raine6/4/2016, 3:26:09 AM1 votes

There's a huge delay, hope this ends soon...

zOcean6/4/2016, 3:41:34 AM1 votes

I just lost a game due to server lag, I tried to submit a ticket but IDK why the video files never load, so I could never submit it. Please help ! My ping and ms were fine, Ping was around 150, and ms was around 50. Amumu item 3070

xXM3troGnomeXx6/4/2016, 4:25:12 AM1 votes

I keep trying to run the client and it says the patcher kernel has stopped working, is this one of the issues?

Niverdin6/4/2016, 4:37:38 AM1 votes

I just queued up in a ranked game and I never connected. Every time I click "Reconnect" it just immediately crashes again. Would this cause that issue?

Phygian Legume6/4/2016, 4:59:11 AM1 votes

Definitely caused me and support to dc for ~30 secs and give laner free double kill. #bullshit

dam you ugly asf6/4/2016, 5:11:09 AM1 votes

Riot i just joined a game with 69 ping i had to leave couldnt bear to play please give me and my team the loss forgiven

TheLazySunbro6/4/2016, 5:57:11 AM1 votes

tfw you need to quit your promos cause the lag makes it unplayable Amumu

JCoellner6/4/2016, 6:11:13 AM1 votes

I just played a game that started approximately 1:15am EST which is 10:15 PDT, as a 4v5 because a player never connected, the enemy jungle then disconnected for several minutes, I was disconnected twice. The first time I disconnected I had to re-log in repeatedly, the connection would freeze around 80% and the client would crash saying I disconnected. the 2nd time i disconnected the same thing kept happening. I tested my internet connection throughout this game and was able to connect to the internet pretty easily but my league client kept crashing. When the game ended I couldnt connect to lobby and my client is currently trying to connect to chat service as I type this. Would really appreciate a loss forgiven for a game where 3 players kept disconnecting.

Astrea Myrth6/4/2016, 6:33:49 AM1 votes

12k ping in SoCal just now. RIP ranked.

Crap its Ness6/4/2016, 6:47:24 AM1 votes

Says i'm at 50 ping. feels like im at 300 NotLikeThis....

any update?

ADL6/4/2016, 7:39:43 AM1 votes

Played a ranked game and was dc devastated. Everyone had lag. Half the team DC on our end and 1 person didn't connect. Lost a 4v5 because of it. I personally had 70 ping, but was playing as if I was at 500 ping. Any chance of post-humous lost forgiven? Friend was on a series into silver...

LeBeastie276/4/2016, 7:58:34 AM1 votes

i just played a game and my ping was a constant 100 ms and was nearly impossible to play. I thought maybe it was my router but i was in a call with my friend who lives in the same city and has a different isp and our ping was the exact same the entire game. there was no "heads up" there was nothing on the client that said "hey our server might go to s***, maybe you shouldnt play" no loss prevention or anything. yes, we didn't disconnect entirely but with that high of a ping the game was just unplayable

Rengar Support6/4/2016, 8:34:08 AM1 votes

I wonder if there was any connection problems prior to 9pm because I was lagging hard around 6:30pm I had 71ping but my abilities were going off like 2 seconds later.

barbarianskull6/4/2016, 11:58:40 AM1 votes

Thanks for letting us know

stone the blacks6/4/2016, 2:51:04 PM1 votes

Off topic question but can anyone tell me if Black Market Brawlers is planned to be added to the RGM roster

Punish me Senpai6/4/2016, 3:55:09 PM1 votes

my league off legends don't work :(

Lt Aldo Raine6/4/2016, 4:45:41 PM1 votes

the bs server delay mixed with all the toxic people ive gotten last night and this morning caused me to demote :D

Pr4Y6/4/2016, 5:34:52 PM1 votes

Has this been completed Riot?

Faceroll Tactics6/3/2016, 7:42:19 PM1 votes

Will this affect the New Orleans/Baton Rouge area in any way?

Captaindaddybear6/3/2016, 8:03:16 PM1 votes

awesome, that you're working on it now! Keep up the good work, riot! :)

destroyer0686/3/2016, 8:55:27 PM1 votes

{quoted}

Hey all,

Recently we’ve been notified that Level 3, a major ISP backbone, is performing major maintenance near Chicago, IL starting on June 3rd 10:00PM PDT and completing sometime the following morning. Since they will be completely cutting one of their main fiber optic lines to alter its path, maintenance at this kind of scale is going to seriously screw up the connection paths we’ve set up with the Riot Direct Network. Because of this, we are going to proactively re-route player connections that go through impacted areas slightly before Level 3’s maintenance at 9:30PM PDT to offset their work.

Here’s what you need to know:

  • If you're in the Dallas area, your ping during this time may rise as you’ll be off the Riot Direct Network. We don’t expect it to increase drastically, but you may want to factor this in when you decide to play!
  • If you’re in Southern California, Arizona, or the general Southwest your connection will probably be taking a slightly longer path via Northern California during this window. You’ll also see a slightly higher ping time while we have these paths out of service.
  • When we re-route connections, there may be some brief disconnects. We’re going to have eyeballs glued to our monitoring during this time, and if we see any server-wide disconnects, we’ll enable Loss Forgiven ASAP.

Once the maintenance by Level 3 is complete, we’ll check everything out on our end to make sure it looks stable. If everything is glowing a bright green, we’ll re-route players back onto our Direct Network. Similarly to when we first did the initial cut over, there may be some disconnects and we’re prepared on our end if we see any server-wide issues.

  1. We’ll provide updates as we get them![zombie-nunu-tears] item 3078 Zed ****

Mautok6/3/2016, 11:32:33 PM1 votes

hey i live in chicago WOOT i feel non neglected Yorick

ilovemnms6/4/2016, 1:00:38 AM1 votes

Does this kinda explain why I've been seeing myself and lots of people having packet loss lately? mine started like a week ago and I've never had problems with it before, I live in Wichita KS btw so kinda the same area but not exactly. Or is this just probably my ISP?