Engineering perspective on the Server Move(would love a rebuttal by Riot).

Kraithor·8/26/2015, 8:51:22 PM·4 votes·648 views

Hi, I felt like posting this after being an engineer at several of the largest tech companies in the valley. I have dealt with many scaling issues and network optimization, on a much larger scale than League of Legends. There is many many companies that deal with scaling issues an order of magnitude greater than the amount of people playing League of Legends.

What annoys me is the "technical" reasons Riot gives out to justify why they implement things in a certain way, may sound correct, but in reality are just compensation for laziness.

For full disclosure, I am under the opinion that moving the server to Chicago was an OK move, but definitely a lazy move and they could have come up with a much better solution. I would absolutely love for a Rioter to explain any problems from a technical standpoint with my argument. Lets start with some basic notes:

Ping is the most important aspect for networking in this type of game. Riot can claim that they have improved other aspects of the connection, but as long as everything else is optimized the most important aspect for if the game feels and responds well is the rountrip time from your computer to their server. Throwing out the argument that while pings may go up in the west coast, the new servers should provide better stability is disingenuous because they could have provided that increased stability and optimization without also increasing the ping, that is just a fact.

Even by the standards of Riot's own client, a 61+ ping is not an optimal ping, this can be shown by if your ping goes above 61MS your network bar in the top right turns from green to yellow. I think while everyone can agree a 60-70 ping is playable, you can feel the difference between a 20-30 ping and 60-70 ping, especially using abilities which have cast times or travel times, of 1/10th of a second. Striving for a 60-70 ping is a lazy goal to strive for. Increasing ping is also increasing the number of hops, which in turn increases the probability of problems happening between the server and you, such as packet loss. This is why the easy coast had more jitter before(people's pings varying from 80 to 100, or 100 to 140), the more hops, the more room for variance in the system.

Riot should be trying to make the game optimal for the highest % of users, and should do strive to do that, this is a process of compromises and balancing. Lets start with things Riot did right:

The server location is a good spot for a shared server for NA, they are correct about Chicago infrastructure. It is relatively close to the center for the amount of infrastructure, but definitely favors the east coast over the west coast. I think the Chicago location is a good spot for some League servers as I will explain.

Peering with ISPs is very helpful and good, but honestly they should do this wherever they host their servers, if they weren't doing this before then they should have been regardless of where the servers are at.

Some assumptions I can't answer:

Does riot own their own server infrastructure? This would affect cost reasons. Are they using a colocation for their servers. If the reason for Riot picking such a lazy solution was cost, then it can't be argued with(but they should maybe admit to this). It is the only reason I can see for why they chose the route they did, because it is cheaper.

So lets talk about the less lazy solution splitting the servers in an intelligent way, first the downside of splitting NA:

Player base would be split, longer queue times for smaller player pool queues: i.e. Dominion/Twisted Treeline/High ELO Ranked. This is a valid concern, but seems to be easily resolved through software solutions. Lets break this down:

If 45% of the people are on the west coat, and 65% of the people are on the east coast, we will work under the assumption that 65% of the people got their ping improved, and 45% got their ping degraded. Sounds good right? More people are in a better state than before. Except that their reasoning for doing it this way to keep NA from splitting and having longer queue times.

I am going to be generous with my guesses of percentages because I don't have access to the real statistics: I would guess that 5% of the league population plays dominion, and maybe 10% of the league population plays twisted treeline. Furthermore around 1% of the league population are in the High ELO long queue times.

This adds up to 16% of the league population that was benefitted from not splitting the servers... while 45% of the population had a degraded server quality as a result of the move. Oh but wait, of those 16%, some of them are on the west coast, by 45% around 7% of the 16% they are trying to help now have degraded pings for what they do play.

Furthermore they could have easily solved this issue by keeping low population pool servers on a shared server, such as chicago, and splitting high population pool servers into an NA West and NA East.

The second lesser problem of splitting the servers: People have friends in NA on different coasts and would like to play together. This is simple, use the same NA servers for storing match history and ranking, and just split the servers that the game is played on. This way if you want to play with friends, you can but you might have to choose a server with a degraded ping, but people that don't need to play with east coast people could choose to have a better experience. Its win win. They could even build it into the client so people don't have to choose, it could just pick the most optimal server for them based on looking at the pings of the group in the party.

Furthermore the last group of people, the 1% of high elo players could either only be placed against other people on their coast(maybe these queue times would be too long), or placed on a central server(no different than what we have with the server move). The way ranking works its not like one person needs to beat another person to rank up, its all about averages, so this doesn't seem like it would affect the ranking system at all.

This gets me to my greater point, to implement a server split would require new client code to correctly split people into the server, and maintain the ability to go back and forth between the two servers, lets call them NA East and NA West. This is where it seems Riot ALWAYS chooses the lazy approach, whenever there is new client code involved Riot seems hesitant to change it from an outside perspective. This could have been handled through a split and a software choice of where players play, and that would result in 90% of the US playing with an optimal 30-40MS ping, no degraded ping for the west coast, and a hugely improved ping for the east coast, if they just did a little bit of client work.

TLDR; If Riot split the servers AND updated the client code, the vast majority of the US could be playing on a very low ping, low population queues could have the current shared ping, or some other compromise. People in the US could still play with their friends on the other coast, with the same quality as the current server move. Riot's explanation has never addressed this, and it makes them seem like an extremely lazy company the way they tend to implement things from an engineering perspective. Would love a rebuttal.

17 Comments

Jubbinaut8/26/2015, 9:02:14 PM3 votes

{quoted}

Ping is the most important aspect for networking in this type of game. Riot can claim that they have improved other aspects of the connection, but as long as everything else is optimized the most important aspect for if the game feels and responds well is the rountrip time from your computer to their server. Throwing out the argument that while pings may go up in the west coast, the new servers should provide better stability is disingenuous because they could have provided that increased stability and optimization without also increasing the ping, that is just a fact.

I disagree. I'm fairly certain that packet loss is the most important aspect for networking in this type of game - communication failures must of necessity be considered more important than communication delays.

Even by the standards of Riot's own client, a 61+ ping is not an optimal ping, this can be shown by if your ping goes above 61MS your network bar in the top right turns from green to yellow. I think while everyone can agree a 60-70 ping is playable, you can feel the difference between a 20-30 ping and 60-70 ping, especially using abilities which have cast times or travel times, of 1/10th of a second. Striving for a 60-70 ping is a lazy goal to strive for.

I've never had ping below 80 (until now). My network bar is always green (unless I'm having massive spikes of ~100+).

Peering with ISPs is very helpful and good, but honestly they should do this wherever they host their servers, if they weren't doing this before then they should have been regardless of where the servers are at.

This is something they've been working on for some time to increase stability. It's the reason the server transfer was delayed - they wanted to be sure they had as strong of an infrastructure as possible before addressing server location.

Niyumi8/27/2015, 4:59:15 AM1 votes

You call them lazy, I call them greedy. Riot is a company that runs an F2P game. From what I've seen, F2P companies will do just about anything, cut any corners, just to save a nickel in the end -- and Riot is no different. If they did the proper thing, and split the servers, it would cost them money to code in all the new things, it would cost them money to keep both servers up and running, and so on.

I'm not really surprised Riot took the easy way out that solves nothing in the long run.

AlsoJinxBot8/27/2015, 7:33:00 AM1 votes

Completely agree with everything you said kraithnor. I live fairly close to portland and use to get an average of 19-26 ping, i'm currently at 65 which is by all means playable. Now i'm not engineer nor do i know the cost of such large servers but besides server hosting costs, why exactly would it make sense not to make 2 servers? Portland one is obviously still active and there, just not being used and the Chicago ones are already up and being used, so why not just utilize both servers?

Also, i feel like you hit it pretty spot on about the whole client situation. I read awhile back that simple things such as naming champions like for example Quinn was supposed to be named "Quinn and Valor" but issues within the code wouldn't let them. I feel like they defiantly aren't "lazy" but they are very horrible at prioritizing things. All that work that went into the 5 recent project skins could of went into work on making a new client, which we should of got more then 2-3 seasons ago. I don't know how much work goes into making a new client, but i know it defiantly shouldn't take this long. I love their game and the content it has but they really need to get their stuff together and get some stuff done.