Engineering perspective on the Server Move(would love a rebuttal by Riot).

Kraithor·Aug 26, 2015·4 votes·648 views

Hi, I felt like posting this after being an engineer at several of the largest tech companies in the valley. I have dealt with many scaling issues and network optimization, on a much larger scale than League of Legends. There is many many companies that deal with scaling issues an order of magnitude greater than the amount of people playing League of Legends.

What annoys me is the "technical" reasons Riot gives out to justify why they implement things in a certain way, may sound correct, but in reality are just compensation for laziness.

For full disclosure, I am under the opinion that moving the server to Chicago was an OK move, but definitely a lazy move and they could have come up with a much better solution. I would absolutely love for a Rioter to explain any problems from a technical standpoint with my argument. Lets start with some basic notes:

Ping is the most important aspect for networking in this type of game. Riot can claim that they have improved other aspects of the connection, but as long as everything else is optimized the most important aspect for if the game feels and responds well is the rountrip time from your computer to their server. Throwing out the argument that while pings may go up in the west coast, the new servers should provide better stability is disingenuous because they could have provided that increased stability and optimization without also increasing the ping, that is just a fact.

Even by the standards of Riot's own client, a 61+ ping is not an optimal ping, this can be shown by if your ping goes above 61MS your network bar in the top right turns from green to yellow. I think while everyone can agree a 60-70 ping is playable, you can feel the difference between a 20-30 ping and 60-70 ping, especially using abilities which have cast times or travel times, of 1/10th of a second. Striving for a 60-70 ping is a lazy goal to strive for. Increasing ping is also increasing the number of hops, which in turn increases the probability of problems happening between the server and you, such as packet loss. This is why the easy coast had more jitter before(people's pings varying from 80 to 100, or 100 to 140), the more hops, the more room for variance in the system.

Riot should be trying to make the game optimal for the highest % of users, and should do strive to do that, this is a process of compromises and balancing. Lets start with things Riot did right:

The server location is a good spot for a shared server for NA, they are correct about Chicago infrastructure. It is relatively close to the center for the amount of infrastructure, but definitely favors the east coast over the west coast. I think the Chicago location is a good spot for some League servers as I will explain.

Peering with ISPs is very helpful and good, but honestly they should do this wherever they host their servers, if they weren't doing this before then they should have been regardless of where the servers are at.

Some assumptions I can't answer:

Does riot own their own server infrastructure? This would affect cost reasons. Are they using a colocation for their servers. If the reason for Riot picking such a lazy solution was cost, then it can't be argued with(but they should maybe admit to this). It is the only reason I can see for why they chose the route they did, because it is cheaper.

So lets talk about the less lazy solution splitting the servers in an intelligent way, first the downside of splitting NA:

Player base would be split, longer queue times for smaller player pool queues: i.e. Dominion/Twisted Treeline/High ELO Ranked. This is a valid concern, but seems to be easily resolved through software solutions. Lets break this down:

If 45% of the people are on the west coat, and 65% of the people are on the east coast, we will work under the assumption that 65% of the people got their ping improved, and 45% got their ping degraded. Sounds good right? More people are in a better state than before. Except that their reasoning for doing it this way to keep NA from splitting and having longer queue times.

I am going to be generous with my guesses of percentages because I don't have access to the real statistics: I would guess that 5% of the league population plays dominion, and maybe 10% of the league population plays twisted treeline. Furthermore around 1% of the league population are in the High ELO long queue times.

This adds up to 16% of the league population that was benefitted from not splitting the servers... while 45% of the population had a degraded server quality as a result of the move. Oh but wait, of those 16%, some of them are on the west coast, by 45% around 7% of the 16% they are trying to help now have degraded pings for what they do play.

Furthermore they could have easily solved this issue by keeping low population pool servers on a shared server, such as chicago, and splitting high population pool servers into an NA West and NA East.

The second lesser problem of splitting the servers: People have friends in NA on different coasts and would like to play together. This is simple, use the same NA servers for storing match history and ranking, and just split the servers that the game is played on. This way if you want to play with friends, you can but you might have to choose a server with a degraded ping, but people that don't need to play with east coast people could choose to have a better experience. Its win win. They could even build it into the client so people don't have to choose, it could just pick the most optimal server for them based on looking at the pings of the group in the party.

Furthermore the last group of people, the 1% of high elo players could either only be placed against other people on their coast(maybe these queue times would be too long), or placed on a central server(no different than what we have with the server move). The way ranking works its not like one person needs to beat another person to rank up, its all about averages, so this doesn't seem like it would affect the ranking system at all.

This gets me to my greater point, to implement a server split would require new client code to correctly split people into the server, and maintain the ability to go back and forth between the two servers, lets call them NA East and NA West. This is where it seems Riot ALWAYS chooses the lazy approach, whenever there is new client code involved Riot seems hesitant to change it from an outside perspective. This could have been handled through a split and a software choice of where players play, and that would result in 90% of the US playing with an optimal 30-40MS ping, no degraded ping for the west coast, and a hugely improved ping for the east coast, if they just did a little bit of client work.

TLDR; If Riot split the servers AND updated the client code, the vast majority of the US could be playing on a very low ping, low population queues could have the current shared ping, or some other compromise. People in the US could still play with their friends on the other coast, with the same quality as the current server move. Riot's explanation has never addressed this, and it makes them seem like an extremely lazy company the way they tend to implement things from an engineering perspective. Would love a rebuttal.

17 Comments

JubbinautAug 26, 20153 votes

{quoted}

Ping is the most important aspect for networking in this type of game. Riot can claim that they have improved other aspects of the connection, but as long as everything else is optimized the most important aspect for if the game feels and responds well is the rountrip time from your computer to their server. Throwing out the argument that while pings may go up in the west coast, the new servers should provide better stability is disingenuous because they could have provided that increased stability and optimization without also increasing the ping, that is just a fact.

I disagree. I'm fairly certain that packet loss is the most important aspect for networking in this type of game - communication failures must of necessity be considered more important than communication delays.

Even by the standards of Riot's own client, a 61+ ping is not an optimal ping, this can be shown by if your ping goes above 61MS your network bar in the top right turns from green to yellow. I think while everyone can agree a 60-70 ping is playable, you can feel the difference between a 20-30 ping and 60-70 ping, especially using abilities which have cast times or travel times, of 1/10th of a second. Striving for a 60-70 ping is a lazy goal to strive for.

I've never had ping below 80 (until now). My network bar is always green (unless I'm having massive spikes of ~100+).

Peering with ISPs is very helpful and good, but honestly they should do this wherever they host their servers, if they weren't doing this before then they should have been regardless of where the servers are at.

This is something they've been working on for some time to increase stability. It's the reason the server transfer was delayed - they wanted to be sure they had as strong of an infrastructure as possible before addressing server location.

KraithorAug 26, 20151 votes

I tried to word it correctly, packet loss obviously would affect the connection more, but like I said, if they set up servers correctly and did peering but split them you could still maintain the same network standards in west coast cities AND reduce the ping, what I was saying that ping is the most important element if the network is working correctly in other regards. Ping is the only thing you can only change through location once things are optimized, and what has the largest affect on gameplay if your connection is otherwise in good shape.

The network bar on my client is green on the loading screen, but if I am at 62MS or above it is yellow in the client, if I am at 61 or below, it is green in the client. I know because I can watch it change as soon as it changes.

This doesn't address any of my major points of why could they not push the low population queues to a central server and have high population queues split to achieve better pings for everyone in the vast majority of cases?

JubbinautAug 26, 20151 votes

{quoted}

I tried to word it correctly, packet loss obviously would affect the connection more, but like I said, if they set up servers correctly and did peering but split them you could still maintain the same network standards in west coast cities AND reduce the ping, what I was saying that ping is the most important element if the network is working correctly in other regards. Ping is the only thing you can only change through location once things are optimized, and what has the largest affect on gameplay if your connection is otherwise in good shape.

Meh. I'm a stickler for clarity and completion of information. Handwaving something as important as packet loss seems disingenuous to me.

Though perhaps I'm merely a little touchy about the subject, because I'm still having strange problems with unpredictable packet loss myself.

The network bar on my client is green on the loading screen, but if I am at 62MS or above it is yellow in the client, if I am at 61 or below, it is green in the client. I know because I can watch it change as soon as it changes.

Then perhaps the color is based on "normal" or average ping - which will take time to adjust, since your previous average was somewhat lower than what you can expect now.

Or, perhaps, it's color-coded based on location - 80-87 (my normal range) was pretty low ping for my location, so it always showed green, whereas it wasn't good ping given your location, so when you see it now it shows yellow. Which would make it something that needs to be shifted in the next client patch, or in a hotfix.

None of which is particularly important. For it to be true that the client itself presumes >61 ping to be less than optimal, it must show that for every player who has >61 ping - which it does not.

This doesn't address any of my major points of why could they not push the low population queues to a central server and have high population queues split to achieve better pings for everyone in the vast majority of cases?

Because they feel that normalizing the region to what the midwest was experiencing - while still keeping a single, unified community - is more than acceptable. It's not about queue population. It's about a centralized community.

And about addressing what problems they can while they deal with more complicated issues in the background (which is to say the client is absolute shit, and any significant changes would require a massive overhaul of the entire thing - something they're working on, but anywhere near ready to bring out yet).

KraithorAug 27, 20151 votes

Meh. I'm a stickler for clarity and completion of information. Handwaving something as important as packet loss seems disingenuous to me.

I wasn't handwaving it, I was saying packet loss can and should be fixed anywhere, regardless of where the server is. Btw, less hops means less chance for packet loss as well.

Or, perhaps, it's color-coded based on location - 80-87 (my normal range) was pretty low ping for my location, so it always showed green, whereas it wasn't good ping given your location, so when you see it now it shows yellow. Which would make it something that needs to be shifted in the next client patch, or in a hotfix.

Does anyone know how this thing works? For me it shows yellow whenever its above 61, ive played several games on the new connection and its been the same. You seem to be guessing at how it works, so am I.

Because they feel that normalizing the region to what the midwest was experiencing - while still keeping a single, unified community - is more than acceptable. It's not about queue population. It's about a centralized community.

This was their main goal? I dont see how dynamic server picking would be any less unified, aside from you would get matched with strangers from your area when convenient instead of strangers from all over? You could still play with any friend in North America, you would still be in the same ranking system, you could still chat. Just if you were playing in an optimal situation youd have a better ping, if you were paying with others that were farther away you would have a central ping. Tell me how the centralized community would change? The ONLY place I could see it changing would be high ELO would maybe not play with as big of a variety of people when optimal ping circumstances were available. But that is like less than 1% of the player base.

JubbinautAug 27, 20151 votes

{quoted}

I wasn't handwaving it, I was saying packet loss can and should be fixed anywhere, regardless of where the server is. Btw, less hops means less chance for packet loss as well.

Yes, it does. Which is a particular benefit here - most users will require fewer hops to reach the servers.

Does anyone know how this thing works? For me it shows yellow whenever its above 61, ive played several games on the new connection and its been the same. You seem to be guessing at how it works, so am I.

I am. But, as I said, it doesn't really matter how it works. My point remains - to validly claim that the client recognizes 61+ ping as "bad", it must show that universally. And it does not - as I said, I have always had 80+ ping, which my status bar shows as green.

If your claim was accurate, it would still be showing yellow for me after the change, since my average ping now is ~65.

This was their main goal? I dont see how dynamic server picking would be any less unified, aside from you would get matched with strangers from your area when convenient instead of strangers from all over? You could still play with any friend in North America, you would still be in the same ranking system, you could still chat. Just if you were playing in an optimal situation youd have a better ping, if you were paying with others that were farther away you would have a central ping. Tell me how the centralized community would change? The ONLY place I could see it changing would be high ELO would maybe not play with as big of a variety of people when optimal ping circumstances were available. But that is like less than 1% of the player base.

Fully dynamic server picking is out of the question any time soon. It would require a complete overhaul of the client and probably related software. Which, as I said, is something they're working on, but nowhere near ready for. In the meantime, the server change acts as a general improvement.

I'd love to see them using dynamic server selection, but frankly, it's just not on the table for the forseeable future. Not because they're lazy, but because their old code is - to be blunt - so bad that it's unworkable. Which is why they're working on fixing that and making it more scaleable.

KraithorAug 27, 20151 votes

I am. But, as I said, it doesn't really matter how it works. My point remains - to validly claim that the client recognizes 61+ ping as "bad", it must show that universally. And it does not - as I said, I have always had 80+ ping, which my status bar shows as green.

Didn't say it was bad, said it wasn't optimal. It shows yellow for me, that is what I was basing it off of. I thought it was universal, now I don't know how it works.

Fully dynamic server picking is out of the question any time soon. It would require a complete overhaul of the client and probably related software. Which, as I said, is something they're working on, but nowhere near ready for. In the meantime, the server change acts as a general improvement. I'd love to see them using dynamic server selection, but frankly, it's just not on the table for the forseeable future. Not because they're lazy, but because their old code is - to be blunt - so bad that it's unworkable. Which is why they're working on fixing that and making it more scaleable.

And you have hit the point of my argument, that moving the server was the laziest possible solution. It is lazyness, to add that feature to the client is NOT that hard, its honestly setting up multiple servers, and having either a server which routes you to the best queue at the moment, or the client code could guess and route you. Riot has been acting like these are extreme engineering challenges. Even if the code is that bad, it doesn't take years to write these upgrades into the client, it just doesn't. I have been developing software for a long time, I have and still do work at much larger companies than Riot, including Apple. The speed at which they develop features is crazy slow.

My entire point was dynamic server selection is the way to actually resolve this, and the server move was the laziest possible way to solve it short of doing nothing at all. There is no reason adding a dynamic server solution should take a team more than a month to get up and running to a PBE testable state, I don't care how bad the client is. There is no reason they couldn't have rewritten the client over multiple years. Saying it isn't lazy is just making excuses for a lazy company.

JubbinautAug 27, 20151 votes

{quoted}

And you have hit the point of my argument, that moving the server was the laziest possible solution.

Except that it's not a matter of laziness. It's not "Eh, that seems like too much work, let's just do something else instead."

It's, "Hm. Rebuilding our entire code-base from basically scratch seems to be taking a long time. Let's see if we can do something else to help."

It is lazyness, to add that feature to the client is NOT that hard, its honestly setting up multiple servers, and having either a server which routes you to the best queue at the moment, or the client code could guess and route you.

Which assumes that the client code can handle that. Which - considering that it's years out of date, and was pretty incredibly bad when it was first coded - it can't. I doubt their server software can, either, for that matter. They planned to support something like 20k players. Total.

They now have 27 million unique players per day.

Riot has been acting like these are extreme engineering challenges.

That's the thing - it's not that it's "extreme engineering challenges." It's that they set themselves up to fail. They built a system that cannot support what they have to support now - and instead, they need to basically redesign everything to make those kinds of changes.

Which they're doing! Albeit slowly. But slow != lazy.

Even if the code is that bad, it doesn't take years to write these upgrades into the client, it just doesn't.

That assumes you want to keep the client. They don't. They're not "writing upgrades." They're redesigning it. More-or-less from scratch. Because it's actually easier than trying to keep everything playing nice as you change things piecemeal.

I have been developing software for a long time, I have and still do work at much larger companies than Riot, including Apple. The speed at which they develop features is crazy slow.

And? Any idiot could have told you that. There's literally thousands of people that frequent these boards that could tell you that. And no one argues with them. Because it's so self-evident that, frankly, it doesn't need stating.

But so what? How does that change the facts of this situation?

Saying it isn't lazy is just making excuses for a lazy company.

Here I thought that hard work put towards developing lasting, sustainable, and growth-oriented solutions was rather the opposite of lazy.

Obviously I have problems understanding English, because I didn't realize that "did not complete project by the timetable I projected" was the official definition of the world lazy.

KraithorAug 27, 20151 votes

Except that it's not a matter of laziness. It's not "Eh, that seems like too much work, let's just do something else instead."

It's, "Hm. Rebuilding our entire code-base from basically scratch seems to be taking a long time. Let's see if we can do something else to help."

I just don't believe its that hard no matter what their code base is like, its a horrible excuse, they have had YEARS to work on their code base if it is really that awful. The amount of code to do dynamic server selection is extremely low, maybe lazy isn't the right word, inept is.

And? Any idiot could have told you that. There's literally thousands of people that frequent these boards that could tell you that. And no one argues with them. Because it's so self-evident that, frankly, it doesn't need stating.

But so what? How does that change the facts of this situation?

You are saying they wrote horrible code, you agree they are extremely code to do ANYTHING that touches the client, you agree its been years. Lazy probably isn't the right word, its bad management, and probably cost. I think the company doesn't care about long term issues and only cares about short term goals. That is what I meant by lazy. They can say all they want that they are working on a new client, but after years of avoiding pretty simple fixes, it doesn't add up.

You also said they don't work on upgrades, but thats not true, they have upgraded the client in many ways, ie champion masteries, new champ selection additions, new match history. I don't see how dynamic server selection is any larger of an engineering tasks than most of those, especially the match history stuff.

Here I thought that hard work put towards developing lasting, sustainable, and growth-oriented solutions was rather the opposite of lazy.

What evidence do you have they are doing this?

JubbinautAug 27, 20151 votes

{quoted}

I just don't believe its that hard no matter what their code base is like, its a horrible excuse, they have had YEARS to work on their code base if it is really that awful. The amount of code to do dynamic server selection is extremely low, maybe lazy isn't the right word, inept is.

They've had years, yes. But they've only recently started doing the actual work necessary. I am by no means trying to imply that they're sterling examples of perfection, but calling them lazy is more than a little deceitful.

You also said they don't work on upgrades, but thats not true, they have upgraded the client in many ways, ie champion masteries, new champ selection additions, new match history. I don't see how dynamic server selection is any larger of an engineering tasks than most of those, especially the match history stuff.

I didn't say that at all. But thanks for putting words in my mouth. And there's rather a massive difference between "overhaul the basic functions of our systems" and "add some extra bells and whistles that our tools already allow for."

What evidence do you have they are doing this?

They've posted about that repeatedly. The Riot Pls post talks about the tech debt they've put themselves into because their previous work wasn't designed with any kind of consideration towards sustainability or growth.

"When we first started working on League, our optimistic target was to support 20,000 players and, in the years before launch, we built a scrappy platform to do just that. It goes without saying that now we need a very, very different architecture to support 67+ million players, and the act of revamping that work without tipping everything over is what we’re talking about when we say we’re paying off our tech debt."

It also talks about how they're working towards remedying that situation.

"rather than just running around putting out fires with our engineers, we’ve started working on things that set us up for cool, sustainable (hopefully fire-proof) projects in the future."

Now you can talk about how you don't approve of their approach. You can talk about how you think it could be done better (though without intimate knowledge of the manpower available, the actual workload itself, and the status of the project, to do so would be more than a little presumptuous). But to sit and dismiss very valid mitigating factors and follow it up by questioning their claims is ridiculous.

The simple fact is, you don't know the situation they're in. You don't know what their codebase looks like. You don't know what it is or isn't capable of supporting. So to call them lazy - or even inept - is a waste of a decent insult.

NiyumiAug 27, 20151 votes

You call them lazy, I call them greedy. Riot is a company that runs an F2P game. From what I've seen, F2P companies will do just about anything, cut any corners, just to save a nickel in the end -- and Riot is no different. If they did the proper thing, and split the servers, it would cost them money to code in all the new things, it would cost them money to keep both servers up and running, and so on.

I'm not really surprised Riot took the easy way out that solves nothing in the long run.

KraithorAug 27, 20151 votes

I do agree that cost probably played the biggest role in their decision.

AlsoJinxBotAug 27, 20151 votes

Completely agree with everything you said kraithnor. I live fairly close to portland and use to get an average of 19-26 ping, i'm currently at 65 which is by all means playable. Now i'm not engineer nor do i know the cost of such large servers but besides server hosting costs, why exactly would it make sense not to make 2 servers? Portland one is obviously still active and there, just not being used and the Chicago ones are already up and being used, so why not just utilize both servers?

Also, i feel like you hit it pretty spot on about the whole client situation. I read awhile back that simple things such as naming champions like for example Quinn was supposed to be named "Quinn and Valor" but issues within the code wouldn't let them. I feel like they defiantly aren't "lazy" but they are very horrible at prioritizing things. All that work that went into the 5 recent project skins could of went into work on making a new client, which we should of got more then 2-3 seasons ago. I don't know how much work goes into making a new client, but i know it defiantly shouldn't take this long. I love their game and the content it has but they really need to get their stuff together and get some stuff done.

JubbinautAug 27, 20151 votes

{quoted}

All that work that went into the 5 recent project skins could of went into work on making a new client, which we should of got more then 2-3 seasons ago.

And what use do you think the concept artists and graphic designers would be at building a new client?

AlsoJinxBotAug 27, 20151 votes

Never said they would be building a new client, just saying 5 new project skins is not priority.

Also yes they did build it on a scrappy platform and yes they are paying off tech debt, but at the same its been nearly 6 years since this game released and we still have no new client. id say 6 years is more than enough time for a new client.

JubbinautAug 27, 20151 votes

{quoted}

Never said they would be building a new client, just saying 5 new project skins is not priority.

I figure making skins is probably a priority for the skins team - since it's their job, and since it's likely that most of them don't have the overlapping skills necessary to prioritize something else.

Which is rather the point. If those guys wouldn't be building a new client, what else would they be doing? Because if they weren't doing that, and they weren't making skins, there's only two options left, really: get paid for doing nothing, or get suspended until the client's done. Neither is a good option for them or for Riot.

Also yes they did build it on a scrappy platform and yes they are paying off tech debt, but at the same its been nearly 6 years since this game released and we still have no new client. id say 6 years is more than enough time for a new client.

Sure! If, y'know, they were working on a client all that time. Which they weren't. They've only recently (within the last year) really genuinely begun work on it.

Should they have been working on it before? Yes. Does that mean that they should - or even can - pull an "all-hands-on-deck" to build a new one now? No, of course not.

I'm not trying to make excuses, here. This is definitely something they should have addressed already. But what's done is done - the situation as it is is more important than berating them over mistakes they've already owned up to. Nor am I trying to imply that things are hunky-dory because they're working on it now - only that we should give them the opportunity to do it - or not - before we start judging their work.

KraithorAug 27, 20151 votes

I'm not trying to make excuses, here. This is definitely something they should have addressed already. But what's done is done - the situation as it is is more important than berating them over mistakes they've already owned up to. Nor am I trying to imply that things are hunky-dory because they're working on it now - only that we should give them the opportunity to do it - or not - before we start judging their work.

Berating them is really all we can do, and point out that they are BSing with their blog posts. You could have said the same things about the client last year, and nothing has changed except match history(which was a welcome addition but i'd rather have a good ping than a nice match history). Where have they actually owned up to mistakes? They wrote a whole blog about the client and how people didnt understand that sure someone could write a better client by themselves(which someone did), but that person couldnt do that and also make it scale. Thats not true, they don't own up to anything.

JubbinautAug 27, 20151 votes

{quoted}

Berating them is really all we can do, and point out that they are BSing with their blog posts. You could have said the same things about the client last year, and nothing has changed except match history(which was a welcome addition but i'd rather have a good ping than a nice match history).

I could have, except they didn't start really working on it until last year. Specifically, they started real dedicated work towards the end of last season.

Where have they actually owned up to mistakes? They wrote a whole blog about the client and how people didnt understand that sure someone could write a better client by themselves(which someone did), but that person couldnt do that and also make it scale.

Never saw that one. Do share!

I saw this one, though, where they talk about how their lack of foresight has put them in a pretty shitty position.

Thats not true, they don't own up to anything.

Nah, you're right. Saying, "we were totally unprepared for this, and we really should have been, and we're really only just now starting to recognize it, so here's what we're doing to fix these problems we left for ourselves" isn't owning up to anything at all.

Except that's exactly what it is. By definition.