A Statistical Analysis of how Matchmaking is Unfair. Should the Team ELO or Map Should Be Balanced?

Neblo·5/26/2016, 3:20:24 AM·199 votes·26,503 views

TL;DR - Win rates for games with evenly ranked players is 60% in favor of blue side, so riot's matchmaking algorithm statistically puts better players on red side to even out the win rate. I find that to be a poor solution, and would prefer to play evenly ranked players with a map disadvantage (although fixing the map or coming up with new and creative solutions would be ideal).

Maybe I would have better luck on Reddit, but my focus is letting Riot know my concerns not just the player base.

My analysis consisted of 41,899 52,000 Silver to Diamond games during the 6.9 patch. When I refer to ELO I am using the formula that LoLKing uses (I posted my code in a comment below along with a few graphs that were requested). It is a number that represents what division and tier the player is in. For example, A Gold III player will have a higher ELO than a Silver IV.

My results show that if two teams are nearly even in ELO scores then the blue team wins 60% of the games. Before the argument of, "but LoLKing says red and blue's win rate is 50/50" comes out let me make sure that we all are on the same page. Yes, the win rate of blue and red is 50/50, but when players are evenly ranked in ELO scores blue has a substantial advantage. The fact that the blue and red win rates are 50/50 suggests that the algorithm is artificially stacking red team with better players to keep the balance. This can be further substantiated in that my analysis shows that ~65% of the time red team is given the higher ELO team.

If you still don't understand here is a simple example: A soccer field has on goal that is much larger than the other. Obviously it is easier to score in a larger goal so one team will have an advantage. Riot's solution seems to be to put the best players on the side with the large goal hoping it will balance out the fact that they will be shooting on the smaller goal. Unfortunately this is very flawed logic.

This is flawed because you cannot prove that you are actually balancing the matches rather than just rigging them. I could get 50/50 blue/red win rates by putting 5 diamond players on one team and 5 bronze players on the other and alternate which side the diamond team is playing on. This would easily artificially create the illusion that the sides are balanced when they are not.

Using a support vector machine (SVM) learning algorithm I was able to predict who was going to win a match ~64% of the time solely based on the ELO of the players and the side they were playing on.

This suggests one of two things may be going on:

  1. Riot's match making system needs major work as it doesn't work very well
  2. Riot's match making system is (perhaps unintentionally) stacking teams to give the appearance that the map is balanced.

If Blue side win's too many games does the algorithm make sure that it goes back to 50/50 by increasing the deviation of elo placed on the red team as compared to blues'?

I think it goes without saying that this is a broken system.... So why does Riot just not fix the map? Well I think they have tried. Rift herald, brush changes, etc. were all changes in attempting to balance the map. It is probably expensive, time consuming, and may cause community backlash (probably not as much if people realized how bad it currently is).

I personally would much rather play evenly ranked teams, even if I had a disadvantage due to the map. In fact it is likely a more balanced solution as well. Like I said, I can predict who will win 64% of the time when given the ELO of each team, but evenly matched games give only a 60% accuracy of who will win. This would imply that it may be possible to gain a 4% better 'balance' to the game by allowing the map to be unbalanced, but ensure teams are even as possible. Since it is impossible to always have perfectly even teams, you could potentially further improve balance by always putting the higher ranked team on the red side.

The statistic that makes me the most angry is that when blue side has a significant advantage in ELO, they have a 70% chance of winning. What is more upsetting is that this accounts for around ~10% of total games. That means statistically 1 out of 10 games you are heavily disadvantaged. Why was that team placed on blue if the odds were known to be that bad? Answers I don't have.

What do y'all think? Is this how Riot should be balancing the win rates? Should the map be tweaked? Should it be completely redone?

note - I remade this post because I messed up the poll and could not find a way to change it

edit - let players know that I have a graph below edit - tweaked the TL;DR and fixed the number of games used in analysis

138 Comments

Neblo5/26/2016, 4:35:40 AM28 votes

Someone asked on Reddit for some graphs and about the ELO calculations so I figured I would post it here too:

I have a few photos from an old data set i made a year or so ago available on hand. I'll try to put together some new stuff if people really want graphs of the data I was citing in my post, but they are very similar so I don't feel like there is much harm.

Red vs blue win rates compared to ELO deviation of each team from the mean of all players in the match

https://s33.postimg.org/c8ns5pd33/figure_5.png

The important thing to take from this figure is that 1. if you have a team with a much higher elo you are vastly favored to win. 2. blue team's win rate is 60% when elo is even on both teams.

Number of games (Y-axis) against the ELO deviation of each team from the mean of all players in the match (X-axis)

https://s33.postimg.org/vmbr3060f/figure_6.png

The important thing to get from this graph is that red plays more games with with a positive ELO deviation mean... or simply that the better team usually is placed on red side.. and you can easily see why when you look at the win rates from the first graph.

Here is the ELO calculating function I used (it is deleting the white space in front of my code, sorry I don't know how to fix that):

def _calcElo(self, tier, division, lp): """Calculates and returns a player's elo""" elo = 450 if tier == 'CHALLENGER': R = 6 elif tier == 'MASTER': R = 6 elif tier == 'DIAMOND': R = 5 elif tier == 'PLATINUM': R = 4 elif tier == 'GOLD': R = 3 elif tier == 'SILVER': R = 2 elif tier == 'BRONZE': R = 1 else: R = 0

elo += R \* 350

if R != 6:
	if division == 'I':
		elo += 4 \* 70
	elif division == 'II':
		elo += 3 \* 70
	elif division == 'III':
		elo += 2 \* 70
	elif division == 'IV':
		elo += 1 \* 70
	elif division == 'V':
		elo += 0 \* 70

elo += 0.5 \* lp

return elo
Goosetard5/26/2016, 1:18:25 PM10 votes

This is very interesting to me. Some questions:

  • Have you checked to see if your Elo AND side predictor is statistically significantly better than just using Elo?
  • Have you checked to see if the advantage with one side has to do with the map and not with the pick/ban order? (analyze blind pick normals or old teambuilder games)
  • Assuming one side is better than the other side, would you prefer to not give one side some type of advantage to compensate?
  • Assuming the advantage is in the design of the map, do you have a suggestion to balance it?
  • Can you analyze the data by lane? For example, does it just suck super hard to be on the easily gankable top side, tops usually lose on that side, and that's usually the cause?

This is great stuff man. I love this shit.

Woook3r5/26/2016, 3:45:16 AM8 votes

Dont understand why they dont just mirror the map...

CoronaGulp5/26/2016, 6:02:11 AM7 votes

Can we get Riot to comment on this?

Summonerrr 15/26/2016, 5:35:43 PM7 votes

I just wish Riot would pony up and actually make an individual ranking that is statistically valid in a team setting, and stop trying to fit ELO into this slot when it will NEVER be statistically valid.

Trying to predict an individuals outcome in a team game based on solely individual metrics, when the team metrics are the ones deciding the outcome, IS STUPID.

ZombieWizard PHD5/26/2016, 4:40:45 AM6 votes

I would trust that Riot is aware and has many highly trained teams and studied professionals who have thought this over. I imagine they have thought this though. Statistically speaking, skill difference can compensate map differences to create an equal chance of both teams winning, giving red side a challenge and blue side a handicap.

I am actually the first person to vote yes. Thoughts?

chessman5/26/2016, 6:47:27 AM5 votes

This is flawed because you cannot prove that you are actually balancing the matches rather than just rigging them. I could get 50/50 blue/red win rates by putting 5 diamond players on one team and 5 bronze players on the other and alternate which side the diamond team is playing on. This would easily artificially create the illusion that the sides are balanced when they are not.

I don't think this is an accurate comparison. Riot isn't altering the team elo's to create a 50/50 winrate between blue and redside, it's doing it to create a more even probability of either side winning in every individual game. The winrate is 50/50 as a result because the games are more even as a result. More even than the games would be if riot did not do the "artificial" balancing.

Vekkna5/26/2016, 10:04:01 PM5 votes

I was skeptical, so I checked to see how this may have affected my own games. I went back through my match history for 26 games with exactly 50% win rate.

Red Side 15 games out of 26, or 57.6% of the time W/L is 5:10 for a 33.3% win rate

Blue Side 11 games out of 26, or 42.3% of the time W/L is 8:3 for a 72.7% win rate

All of the games I won on red side were against lower ranked opponents. Almost all of the games I lost on red side were equal or with a slight advantage to blue.

I remember posting a very long time ago about how I thought matchmaking essentially assigned players to a "loser queue" to equalize their win streaks. At the time, I suspected the loser queue paired you with other players on a losing streak or a high number of reports/afks. Apparently that's true, but it works through map side. Interesting.

Norebert5/26/2016, 6:08:22 AM5 votes

This is somewhat of a tangent, but I've considered Riot's matchmaking/ranking system to be flawed for years now. Back in late 2012 they made changes to match players not only based on ELO, but also on the number of games that player had participated in.

The problem this creates (IMO) is one of Local Maximums when looking at player rankings.

Example:
Player A has a 1200 ELO rating and 1000 games played, the system matches him against other players with around 1000 games played.

Player B has a 1500 ELO rating and 50 games played, the system matches him with/against other players with around 50 games played.

The problem as I see it is that player A might actually be BETTER than player B in the empirical sense! However, because the system is separating experienced accounts from newer accounts, there's no way to actually know for sure. This creates a weird system where it's possible for 1200 ELO > 1500 ELO.

Infact, if you think about it... a player that has a long-standing account might actually find it easier to climb to a higher ELO bracket by leveling up a new account and getting queued against other newer accounts.

I just think it's weird. I understand why they did it - they wanted to create the most 50/50 chance of winning balanced games they could, and there IS a direct correlation between games played (+ champs unlocked / etc) and skill, but I feel separating newer accounts from older ones makes rankings super wonky.

A source for what I'm referring to:
http://www.surrenderat20.net/2012/11/matchmaking-factors-in-number-of-wins.html

Lipton5/26/2016, 6:30:47 AM5 votes

Let's ask why blue wins more then ? My guess is that's it's not linked to the In-game objective but rather the Camera. People have harder time playing while facing the camera. You can't fix this problem without a major rework of the game engine. Also that would mean that purple side botlane would see itself playing top.

Now here comes the E-sport part. Rivington will have the brain twisted when he will have to call a part of the map. Bot & top being different depending which side you take it from.

ZT Xperimentor5/26/2016, 1:11:04 PM5 votes

They could just turn the camera around for red side, but that would take effort. I agree matchmaking has been butchered for years though.

EfficientDynamo5/26/2016, 5:40:29 AM4 votes

Interesting analysis. Have you run these same calculations on master/challenger matches only, or at low ELO only? It would be interesting to see if there was a greater advantage given to teams at varying levels of skills. In theory that could give evidence that this imbalance is intentional, as the matchmaking formula might need to give much smaller advantages to teams at high ELO's in order to "balance" things out.

I was Lupxel5/26/2016, 12:51:47 PM4 votes

Ok... now I will tilt if I get red side. And by pressure I will tilt if I am blue side

Darkslayer855/26/2016, 12:51:08 PM3 votes

602 votes VS 38 comments. Wow.

Troy2426215/26/2016, 5:04:12 PM3 votes

Interesting. I'm glad someone skilled such as yourself was able to compute the statistics and highlight the problems. Having said this, if we assume your premise is accurate, and that there is indeed a problem, how would you propose it be fixed? Surely simply doing away with team stacking would still leave considerable amounts of discrepancy in skill between teams.

I do agree, though, that deliberate imbalances to offset a hypothetical map advantage is a band-aid solution.

Yvaelle5/26/2016, 5:53:46 PM2 votes

Switch the camera and some of the gank paths, and then you wouldn't need to have an Elo discrepancy to compensate.

Minor map adjustments would really help too, example - take a look at this map:

http://static4.gamespot.com/uploads/scale_large/1527/15278763/3043761-ogImport-18374-fullmap2.jpg

That's all the kills at Worlds 2014 (I couldn't find a more up to date one) - a Red dot means a kill by the Red team, a blue Dot means a kill by the Blue team (slightly confusing I know).

The deep sides of both maps are going to look pretty much identical - the spawns and early jungles - but look at some of the minor differences around the river.

Baron pit: Blue tends to score a lot more kills in the Baron pit, because Red side can't easily disengage - while Red can - this leads to a lot of Red side players getting backed into the Baron and then being trapped and dying there - rather than being squished against the River wall: Blue side just backs up toward their Blue buff.

Mid: Blue Side tends to get a lot more kills in the River just south of Mid lane - this is almost certainly due to easier warding paths for Blue side mid/jungle/suppport - since they have an alternate 'safe' route behind Dragon - while Red side has to back up all the way around their Blue buff to get wards in that area safely - and then sneak through the narrow river-entry bush just south of Mid: look at all the kills Blue team gets by controlling the river just south of Mid: a massive discrepancy - almost certainly due to better vision and a bush leading into that chokepoint between Red mid and Red's Blue buff.

Bot Lane: Blue team seems to get slightly more kills around Red's outer turret. This is likely due to easier access from the Blue jungle to the Blue outer bot tower: it's easier for Blue jungle and mid to rotate to help defend the bottom Blue tower, and harder for Red jungler/mid to rotate safely (not straight down the river through wards / ganks) to help defend Red bot lane's outer tower. Additionally / alternatively, Blue team has alternate means of secondary egress in the event of a tower dive on their outer bot tower - easier access to their jungle. Red team may not have as easy a time getting into a bush for safety - since it's so easy for a Blue jungler to get into the Red-side tribush - and no available similar position so near to the Blue outer bot tower (their tribush is over by dragon, or the bush by rock dudes).

Recommendations:

  • doing something with the camera seems the obvious choice, LoL is a very low requirement game, it could stand to increase the requirements some to render both directions
  • Baron pit: potentially put the baron in the middle of the river with ~mirrored access from both sides
  • South of Mid: adjust the Red side walls slightly around Red's blue buff to make it faster and safer for Red to rotate and ward the areas south of Mid lane
  • Bot lane: give red faster/safer access to bot lane rotates, potentially by thinning out the rock wall attached to Red's tribush - between frog and Red's blue buff

Crazy idea: think about putting all turrets in the middle of their lanes - this would likely help Red more than Blue due to the generally easier ganks on Red side (in both Bot and Top lane outer turrets).

Quepha5/26/2016, 6:43:33 PM2 votes

On the one hand, this feels like a really artificial way to even out the winrates for each team and it also feels like it would obscure certain game skills (learning to handle the map while you're on red side isn't really as important compared to knowing how to snowball a lead because you're probably better than your opponent)

On the other hand, a 60/40 difference in winrate is really significant and if teams were matched evenly that would heavily incentivize dodging in champ select as soon as you see you're on the side with a 20% winrate deficit.

I want to know if it looks like the matchmaking system seeks to achieve an even matchup based on skill level before sides are chosen or based on success rates after sides are chosen. That is, does the game prefer to find teams of equal skill level, then give the advantaged team by MMR the red side to compensate if there is a disparity (and distributing sides randomly if skill is approximately equal), resulting in more games where team skill is even but their odds of success are actually uneven due to side advantage? Or does it instead seek to find teams at that apparent sweet spot of a 20 MMR difference and give the advantaged team red side so teams are usually slightly different in skill level but have equal odds of success after side advantage is factored in?

Stillname5/27/2016, 5:28:21 PM2 votes

Both Dota 2 and LoL have uneven winrates/uneven elo ratings on "left side" vs "right side" but HoTS doesn't because the map goes left to right. The problem is very obviously the map tilt but they can't change that without a LOAD of community... lets call it "feedback."

So what is the solution? For a little while League red side minnions used to hit slightly harder but that was a "hidden power" so they scrapped that and just put players who were slightly better on red side in order to make the game win rate more even.

chëw5/26/2016, 10:26:25 AM1 votes

[deleted]