Boards›Gameplay

Response Bias in Dynamic Queue Survey

Lucifer··75 votes·3,897 views

http://i.imgur.com/vydg02r.png

The way Riot's current survey is worded is honestly confusing and hard to decipher. I had to do a double take to see what the "correct" answer would be for myself. A survey like this just encourages players to say "No, I enjoy queuing with friends!" which is not enough to actually data mine if players prefer one feature over another. Those same players who are saying they enjoy queuing with others may also answer that they hate dynamic queue if given a separate survey. It's easier for players to just "agree" with the statement in its currently worded form.

The question should honestly just be something like this:

Sample Question: "Which season of ranked play is more enjoyable to you ONLY in regard to number of players allowed in a premade group? This season (dynamic queue's 1-5 players), or previous seasons (solo/duo queue's 1-2 players).

Sample Answer: This Season of Ranked is More Enjoyable Previous Seasons are More Enjoyable

While this survey would not include those who have never played ranked in previous seasons, this would be a lot more valid for data collection to see what people honestly like better.

As a side note, I have not heard of any of my friends in platinum+ receiving this survey, while quite a few of my lower elo friends have. It's important to section this data off by elo, because a bronze player perception of dynamic queue should not have the same weight as a platinum-challenger player.

Edit: Edited the OP to reflect some good feedback. Didn't consider changes like promo helper.

97 Comments

Meriipu50 votes

itis pretty ironic that your suggestion is just as biased, since it immediately draws in associations to seasonal differences other than the dynamic queue boogeyman

in fact it is more biased, the original one is just awkwardly worded

Lucifer10 votes

{quoted}

itis pretty ironic that your suggestion is just as biased, since it immediately draws in associations to seasonal differences other than the dynamic queue boogeyman

How is it biased? It asks which one you prefer. Older (solo/duo) or Newer (dynamic). There's no bias within the question at all, and it doesn't encourage one response over another.

DrCyanide27 votes

Which season of ranked play is more enjoyable to you? This season (dynamic queue's 1-5 players), or previous seasons (solo/duo queue's 1-2 players).

This season also added the Promos helper. Should I vote for this season because I like that? Should I vote last season because I liked the balance better? Your actual question (which season was more enjoyable) is very different from the response you're looking for (do you enjoy playing with/against Premades in Ranked).

kile1476 votes

Riot's way of asking is very direct about what factor they are asking us to judge our experience on, but it has an inherent bias in the question.

OP's method of asking does not have a bias one way or another but does not sufficiently isolate the Dyna Que vs Solo Que question. It implies that you should only take those aspects into account, but the actual question being asked is to compare the seasons which would mean this season would actually have an advantage because of improved champ select and other factors you noted.

I think the ideal way to word this would be:

"Please rate your agreement: I would prefer a ranked ladder that only allows queuing up in groups as large as you and one friend, as opposed to the current system of queuing up in any group size. (Dynamic Queue Removed)"

I believe this is a fairly unbiased and clear way to word this question while retaining their rating format.

Lucifer3 votes

{quoted}

Riot's way of asking is very direct about what factor they are asking us to judge our experience on, but it has an inherent bias in the question.

OP's method of asking does not have a bias one way or another but does not sufficiently isolate the Dyna Que vs Solo Que question. It implies that you should only take those aspects into account, but the actual question being asked is to compare the seasons which would mean this season would actually have an advantage because of improved champ select and other factors you noted.

I think the ideal way to word this would be:

"Please rate your agreement: I would prefer a ranked ladder that only allows queuing up in groups as large as you and one friend, as opposed to the current system of queuing up in any group size. (Dynamic Queue Removed)"

I believe this is a fairly unbiased and clear way to word this question while retaining their rating format.

Yah, I agree. I didn't consider changes like promo helper, so it would be useful to isolate what we want to know about (dynamic preferred or not)

Kadavik2 votes

I was thinking exactly the same thing.

MISTAFUNKTASTIC17 votes

The question is not that hard to read imo.

"Rate your agreement: I would be upset if" (This describes your emotional response to the proposed action)

"you could not que up with more than 1 other player in ranked" (This is the proposed action)

"(Dynamic que removed)" (This is the proposed action restated)

Critmaster Garen7 votes

{quoted}

The question is not that hard to read imo.

"Rate your agreement: I would be upset if" (This describes your emotional response to the proposed action)

"you could not que up with more than 1 other player in ranked" (This is the proposed action)

"(Dynamic que removed)" (This is the proposed action restated)

its not hard to read, but its suggestive.

the question itself already has a negative tone to it so people tend to vote "agree" more often.

DarkBillyBoy3 votes

dude listen here how things are. some1 starts reading the question, but it its kinda hard to read it, so when they reach the end of it, they view a sum-up given from riot : (dynamic queue removed), so here is what they have in their mind now : "do you agree dynamic queue to be removed ?" And so they answer YES STRONGLY AGREE. which is the exact opposite.

zlumpy15 votes

They always make their surveys biased towards the outcome they wanna hear. I don't get why they even bother with a survey for stuff like this when a large number of people are so vocal about it on forums. It's pretty obvious people don't care for dynamic que.

DrCyanide24 votes

A larger group of forum users voted their displeasure at Tyler1 being banned than have ever voted for a single thread on Dynamic Queue. Clearly votes here don't (nor should they) mean anything.

zero35612 votes

It's also consistency. Solo queue threads have been around for a while, and many got and still get attention. Tyler1 came and went on the boards

Xonra4 votes

A lot of that was also trolly. He was also spamming links to the forum and people that don't normally gather here came.

You really seem to just want to disprove the OP. It seems by your responses you are VERY clearly biased towards a specific side, and are trying to disprove the OP at any cost with arguments that are near irrelevant (like bringing up Tyler1).

What's more sad, is you would make a perfect politician, bringing up off-topic comments that are sure to get upvotes, and attaching them to your point.

DrCyanide2 votes

You really seem to just want to disprove the OP. It seems by your responses you are VERY clearly biased towards a specific side, and are trying to disprove the OP at any cost with arguments that are near irrelevant (like bringing up Tyler1).

How so? I pointed out a flaw with the OP's survey question idea and they altered it accordingly. This response wasn't directed at the OP at all, but at another user who thought "a large number of people are so vocal about it on forums" was meaningful, to which I brought up a counter example (Tyler1). I could have just as easily brought up the requests for URF to be a permanent game mode, but those are a distant memory by now and far harder to show without a screenshot.

Am I biased? Probably. I've played maybe a grand total of three ranked games with a group larger than 2, but I still want the option. I think it's safe to say that everyone who enters a Dynamic Queue/Solo Queue thread these days is biased though.

As for being a politician, I couldn't. I'm far more eloquent with a keyboard than I am with my mouth, especially when I've had time to think things through. I also think that having career politicians is probably not the best choice for governments, but that a jury duty style system would work better. It just seems odd to me that someone who knows nothing about teaching (the politician) tells teachers what and how to teach...

Mandang01 votes

You know the whole "vocal minority" complaint only works if you can prove the majority disagrees, right?

DrCyanide1 votes

http://boards.na.leagueoflegends.com/en/c/gameplay-balance/d5WLfgiq-fact-check-around-11-of-the-global-player-base-is-really-pissed-at-riot-over-dq

It would be one thing if the feedback that we got from most players was that they don't like Dynamic Queue, but that's not what we're seeing. It's a segment of about 11% of the global player base that's really pissed at us because of the switch.

Cheekynorth2 votes

I'd like to mention that Riot's statistics are almost never never the actual stats

DrCyanide2 votes

It's still far more likely to be accurate than a poll attached to some random thread, which is all the anti-Dynamic Queue community has been able to produce.

Hyrum Graff2 votes

<3

Critmaster Garen5 votes

{quoted}

They always make their surveys biased towards the outcome they wanna hear. I don't get why they even bother with a survey for stuff like this when a large number of people are so vocal about it on forums. It's pretty obvious people don't care for dynamic que.

because they can bring the results up as "evidence"

zlumpy3 votes

Yup gotta love bs politics and business practices.

bill842 votes

Wait but don't they want to hear what people actually want since that would make the game more enjoyable for players and therefore make them more money?

Xonra2 votes

Nope.

Sadly not how business works.

The whole idea and way it goes is

  1. Find your conclusion

  2. Do "in-house" research to come to an "informed opinion" among those working with said thing

  3. Sculpt your finding to reflect your already found and wanted conclusion

  4. Do "outside" research and since you are the one with said data, just take of it what you want, so you aren't actually lying when presenting the information. Just because you leave information out, doesn't mean you lie. No one can prove otherwise, so what you present is all that will ever be seen.

  5. Present your "findings" that have now been shaped to reach your conclusion you started with to the higher ups.

  6. Present your conclusion to the "public" with the information they supposedly helped you with, in essence making them think THEY came to the conclusion, not you.

So basically the whole idea is to already know the outcome, but make it seem like the people you are presenting it to came to the conclusion for you, and not the other way around. Riot just didn't do a good enough job with making it seem like WE came to this conclusion.

Dealth12 votes

There is nothing wrong with how Riot worded this question. People just like to complain about Riot.

Kadavik6 votes

Its probably a safety net. In case people vote to keep DQ. Then they can throw on the classic "you cheated!" Playground tecnique.

maikokk3 votes

ya, if anything the wording biased me toward saying disagree even though im a dyn-q fan. because they were very clear on what they meant. no, i would not be upset if it wasnt available. i understand why it wouldnt be a thing after all the backlash

Anotherherolost8 votes

Stop getting caught up in OPs wording. Riot purposely worded this, the last thing you see is "Dynamic Queue removed". Most of the player base will skim it as they log in, see "Dynamic Queue removed" and if they are opposed to dynamic queue, hit "Strongly Agree". If you combine all the idiots that don't read it through with the people that actually read it and like Dynamic Queue they get the answer they want: The community likes Dynamic Queue.

Riot could have easily worded it like this: "I enjoy Dynamic Queue. (Being able to play with 2 or more friends in ranked.)"

Or could have been even more honest: "I enjoy occasionally being destroyed by 3 or more opponents who are all coordinating via voice chat and/or being ignored by 3 or more teammates who are also coordinating via voice chat and could give a shit less about the rest of the team. (Dynamic Queue).

When the results of this come out it will be overwhelmingly in favor of Dynamic Queue due to the wording and they can tell us all to shut up.

Jalkan6 votes

just choose 'strongly disagree'

you know the client surveys are worded randomly for different people to stop bias right

{quoted} As a side note, I have not heard of any of my friends in platinum+ receiving this survey, while quite a few of my lower elo friends have. It&#039;s important to section this data off by elo, because a bronze player perception of dynamic queue should not have the same weight as a platinum-challenger player.

uh high rank means you're good at the game, dynamic queue has nothing to do with balance or gameplay

Divewing6 votes

It's not poorly worded. It's worded exactly as manipulative as they want.

Van Herrington5 votes

lol it's hard to understand for people with a third grade or below reading level.

  • idk what ur talking about, i got the survey on my main which is dia, and both my smurfs which are plat
briss84 votes

A survey like this just encourages players to say "No, I enjoy queuing with friends!"

That is the point of they survey in the 1st place .... to get a baised opinion in their defence so they can say LOOK u guys dont actually want solo Q back our surverys said so! even tho we purposely skewed the question to get you to answer it in the exact way we wanted !.

I work for a survey company that Literally I have to talk to the people about the survey and ask each question ... and its biased as hell. Every company does this crap .... they word questions to subliminally get you to choose certain answers . or just skew questions a certain way. Among other dastardly efforts to get the satisfactory survey results they would like to see.

EROTIC RAID BOSS4 votes

I don't see how its confusing or biased

in fact if anything it seems biased towards SQ

PiVoRx3 votes

I would make it even simpler: What you pefer: Dynamic Queue Solo Queue

Bladerunner77772 votes

Most players don't know that Dynamic Queue means 2-5 man premades (mistaken for New Champion Select)

It was worded fine

Bladerunner77773 votes

The surveys aren't the same for everyone ya dingus

For half of the people it is worded the opposite so the answers aren't biased.

Some people have it worded like "I would be happy" and others worded like "I would be sad". That's the most effective way to get unbiased answers

Stephenizgod3 votes

Your sample question is not very accurate. I could hate Dynamic Queue but enjoy the balance of the game in this season. I could enjoy previous seasons more than this season because of the meta, but really enjoy Dynamic Queue. I could dislike Dynamic Queue but also hate all previous seasons (not sure why this person would be playing League, but this community's stupidity never ceases to amaze me).

A better question would be more straight forward so there is no confusion: Do You Prefer Dynamic Queue Over Solo Queue?

This is a simple "Yes" or "No" question, no bias, no confusion.

Emerald Fang2 votes

They probably purposely do this, anyway. Most players will hit "Strongly Agree" because they didn't read the question right. Then, Riot can come out with "90% of players prefer DQ over soloqueue!"

Pick Strongly Disagree.

DeynaTaggerung3 votes

The question itself is perfectly fine, my only issue is their switch from I to you part way through. It should be "I would be upset if I..."

Dolasaur3 votes

I think it's poorly worded because it has a non-sequitur typo at the start: "I would be upset if you could not..."

I take lots of consumer surveys all the time, and without that typo, it would be perfectly fine. Very easy to understand.

For the "my friends" part, check out Availability Heuristic on Wikipedia or your favorite similar website.

For the "It's important to section this data off by elo, because a bronze player perception of dynamic queue should not have the same weight as a platinum-challenger player"- you shouldn't stick that in at the end. That's extremely opinionated and nuanced. You could have just as easily said "poor people shouldn't have the same weight as rich people" or "black people shouldn't have the same weight as white people". Finally, you didn't even say what the weight was for. It sounds like you're hoping Riot would say "if enough Plat-Challenger people don't like DQ, then we'll remove it!" But presumably you don't know what Riot's going to use the results of that survey for.

briss83 votes

I find it funny that they are only sending that to players who are dynamic que players only ... as I am a solo player and I did not recieve that survey funny as hell.

Xonra3 votes

I was with you until this bs

"As a side note, I have not heard of any of my friends in platinum+ receiving this survey, while quite a few of my lower elo friends have. It's important to section this data off by elo, because a bronze player perception of dynamic queue should not have the same weight as a platinum-challenger player."

Then I just went back and downvoted.

Not because I'm in Silver, but because I'm tired of people thinking they are gods of league because they are above Gold. It has nothing to do with the topic, and it just makes you sound like an elitist a-hole. Playing with friends has nothing to do with what rank you are.

Get over yourself and realize no one cares about your rank or your e-peen.

Lucifer2 votes

{quoted}

I was with you until this bs

"As a side note, I have not heard of any of my friends in platinum+ receiving this survey, while quite a few of my lower elo friends have. It's important to section this data off by elo, because a bronze player perception of dynamic queue should not have the same weight as a platinum-challenger player."

Then I just went back and downvoted.

Not because I'm in Silver, but because I'm tired of people thinking they are gods of league because they are above Gold. It has nothing to do with the topic, and it just makes you sound like an elitist a-hole. Playing with friends has nothing to do with what rank you are.

Get over yourself and realize no one cares about your rank or your e-peen.

People abusing dynamic queue (3-5 teammates) to climb ranked above their skill level has a lot to do with this topic. Besides that, the game gets increasingly more competitive the higher you climb. If you value competitive play (something bronze-gold might not, honestly), then your perception of this system may be vastly different. A lot of users in gold and below simply view dynamic queue as an easy way to get the season rewards by queuing with higher elo friends. Additionally the effect of playing with and against premades is different at different elos. Playing against 3-5 members of C9 isn't the same as playing against 3-5 bronze teammates queued together.

Slythion2 votes

When was this survey even sent out?

Spuky2 votes

So it should read "I would be upset if Dynamic Que was removed".

In which I would Strongly Disagree.

KuroTheHero2 votes

I agree, anyone who's taken a statistics class or has a reasonable amount of logic would be wary of this. I'm fairly disappointed a company as large as Riot that requires accurate measurements of statistics for balancing would either be unable or complacent in creating such a biased question

AwesomeChad2 votes

But its intentionally worded with bias. Whats the name of that guy with the Phd in psychology who left the company? I completely forgot.

The Deckowner2 votes

Lyte

Blanchimont2 votes

I feel there's no way to get a person's opinion in Dynamic Queue through a simple survey with just a few questions. They need at least several and at least open up the potential for an open ended response like they did with the recent survey about viewing LoL Esports in emails. At the end of the survey, they asked you for comments on watching it.

  1. Have you played ranked through Dynamic Queue? Yes No

  2. Do you like the ability to choose your role when you enter queue? Yes No

  3. Do you prefer to play ranked alone or group with others? Play Alone Play With Others

  4. Do you feel that both teams have an even chance to win in ranked games? Yes No

  5. Would you be willing to sit through longer queue times for improved matchmaking? Yes No

  6. How do you feel about the number of bans given to each team? Too many It's fine Too few

Ulriah2 votes

Seems very clear to me, and English is not even my mother tongue.

More from Gameplay

3 Refunds per season, upvote if you want 3 refunds per season.
Gameplay764 comments
7,167
votes
3 Refunds per season, upvote if you want 3 refunds per season.
Gameplay156 comments
3,510
votes
UPVOTE FOR URF
Gameplay606 comments
2,440
votes
Yes plz.
Gameplay456 comments
1,721
votes
3 Refunds per season, up vote if you want 3 refunds per season.
Gameplay114 comments
1,716
votes
When is ANYONE at riot going to address the awful new player experience?
Gameplay634 comments
1,383
votes

More by Lucifer

Yasuo's Windwall Should NOT Stop Taliyah's Ult
Gameplay244 comments
285
votes
I will no longer be using the PBE to test changes
Gameplay101 comments
200
votes
Please do NOT Buff Tahm
Gameplay131 comments
104
votes
Every Yasuo Should Default to Honor 0
Memes & Games9 comments
69
votes
Please Don't Make the Poro Sakura Icons Japan Exclusive :(
Story, Art, & Sound44 comments
68
votes
Concerned about Support Items
Gameplay183 comments
64
votes