The Obvious Bias (MMO research)

Tobold plugged a Newcastle University Business School research project based on the potential beneficial or detrimental effects of playing MMORPGs during off-job hours on employee well-being.  The survey reads like some undergraduate amateur hour where they did not really take time to contemplate the responses (e.g., they don’t even have my degree, grammatical errors), but I can deal with rough surveys on topics I am interested in.  What I cannot deal with and what caused me to immediately dead stop on my progress for their “research” was their opening list of how strongly do you agree/disagree with the following:

1. PLAYING MMORPGs: The following statements concern how important massively-multiplayer online role playing games (MMROPGs) are for you.

  • MMORPGs have created real problems for me, but I keep playing.
  • Sometimes I only plan to play a MMORPG for a few minutes and wind up spending hours in front of it.
  • Playing MMORPGs takes up almost all of my leisure time.
  • I would be a lot more productive if I didn’t play MMORPGs so much.
  • My family and friends get angry and tell me that I play too much, but I can’t stop.
  • I often play for a longer time than I intended.
  • I spend much more time playing than just about anything else.
  • I would spend more time with hobbies if I didn’t play so much.
  • Sometimes I feel like my whole life revolves around MMORPGs.
  • I often think that I should cut down on the amount of MMORPGs that I play.
  • When I can’t play I get restless or irritable.

To be fair later on in separate sections they ask whether you get enjoyment from playing, if you are cheerful when you are playing, when you play you play for yourself, etc. BUT, the damage is done. The tone was set. I believe I have a very healthy amount of computer gaming time that allows me to amuse my wife and play with my toddler and be fully successful at my job; yet, this Monday morning I felt like MMOs were the worst and most unhealthy thing in my life after reading that list. I refuse to complete the survey because it is clearly apparent what result Dr. Savvas Papagiannidis from Newcastle University (U.K.) and Dr. Despoina Xanthopoulou from Erasmus University Rotterdam (Netherlands) wish to have. Look for them later this year on a depressing report coinciding with a gamer suicide or divorce.

it’s all about the small stuff

15 thoughts on “The Obvious Bias (MMO research)”

  1. I completed the questionnaire earlier and I think you may be being a little unfair regarding the balance of questions. For instance, the example you posted is later repeated in relation to work situation, and there are also opportunities to report the positive aspects of our MMORPG experiences.

    I’d suggest that you will bring more enlightenment to this issue by completing the questionnaire honestly than by not doing so at all. In my experience, most scientists are more concerned with peer-reviewed publication than with engineering a good headline, and a poorly designed questionnaire is a great way to ensure your paper won’t be published.

  2. I agree that there are opportunities to put forth good things on MMOs in this survey, and I honestly did not tally the balance of the negative vs. positive questions. However, it is the tone that opening agree/disagree questions that I believe shows complete bias.

    Surveys (and statistics therefrom) are such subjective creatures. This one clearly sets the reader into a feeling of self-doubt or annoyance (thereby coloring further responses). There is no balance.

    It gets the responder thinking about the one time their significant other got mad at raid night (no matter that responder’s norm) or the times that the responder’s parents openly prayed for a “normal” son instead of a gamer dork.

    The survey does not even pretend to keep the responder at an even-keel objective norm.

  3. While you probably have a soothful intuitive reaction to the survey-creators’ biases, I gotta echo unwize and point out that not completing the survey only serves to help your POV be under-reported.

  4. “I would spend more time with hobbies if I didn’t play so much. ”

    If by “hobbies” they mean “Other mmos” then… yes. Yes I would.

  5. I had no problems with it because it had the Strongly Disagree response available. I used that a lot in the survey.

  6. Aside from how the questions flavor the mood of the survey (hey, I enjoy playing MMOs more than I enjoy my job… am I a bad person?) the whole thing is a self-selecting joke. Go look up the 1936 US presidential election and read about the Literary Digest survey for one of the most famous cases of self-selection.

    People who will bother finishing the survey will be overly represented by people who feel they need to defend MMOs or have an axe to grind. The “average” player, who probably doesn’t read blogs or fan sites and who will thus be unlikely to hear of this survey will be under represented.

    So the survey results will be representative of “those who found the survey” as opposed to the MMO playing community.

    I’ll be interested to see the results if only to see how they rationalize their methodology.

  7. @ unwize: “I’d suggest that you will bring more enlightenment to this issue by completing the questionnaire honestly than by not doing so at all.”

    That may be true, but I think he might bring yet more enlightenment by pointing out flawed research methodology. Looking at section 1 as I fill it out, I can’t help but see an obvious bias as Ravious notes. E.g. in a section subtitled “The following statements concern how important massively-multiplayer online role playing games (MMROPGs) are for you”, where are questions such as:

    * I feel MMOs are a valuable source of social bonds for me.
    * I feel MMOs provide me with good entertainment value.
    * I feel MMOs are the hobby I enjoy most.

    Answer: nowhere to be found. Instead, the survey focuses only on the negative factors ~when considering the importance of MMOs~. Whether it includes mention of positive factors in other sectors does not make this survey any less balanced.

    … Having just completed the survey, I can say that there wasn’t any later section introducing this balanced approach. Instead, all the questions focus on a) how much do MMOs ruin my life, b) how do I act or feel when I play MMOs, and c) how do I act and feel at work.

    This was a very biased survey indeed. The conclusions drawn from it will be of little value; garbage in, garbage out. My old research methodology profs would suffer a collective apoplectic fit if they saw this.

  8. I didn’t see the whole survey but from the blog post here, the surveu is obviously treating MMORPGs as addictions. Which they obviously can be. However I do agree that those are not what I would call balanced questions.

    As such I do expect that that survey will end up in what I call ‘Shit In Shit Out’.

  9. I just wonder what kind of background knowledge they have about gaming and researching gaming. What ever the researcher background is he/she should always try to stay on neutral ground. I cannot see how this research could produce valid uncontaminated research data. It has very negative tone toward the gaming subculture and people most likely build up “deffensive” attitude toward the questionaries.

    I wonder if quantitative research is best method to collect data from this. I would have chosen qualitative method since that way I could really find what kind of engagement people have on MMO games. Qualitative data could create good basis on quantitative data that could take consideration both negative and positive effects.

    In this kind of research it would also be good to consider how MMOG guild structure reflects organisational practices and contributes to player base skills on that area.

Comments are closed.