Something clearly better in small doses may not be better in large doses. I accidentally replicated the New Coke taste test problem last weekend when comparing the store and national brands of a soda flavor. The flavor I preferred in small quantity started seeming unpleasant when drinking an entire glass. Also, some of the merit of the small quantity may have been that I was already familiar with the national brand, so the store brand’s minor variation had a bit of novelty that was gone a few ounces in.
MMOs are typically designed to be played in thousands of hours but tested in dozens of hours. What is awesome in the beta, warts and all, may pale quickly once it becomes familiar, and then it will become far more than familiar when the developers multiply some numbers by 10 to create a theoretically satisfying grind. The only people who are going to repeat the same content ten times in beta are the ones who really really like that content or really really like that grind. This may not translate well to the median player once you go live, even if the beta feedback was entirely positive.
In program evaluation, we must constantly remind policymakers that average results will be worse than the pilot. The pilot project involved finding the most enthusiastic volunteers and setting them loose on the problem with the greatest room for improvement, and the specifics of the project were tuned to their capabilities. You are testing the best case scenario. In game development, beta testers (the ones actually testing) are the people so enthusiastic about the project that they are willing to work for free and then pay you for the output. These are not typical customers.
: Zubon
You might have made a mistake in format after you link good grind bad grind to this post.
Fixed, thanks.
You make a very good case for why some people, myself among them, contend that the best experience you are likely to get in any MMO is during the Closed Beta phase. After all this time, wouldn’t you think developers would realize this? Evidence says otherwise, though.
I suppose the question is, what’s the alternative?
I don’t know about that. I’ve been in most betas since UO and I can assure you the gaming experiences as a whole were attrocious. Rift is the outlier on the plus side and Star Trek on the negative.
I would say that the social experience is better though. Devs talk and the players all have a common cause.
I’m wondering if the reverse is true as well. For instance, if EVE just started up in beta form right now, how positive would the reaction be? Remove all of the player history, the 10 years of established stuff like the economy and major players, and what is left?
Mining, mission running, and what seems like an impossible grind towards even remotely decent ships (self-producing even a battleship is a crazy effort if you go it alone from zero). In a short-ish beta, there would not be enough time to really establish the ‘good stuff’.