Pareto Superior Testing

Our testers can veto releases at work, but we have an allied tradition that half a loaf is better than none. We may not get everything we want from an update, but if it makes some things better and no things worse, we go live. We can add the rest in a future update.

A gaming example comes from GW2 crafting. At launch, crafting could use items only from your character’s inventory. Soon after, you could craft from the vault but discovery was still inventory only. Now both check character inventory and the entire vault.

This is easier in my work than in gaming because our users are not competing with each other. If we can implement new functionality for one interface but need another month to accommodate the rest of our users, bonus for the users with the easy update. If your FPS added rocket launchers for PC players but needed another month to add it to the Mac client, forums would explode, especially if PC and Mac players were on the same servers. You can see this in games that are gradually rebalancing one class at a time rather than all at once. The relative values of classes are having large swings each month. LotRO had “the month of the [class],” TF2 had class-specific updates, and other games have similarly revamped single classes. See also City of Heroes gradually adding heroes’ passive archetype abilities over time, so there were months in which only half the classes had them.

Sometimes half a loaf is worse than none. Beyond the cases where it distorts your competitive balance, a function that only half-works can make some things worse and no things better. Adding something that only works for a known half of the users is inconsistent but reliable, which can be okay; adding something that works for everyone a seemingly random half of the time is inconsistent and unreliable, which is bad. The new functionality must work as expected, even if only under additional assumptions, and those assumptions must not cause other problems. Half a loaf is better than a whole loaf with gravel scattered through it.

: Zubon

3 thoughts on “Pareto Superior Testing”

  1. I had no idea the crafting discovery window was including items in my main bank vault. That’s the first reason I’ve seen to spend money on gems. When did that change?

  2. Note that the half-loaf solution has another problem: even inconsistent but reliable solutions have a nasty tendency to become a ‘working solution’ when manpower or resources are limited. There’s a lot of inertia when trying to change bad fixes into good ones, either because there are always more valuable targets, or just because people are used to the bad fixes.

    Business software and especially programmer-tool software is particularly prone to this — many parts of the open-source community are built around half-loaf functions like terrible command-line interfaces or ‘don’t press this’ buttons — but video games and MMOs are far from immune. These are always things that are steps forward originally, but the lack of followup eventually drags down.

    On the other hand, the perfect is the enemy of the good.

Comments are closed.