Two Day Downtime…

Is there any circumstance that makes it even remotely acceptable for an MMO studio to bring its servers down for two full days?

Blizzard has announced a two day maintenance on select realms, with other servers to undergo similar downtimes in the future.

File this under, “WTF were they thinking?”

~Cyndre

27 thoughts on “Two Day Downtime…”

  1. I thought the same thing. Makes me wonder if it is some real crisis (scalability? game server hack? billing server hack?) that they are having trouble figuring out how to solve quickly. 2 days isn’t acceptable and I’d think they’d know that.

    I wonder if they plan on extending affected users’ subscriptions by 2 days?

  2. I believe they are thinking: It might take awhile to patch in voice chat perhaps we should schedule an extended maintenence.

    Most likely they are upgrading half the servers this week and will do the other half next week.

  3. New servers are tested and released apart from the live servers. The ones they are bringing down are old servers, including my former server, Magtheridon. It must be an infrastructure issue or possibly a hack/ dupe issue, but those are usually done with quick restarts or short maintenance. Keep in mind that Tuesdays are scheduled downtime already so this is something they simply could not accomplish over a few weeks of server downs.

    I haven’t had a chance to read their reasoning for the downtime, but if doesn’t involve MAJOR stability improvements or content additions, it can’t possibly be worth the hassle and criticism and lost revenue this will create.

  4. I think Changling Bob hit the nail on the head, for whatever reason. I remember in DAOC, they had to bring servers down, and they credited time. I can’t remember what other games have, but I recall reading about a few that credited time and/or XP from resets, etc.

    Because its blizzard, they dont have to worry about those things. Also though, would be fun to know how much it would cost them to credit 2 days on every account effected =)

  5. Eh, I’m not on the affected servers (fortunately), but when I thought I was, I was hopping mad. While getting a 2-day account credit (which they haven’t said they’re doing) is nice, the truth is that it doesn’t really mean much. You were paying for unbroken service anyway. It’s not like you get a bonus two days. Getting a credit just means that when you finally get around to canceling, your sub will have an extra 2 days to run out while you don’t play than it would otherwise (and it shuts some complainers up, maybe).

    Two day downtime is ridiculous. Most games don’t even have the one day per week that we’ve just gotten used to and accepted because it’s WoW.

  6. Yes, there are definatly circumstances that make it *perfectly* reasonable for servers to be down for two days and I’ll tell you what that reason is. “They Need It”. If the server admins or whoever is doing the work on them say that the servers need to be down for two days, then they do. Do you *really* think that they just said “hey, you know what would be fun? Let’s turn off x number of servers for 2 days! Can you imagine what the forums and mmo blogs will look like? It will be glorious!”?

    Really?

  7. […“They Need It”. If the server admins or whoever is doing the work on them say that the servers need to be down for two days, then they do…]

    In the Server maintenance and database management field, anything less than 99.9% uptime is failure. Blizzard has long since set the expectation that they will be down for a weekly maintainence. If Blizzard has stability issues or load problems, they should be addressing them with sustained, ongoing maintenance, not emergency extended maintenance.

    If you recall, about the time of the Naxxrammas patch, Blizzard performed a complete retrofit of all old servers with new hardward and databases… During that transition, they only took the servers down for 13 hours…

    What could be worse than a complete retrofit, that would require 48 hours? And if they just spent all that time and money on new hardware and maintenance, why are they having to go back in so soon for major upgrades?

  8. Here is the list of Servers affected by this downtime:

    Aggramar, Argent Dawn, Arthas, Azgalor, Bleeding Hollow, Bloodhoof, Burning Blade, Burning Legion Durotan, Earthen Ring, Elune, Eonar, Eredar, Gilneas, Gorefiend, Kargath, Laughing Skull, Lihgning’s Blade, Llane, Lothar, Madoran, Magtheridon, Malygos, Mannoroth, Medivh, Shadowmoon, Shattered Hand, Skullcrusher, Stormrage, Thunderhorn, Thunderlord, Trollbane, Warsong, Zul’jin

  9. If the server admins at my company said they needed to take production servers down for 2 days, we’d probably be looking for new admins.
    The only time any server was down for 2 days I can remember here or at other companies was when moving servers to new datacenters. Even then i don’t think it was 2 days, i don’t remember.

    that COULD be it, now that I think of it. But you’d think they’d say so.

  10. […In the Server maintenance and database management field, anything less than 99.9% uptime is failure….]

    So, every MMO ever has been a failure? According to your standard of 99.9% uptime, that is less than half a day per year.

  11. [If the server admins at my company said they needed to take production servers down for 2 days, we’d probably be looking for new admins.]

    It all depends on the business – for enterprises which may loose millions for every hour the production environment is down, or people may get hurt or loose their lives if the systems are not working – then 2 days downtime would be totally out of the question obviously.

    Blizzard and many other companies with subscription-based games are not that likely to have a significant revenue impact if the systems are down for extended times, at least if it is not that often.
    Whatever they are doing might very well be possible to do faster or with less downtime, but might then be more expensive and more complex.

  12. I agree with Sente. I’ve heard the 99.9% number before, but that’s running something that is generally less complex than WoW, and something that will be much more damaging if it goes down.

  13. Ok, agreed… my 99.9% figure is apples to oranges and not fair to MMO studios in general, but 48 hours is still an exceptionally unorthodox downtime, no matter what the industry.

  14. Everquest 2 having extended maintenance for 8 hours, WoW having a 2 day downtime, both having this overlap period. The end is here! What are we MMO kids gonna do :(

  15. […I agree with Sente. I’ve heard the 99.9% number before, but that’s running something that is generally less complex than WoW, and something that will be much more damaging if it goes down…]

    Systems much -more- complex than WoW (large-scale payment systems, telecom networks, etc.) actually have -higher- standards and track records than 99.9%. Of course, those industries have had decades of practice at this. But it is achievable.

    I don’t hold WoW to the same standard I hold my phone or credit card, but 2 days is 1/15th of a subscription period, or a $1 value. Not that $1 is going to affect me at all, but a company shouldn’t just take $1 of value from hundreds of thousands of subscribers and not reimburse.

    And of course, the “$1” figure is from a very narrow, numbers-focused perspective. The emotional frustration of not having the servers available for two days makes this a PR hit and Blizzard should be apologizing and reimbursing just to try to smooth things over.

  16. “and Blizzard should be apologizing and reimbursing just to try to smooth things over.”

    Only, they won’t… cause they are Blizzard (as previous posters have said), they know WoW is the best game in town right now and almost no one will cancel a subscription over this.

    Players: ‘How can you take down the servers for 2 days without any explanation or promise of reimbursement?’
    Blizzard: ‘We fart in your general direction!’

    With apologies to Monty Python… ;)

  17. They are all old (launch or just after) servers, from what it looks like.

    I’m guessing “Tired, worn-out server hardware on the verge of implosion” combined with a side order of “implementing shiny new & upgraded hardware” and a garnish of “If we don’t test this thoroughly, people will whine even more when the server lags/crashes/etc.”

    …or would people rather Blizzard waiting until [i]after[/i] their servers caught fire, and then said “Two day rollback [b]and[/b] two day downtime”?

  18. I personally do not have a problem with it. My server is an affected server. But with the recent lag and other problems with that hardware 2 days to go out and be in the sun is not a bad thing. It could be worse, the servers could have gone down last month and all the content be lost like what happened in one of the data centers in California for some sites.

    If they need to take 2 days to make sure the hardware is up to snuff and have all the problems figured out then they take two days to do it. It’s better then servers being up and down non stop for 2 days or more while they work out the kinks.

  19. Oh above I was talking about just internal stuff, not the external sales site, yes those need 6 sigma uptime not just 3. :) But even the internal servers have never been down that long.

    Of course, I guess you lose money in lost productivity. I wouldn’t know since our department has no funding for our project we sit around surfing all day. 3 senior developers with nothing to do for 6 months has already cost them more than they’re arguing about whether to spend or not. So uncapitalized expenses are not something they really track that well here :)

  20. “Systems much -more- complex than WoW (large-scale payment systems, telecom networks, etc.) ”

    Yeah, 2 days is a lot even for WoW. And I can see telecom networks maybe being more complex, but not many other systems. Something that’s mostly a database is not as complicated as a database hooked up to a game that’s running in real time for 1k people. Some clients are making rather large demands on the database, so to speak (ie all the people in IF or Org or Shatt).

    I guess I just consider most other things comparable more to battle.net than to WoW. Definately correct me if I’m wrong about that though, I’m curious.

  21. “I guess I just consider most other things comparable more to battle.net than to WoW. Definately correct me if I’m wrong about that though, I’m curious.”

    Well, I can talk about payments systems as a specific example, since I worked at PayPal for 7 years. You’ve got:

    – Monolithic data model: No sharding to make things easier — everyone has to be able to pay everyone else
    – Real-time fraud processing: A lot more complex than you think :-)
    – Integration with dozens of external services (think about all the payments systems in all the different countries, integration with USPS and UPS for printing shipping labels, integration with eBay for synchronizing data, etc.)
    – Integration with internal tools for customer service
    – Real-time processing: People expect their payments to happen quickly and webpages to load fast (not as quickly as WoW events, but still real-time).

    The system diagram of servers and connections is HUGE, more so than one would reasonably expect from being an occasional user. And this is true of most applications we use every day!

  22. You still have access to your account during server down time. You can roll on another server. I imagine you won’t get a credit.

    By the way, don’t forget to read the EULA you clicked “Accept” too.

  23. “I haven’t had a chance to read their reasoning for the downtime, but if doesn’t involve MAJOR stability improvements or content additions, it can’t possibly be worth the hassle and criticism and lost revenue this will create.”

    In other words, I have no idea what I’m talking about. I’m just cranky want to cry and post a troll thread.

  24. […In other words, I have no idea what I’m talking about. I’m just cranky want to cry and post a troll thread….]

    No, that isn’t what I was trying to say, but thanks for your efforts in trying to translate my thoughts. If you read the other 20 posts in this thread, you will notice that those of us with previous genre experience or knoledge of online infrastrutures were discussing whether it is acceptible for an MMO to have a 48 hour downtime.

    The reasons underlying the downtime are largly irrelevant. If you can’t maintain your systems under your standard maintenance schedule and you need to re-retrofit server hardware that your fixed just a few short months ago, something is wrong.

    In any event, thanks for stopping by, and I look forward to you adding something constructive to the commentary in the future.

Comments are closed.