One week ago today, Warlords of Draenor was released worldwide. The launch of the expansion set, like many other expansions, had its share of hiccups. What caused those hiccups? Read on for the whole scoop!
We here at Warcraft Daily hope you’re all enjoying the new challenges, quests, and story of Warlords of Draenor. We certainly are! However, many players found themselves unable to play for quite some time after the launch. So, what happened? We’ll break it down by day!
November 13, 2014
Launch Day! While many players opted for the digital version of the expansion, mainly to avoid having to wait until the next morning to pick up their game, there were still those who purchased physical copies. Hardcore players who have to get the Collector’s Edition were upset some retailers were not doing a midnight release event. This caused much outrage on the Battle.net forums, with refund requests at an all time high.
Later on that day, AT&T, Blizzard’s ISP for realm services, was hit by a massive DDoS (Distributed Denial of Service) attack. This caused realm stability issues, mainly on the high population servers. In response to this, Realm Techs temporarily lowered the maximum number of players that can be connected at any one time. Login queues were seen on many servers, if not all servers. Many threads popped up demanding compensation and for Blizzard to “fix it”. While Blizzard was indeed fixing it, the actions required to fix it were seen by many as incompetence. That was simply not the case. The realms had to be brought to a stable level before any fixes could be deployed.
I’d like to use Warcraft Daily’s infrastructure as an example. We have many services running across multiple servers. Our main server, which performs load balancing (among other things), became unstable at one point:
Due to a large scale DDoS attack on nodes used by Warcraft Daily, our site and related services are down. Updates to follow.
— Warcraft Daily (@WoWDailyBlog) November 19, 2014
We first had to attempt to shut down all nodes and start them up one by one. Our storage system came back up immediately, but the server which houses our main website was stuck in a failed state. Our provider was aware of the issue, and was working quickly to restore service. However, Warcraft Daily was completely down for a total of 3 hours.
While its hard to say exactly how Blizzard handles things like this, it is safe to assume they would do things in a much similar way. First they would attempt to stabilize the services as they stand at that time. Next, they would start work on a fix to address all of the issues at once. Finally, they would deploy the fixes and, if necessary, restart the affected services.
Some fixes (hotfixes) don’t require the service to be restarted. However, things like realm caps and certain setting changes do require a restart. This isn’t due to poor coding, but rather how Linux applications work. When a Linux application starts, it loads many of its configuration settings into memory. Those settings cannot be changed unless the application is restarted. Which is exactly what had to happen. There were many realm restarts that day, many of them unannounced due to the frequency in which the restarts had to happen to resolve the issue.
November 14 to November 16
Realms continued to have queue times of many hours, with tens of thousands of players attempting to login to the most populated realms. More outrage on the forums caused Blizzard to release a series of announcements, which went a bit like this:
As most of you are aware, the demand on the game is extremely high right now, leading to increased queue times on many realms. We’re also seeing some login errors as a result. This is compounded by us needing to temporarily lower the population caps on all realms to help assist in realm stability issues due to a number of in-game factors that are putting greater than normal strain on the game servers. For those in-game now we’re seeing fairly stable gameplay, but with some continued issues when phasing into the Garrison. Even with lowered population caps we’re not seeing performance where it needs to be.
As previously posted, both Europe and the Americas/Oceanic realms went into an off-peak maintenance to upgrade realm hardware, and implement a new Draenor-wide instancing tech to improve game performance. Both regions’ maintenances concluded successfully, and so far we’ve seen very positive results, allowing us to dramatically raise the realm population caps for all realms. We’re going to continue optimizing the in-game experience, focus on the new instancing tech, and hotfix gameplay issues as they come up.
November 17 to Today
As the initial surge of players decreases, the realms are appearing more and more stable. Of course, this is also due to us analyzing things in the middle of the week, when most players are at school or work. We’ll continue to monitor things and post new information when it becomes available.
Follow Warcraft Daily on Twitter! @WoWDailyBlog : http://www.twitter.com/WoWDailyBlog