Google has revised its procedures for dealing with data center power outages, following a Feb. 24th incident that led to two hours of downtime for all App Engine applications and lingering problems with the cloud platform for several days after. The App Engine team is also planning to implement major changes of the platform's infrastructure.

The new procedural rules include regular drills for on-call data center staff, bi-monthly audit of operations docs and new policy framework "to assist oncall staff to quickly and decisively make decisions about taking intrusive, user-facing actions during failures," according to an incident report Google staff posted on the App Engine discussion board.

The infrastructural changes planned consist of creating two customer options for using Datastore, the engine's on-demand data storage service. In addition to the current single option of low-write-latency and strong consistency in exchange for lower availability, customers will have the choice to opt for higher availability, "at the cost of significantly higher latency."

The company did not disclose the cause of the power failure at its primary data center. When it did occur, about 25 percent of servers in the facility did not receive back-up generator power before crashing and bringing App Engine down.

After Google engineers decided to perform a failover to an alternate data center, they discovered two conflicting sets of procedures - result of a recent Datastore migration. As some of the engineers tried to determine which set of procedures was the correct one, one engineer proceeded to move all traffic to the alternate data center, where a configuration problem prevented it from working properly.

Another mistake followed when the team mistakenly believed that the primary data center was back up and moved the traffic back from the back-up facility. The primary facility had not in fact recovered fully and all traffic again had to be moved to the failover location.

Issues were first determined at 7:48 am and App Engine was back up around 10:10 am. The team continued working to resolve "lingering" issues for several days that followed however.