Managed hosting specialist Rackspace has apologized for the way it handled emergency server maintenance over the weekend, which patched a vulnerability in the Xen hypervisor.

Meanwhile IBM subsidiary SoftLayer has been criticized for not starting the update process until Wednesday afternoon, after the vulnerability was made public, and therefore potentially open to exploitation by attackers.

All of the major cloud providers relying on Xen are expected to take their servers offline at some point to fix the security issue, which was known to a select few before being publicly disclosed on Wednesday. That leaves Google off the hook, DCD understands, as it does not use Xen.

AWS had already finished its maintenance by Tuesday, after restarting thousands of Elastic Compute Cloud (EC2) servers on a rolling schedule over a four day period.

The good, the bad and the ugly
The vulnerability in the open source Xen hypervisor (CVE-2014-7188) was discovered by Jan Beulich from the SUSE project early last week. It could allow a “buggy or malicious” Hardware-Assisted Virtual Machine (HVM) guest to “crash the host or read data relating to other guests or the hypervisor itself.”

This problem affected versions of Xen from 4.1 upwards, running on x86 systems. The existence of the bug was revealed to some cloud providers but kept a secret from their customers, with only a handful of sources suggesting that sudden maintenance updates announced by AWS and others were actually fixing an issue with Xen.

Updating the hypervisor apparently involves taking servers offline for several minutes, so the operation requires preparation - by both the service provider and the customer.

Rackspace's recently appointed CEO Taylor Rhodes has apologized for the inconvenience and lack of information in an email to customers. He explained that the company was trying to avoid alerting cyber criminals to the issue before a patch was in place.

“The key, once a bug is identified, is to fix it swiftly and quietly. This particular vulnerability could have allowed bad actors who followed a certain series of memory commands to read snippets of data belonging to other customers, or to crash the host server,” wrote Rhodes.

“We were faced with the difficult decision of whether to start our reboots over the weekend, with short notice to our customers, or postpone it until Monday. The latter course would not allow us to sufficiently stagger the reboots. It would jeopardize our ability to fully patch all the affected servers before the vulnerability became public, thus exposing our customers to heightened risk.

“We decided the lesser evil was to proceed immediately, at which time we notified you, and our partners in the Xen community, of the need for an urgent server reboot.”

The Rackspace CEO also revealed that the maintenance affected nearly 50,000 customers.

Meanwhile, SoftLayer, the cloud provider acquired by IBM in 2013, only notified its customers about the need for an update on Sunday. It began restarting servers three hours after the bug went public, potentially putting its customers at risk. The maintenance was completed earlier today.