Rackspace is running out of cloud storage in its LON region in London, UK.
The problems began on November 17, with customers reporting 503 errors when accessing files in the LON region, as first reported by The Register.
According to a post from Rackspace, the company's engineers increased storage capacity in the region a week later on November 24.
Rackspace also noted that the mitigation would take effect "over a number of days," though its website still cites cloud files as suffering from service degradation as of writing on December 4.
Since problems first arose, Rackspace has been providing its customers with regular updates. On November 28, the company said that engineers continued to work on the issue, and that customers should be able to access already uploaded content by may experience issues with new uploads.
On November 29, Rackspace said that its engineers were continuing to "monitor the frequency of errors" and that "further additional storage capacity" would be added in a "controlled manner."
In order to increase space, engineers had to replace the utilized drives and pull in inventory from Rackspace data centers across the globe.
"As part of this process, engineers began to utilize drives with higher capacity. Given the type of storage configuration in this environment, the drive replacement process necessitates that drives are replaced as a group, rather than individually," the post goes on to say.
Some of the drives that were installed did not work, however, with the configuration failing to recognize them (and thus the process having to be re-done).
Error rates were significantly reduced by December 1, but affected customers may "continue to experience impact." Rackspace's engineers continue to work on the problem.
A Rackspace spokesperson told the Reg that no data was lost in the incident and that it was only related to customers uploading new objects.
"Reasonably foreseeable storage needs were forecasted and accounted for. High demand for Rackspace's Object Storage services significantly exceeded those forecasts," added the spokesperson.
The LON region is hosted out of Rackspace's three London, UK, data centers: LON3, LON5, and LON8.
LON3 is a 5,000 sqm (53,820 sq ft) facility in Slough offering 3.1MW of power per floor plate. LON5 is a 6,600 sqm (71,040 sq ft) facility in Crawley, West Sussex, and LON8 is the company's largest London data center with 7,560 sqm (81,375 sq ft) of IT space and 3.6MW of capacity.
Rackspace also suffered an outage in its London region in May this year, with the LON5 and LON3 data centers affected. The outage extended to the company's APAC region, with Sydney and Hong Kong data centers impacted.
