All power distribution units at the Dallas data center of colocation provider Colo4 came back online Wednesday evening after a partial outage that lasted for several hours, keeping numerous customers' equipment down.
The incident started when an automatic transfer switch (ATS) at one of the six "service entrances" (where utility power comes into the facility) failed, according to status updates on Colo4's website. The first update that alerted of a "power issue" came at 12:40 p.m.
The data center had a spare ATS on site and the company's technicians installed it as a temporary replacement until a new primary one could be installed. "Bypassing an ATS is not a simple fix, like putting on jumper cables," a Colo4 update, apparently explaining the length of time it took to restore power to the affected customers, read.
"Given the size and power of the ATS, the safety of our people and our contractors must remain the highest priority."
Network connectivity was also affected for some customers. While the data center has a redundant power-infrastructure configuration for most of its routing gear, one switch was connected to only one feed. Colo4 admitted that that was a mistake and said it was corrected on Wednesday.
Power to distribution gear from the temporary ATS was restored around 5:40 p.m., and all HVAC units were brought back online. At that time, technicians began to restore power to affected UPS systems, power distribution units and then to customer equipment.
As of 6:30 p.m., all PDUs were online in bypass mode to let UPS batteries charge. The data center operated on generator power while charging took place, after which technicians placed it back on utility power.
Colo4's Dallas data center is an 11.1MW facility, following an upgrade in June, which added 4MW of capacity.