A switch failure at a State data center in Virginia took a number of state agencies offline on Saturday, May 23. The State is already in the process of ending an exclusive agreement with the private owner-operator of the site, Northrop Grumman, according to reports, and outage is likely to increase tensions between the two organizations. 

The Commonwealth Enterprise Solutions Center (CESC), a joint venture owned and operated by Northrop Grumman, serves the majority of Virginia’s state agencies, including the Department of Motor Vehicles, which controls vehicle and driver registration in the state. The shutdown began at about 7am (EDT) and lasted five hours, during which time police could not access driver records, and access to email, cellphones and agency computer servers were also affected, according to a report in the Richmond Times-Dispatch.

Backup failure

The State is asking Northrop Grumman why the data center’s backup system did not operate as designed, according to Marcella Williamson, spokesperson for Virginia Information Technologies Agency (VITA), the public body that supervises operation of the facility.

No details of the cause of the fault have been released at the time of writing, but VITA and Northrup Grumman are conducting analysis to prevent a similar outage in the future.

The CESC was opened in 2007 as the flagship of a $2.3 billion public-private partnership, which dates back to 2006. Initial public tests of the facility demonstrated its ability to provide a reliable service. 

Now, however, Virginia wants to “disentangle” itself from the exclusive deal, and announced a month ago that it would open up its agencies’ IT purchasing to allow contractor status to multiple vendors.

In its turn, Northrup Grumman warned VITA that the new arrangement “would compromise security and raise costs for taxpayers”, declining the role of primary contractor under the proposed new arrangement.

The fault occurred when a hardware component in one of the data center’s switches failed. The outage could have lasted longer, but staff fitted a switch from a test environment to replace the original part, instead of waiting for a new part to arrive. 

Although some police services were affected, the majority of police work was left unaffected, according to the Times-Dispatch, because the agency has been slow in migrating its IT to the data center.