The US Office of Inspector General has found that the Department of Veterans Affairs failed to meet key data center closure and efficiency targets, mandated under the Federal IT Acquisition Reform Act of 2014, set by the Office of Management and Budget (OMB).
In 2018, VA closed 32 data centers, below its own target of 85, and the OMB's target of 130. The department also failed to meet the OMB's power usage effectiveness (PUE) target of 1.5 by the end of 2018 - with many facilities not tracking their PUE at all.
What is a data center?
The OIG's report, seen here, presents a picture of a vast number of data centers run by VA that the department failed to even report, let alone close. But it should be noted that the agency's definition of what constitutes a data center is different from what many DCD readers would be used to.
The OIG applied the definition of a data center used in 2016 Data Center Optimization Initiative (DCOI followed on from the Federal Data Center Consolidation Initiative, which launched in 2010).
"DCOI defines data centers as rooms with at least one server providing services and categorized data centers into two groups: tiered or non-tiered," the report states.
"DCOI defines tiered data centers as those that use each of the following: a separate physical space for IT infrastructure, an uninterruptible power supply, a dedicated cooling system or zone, and a backup power generator for prolonged power outages. DCOI defines all other data centers as non-tiered. Rooms containing only print servers; routing equipment; switches; security devices, such as firewalls; or other telecommunications components were not considered data centers."
Under that criterion, the OIG says that 860 'data centers' run by VA went unreported, although 520 of them had a single server. This definition confused some, with the report noting that staff at the VA's Office of Information Technology (OIT) "were unclear about what defined a data center and what should be reported."
It should also be noted that all the projections and calculations in the report came out of weighted sample data: "for example, the OIG calculated the projected data centers by summing the sampling weights for all sample records that contained the error - 1,138 (population) / 82 (sample size) = 13.87 (sampling weights) x 62 (unreported data centers) = 860 (projected data centers)."
Now back to the story
The report, titled Lost Opportunities for Efficiencies and Savings During Data Center Consolidation, aimed to not only determine whether VA was meeting FITRA, reporting its facilities, and cutting costs, but also hoped to uncover whether the department had wasted money during a data center move.
A complainant alleged that a data center move from the space located at a leased SunGard facility in Philadelphia, Pennsylvania, to a leased space at a TechAnax data center in Secaucus, New Jersey, and VA-owned space at the Philadelphia Information Technology Center, was handled poorly, with $3.9 million in government funds wasted on empty space.
The OIG investigated the complaint, but found no cause for concern. "While the OIG determined VA made lease payments for both facilities for 22 months, VA reported delays in the migration from SunGard to TechAnax were due to issues related to the physical infrastructure required to operate a high availability environment within the TechAnax facility. These changes were necessary to ensure continuity of operations."
While the OIG gave a clean bill of health to that particular transaction, it said that a lack of planning, and a "lack of coordination and communication" between the OIT and the wider VA officials led to a significant number of unreported data centers, and inability to meet the OMB's target of $85.35 million in savings (VA managed $3.8m).
"Without an accurate inventory of data centers or a credible plan to increase operational efficiency and achieve cost savings, VA will continue to operate in an IT environment that is at greater risk for duplication and waste," the OIG report states.
As for PUE and general efficiency improvement attempts, the report notes: "According to VA’s DCOI strategic plan, PUE [of 1.5] will not be achieved because data center infrastructure management software tools have not been fully integrated VA-wide and instructions to responsible VA organizations for how to conduct requirements analyses, engineer solutions, program funds, implement solutions, and report results have not been distributed."
According to VA’s DCOI strategic plan, in October 2016, the organization began a comprehensive energy metering project to implement DCIM software tools across agency-owned tiered data centers. By September 30, 2018, the OMB requires agencies to to install and monitor energy metering tools in all tiered data centers.
"The audit team found that most of the VA facilities it visited did not have power metering capability and did not report PUE." The OIG visited 66 randomly sampled facilities.
In an attached response signed by Deputy CIO and CISO Dominic Cussatt, VA agreed with four of five recommendations on how to improve its performance and speed up consolidation, primarily by improving communication and transparency within the department.
However, Cussatt said that he did not concur with the OIG recommendation to "establish a timeline that clearly defines how VA plans to achieve OMB’s $85.35 million cost savings and data center closures targets, and optimization performance metrics for energy metering and power usage effectiveness."
Cussatt said: "Given VA’s environment, the majority of VA’s data center closures are due to consolidation efforts (i.e. moving servers from one room to another) rather than the elimination of physical space. OMB agreed that the target of $85.35m would not be feasible for VA to achieve; therefore, VA’s Data Center Optimization Initiative strategic plan does not include a path to meet this target.
"When discussing energy metering and PUE goals, OMB agreed that the costs associated with capturing power consumption across VA would surpass any potential savings. Updated OMB guidance for data center optimization will show modified metrics for energy metering, and no metrics associated with consolidation, optimization, or energy efficiency."
An additional comment suggested that the recommendation to track PUE "should be removed... new OMB guidance for FY 2019 will remove the power metering objective from reporting requirements."
