The US Department of Energy's (DOE) watchdog has reported 'poorly managed' operations and maintenance at the Oak Ridge National Laboratory data center in Oak Ridge, Tennessee.
In September 2022, the DOE Office of Inspector General (OIG) received an allegation of insufficient maintenance and calibration in Oak Ridge's data centers, home to the world's current most powerful supercomputer, Frontier.
The DOE released an inspection report on November 29 detailing its investigation into the allegation between January 2023 and September 2023.
The allegation made was related to buildings 5300, 5600, and 5800 at ORNL, where the complainant said that operations and maintenance were poorly managed, that the calibration program was inadequate, and there was "poor or no maintenance" on pressure relief valves.
The DOE investigation 'substantiated' these claims, finding that the calibration program did not meet quality assurance requirements, that UT-Battelle (which manages and operates ORNL) could not produce sufficient documentation, and similarly that the pressure relief valves were not always maintained "in accordance with applicable criteria."
The OIG has put these issues down to a lack of ORNL oversight, stating that: "According to a UT-Battelle manager, pressure relief valve testing did not occur because it was overlooked in some cases."
In relation to the pressure relief valves, UT-Battelle found similar issues in a 2020 assessment. If unresolved, the pressure relief valves could exceed allowable limits, potentially harming personnel and equipment.
The OIG further found that UT-Battelle's QA requirements did not always flow down properly to subcontractors.
Oak Ridge National Laboratory and UT-Battelle have received three recommendations: To meet quality assurance requirements and related monitoring and documentation actions, maintain the pressure relief values according to their procedure, and include the order in the subcontract that lacks proper flow-down.
Management from Oak Ridge concurred with the audit's findings, agreeing to complete the adjustments by February 29, April 30, and March 31, respectively.
Update: Oak Ridge told DCD in a statement: “UT-Battelle is implementing OIG recommendations related to procedures used in operating the buildings where the laboratory’s supercomputers are located.
"The identified issues do not pose a risk to operation of the Oak Ridge Leadership Computing Facility’s supercomputers.”
Original story continues: In addition to the Frontier supercomputer, Oak Ridge National Laboratory is home to a 10 petaflops supercomputer used by the National Ocean and Atmospheric Administration, and the Summit system.
