A flood at the National Weather Service (NSW) headquarters in Silver Spring, Maryland last month is still causing issues for National Oceanic and Atmospheric Administration (NOAA) services that relied on the data center there.
NSW provides weather forecasts, warnings of hazardous weather, and other weather-related services. A flood last month caused by a burst pipe inflicted large amounts of damage to the Silver Spring building, including disabling the organization’s data center.
Susan Buchanan, director of public affairs for the NWS, told MyMCM the building “experienced significant and widespread flooding” on the first seven floors, and while water to the building was quickly shut off the water had caused damage to mission-essential data systems, personal workstations and equipment, exterior and interior walls, carpets, and raised floors.
For breaking data center news, features, and opinions, subscribe to DCD's newsletter
National Weather Service still feeling flood after-effects
At the time, NOAA’s National Data Buoy Center was brought offline. The service is seemingly back online, though hosting of data from the Gulf of Mexico oil and gas platforms was transferred from NDBC to the Gulf of Mexico Coastal Ocean Observing System (GCOOS) on March 31, 2021.
While a number of mission-essential systems were moved to backup data centers after the flood, issues still continue. NOAA’s Precipitation Frequency Data Server (PFDS), which provides precipitation frequency estimates and associated information, remains unavailable.
“Due to flooding that occurred at the Silver Spring data center on March 9, 2021, precipitation frequency estimates will not be available through the Precipitation Frequency Data Server (PFDS) until further notice.”
While it says teams are working to resolve the issues, but doesn’t have a time estimate for restoration of services. In the meantime NOAA is advising to post queries via email; users can provide latitude and longitude information in the decimal degrees they need estimates for, and Hydrologic Design Studies Center (HDSC) can provide the estimates in a tabular format.
“The NOAA Hydrologic Design Studies Center (HDSC) will try to provide alternative options to support your data request, but cannot guarantee that all requests can be accommodated. Thank you for your patience.”
The Washington Post also reports that the Weather Service is experiencing bandwidth issues exacerbated by limited hardware and increased user traffic during more regular periods of extreme weather, causing it to limit data customers can download, while recent outages to the NWS Chat service has seen some users temporarily switch to Slack.
Alerts on the National Centers for Environmental Prediction (NCEP) site say there is no NWSChat backup due to the Silver Spring outage and so users were advised to limit usage in order to keep the service as stable as possible during times of high utilization.
The NSW Weather Prediction Center suffered a large outage on March 30 lasting several hours, which Buchanan told the Post resulted from a “combination of increased web traffic associated with the severe weather in the Southeast and the loss of one data center.”
Neil Jacobs, former acting head of the National Oceanic Atmospheric Administration, told the Post many of the agency’s Internet infrastructure problems are tied to the fact they run on internal hardware rather than through cloud providers.
“I’ve demanded in writing that NWS transition these applications … to our Cloud partners. It’s part of an internal strategy I’ve laid out,” he said before he left office. NOAA released its Cloud strategy in July 2020.
DCD has reached out to NSW for comment.
