In an extensive audit of the Securities and Exchange Commission’s data centers, the agency’s Inspector General highlighted several major problems with the facilities used to store and process important financial data.

The report was published a week after regulators disclosed that the SEC’s corporate-filing system, Edgar, was hacked in 2016. That breach, which may have allowed hackers to trade illegally, is being investigated separately.

How not to run a data center

While the full report is redacted, the Office of Inspector General released an executive summary of its findings.

Between 2012 and 2013, the SEC relocated its data centers to two sites managed by companies referred to as ‘D1’ and ‘D2’ at a cost of $16 million and $18 million respectively, after spending $162,000 on commissioning a migration plan in 2008.

“However, the SEC did not follow the plan’s recommended steps or timeline to ensure the 2012-2013 data center relocations were properly executed and that the SEC’s data center providers, D1 and D2, could meet the agency’s needs before awarding contracts and migrating data, thereby exposing SEC data to vulnerabilities.

“We were unable to determine why the SEC did not follow the recommended data center relocation steps or timeline because the current officials responsible for the SEC’s data centers were not aware of the relocation plan, many key officials responsible for the data center relocations no longer work at the SEC, and, as discussed further below, contract files were incomplete.”

The report added: “Because the agency derived little, if any, benefit from the 2008 data center relocation plan, we believe the $162,000 paid for the plan represents funds that the SEC may have wasted.”

In addition, the Office of Inspector General determined that “SEC data and equipment at the D1 data center have been exposed to certain physical and environmental control vulnerabilities since the inception of the contract. These vulnerabilities have disrupted SEC operations and resulted in increased costs to the agency.

“Specifically, we estimate that since 2014 the SEC spent about $370,000 in questioned costs to mitigate the physical and environmental vulnerabilities at the D1 data center.”

The office also questioned whether the D1 data center actually meets a key contractual obligation - to be a Tier 3 data center or higher, as defined by Telecommunications Industry Association standards.

On top of that, the “SEC did not adequately manage or monitor its data center contracts,” with contracting officer’s representatives (CORs) failing to validate invoices or maintain complete files - with files “missing required deliverables, justifications and support for critical decisions related to the data centers, and monthly reports.”

The state of poor record-keeping meant that “D1’s monthly power consumption reports were unusable and the SEC did not timely or adequately address known vulnerabilities at the D1 data center, or effectively assess physical and environmental controls at either data center.”

As an example, the office cited that the SEC’s 2016 and 2017 data center assessments “identified no findings at either location, despite vulnerabilities at the D1 data center and a report from a contractor we hired that identified 14 physical and environmental control deficiencies at the D2 data center.”

As a result of its audit, the office made ten recommendations for the SEC, most of which were not released to the public. It did make recommendations regarding data center-related contract management, and “strongly” encouraged the Director of the Office of Acquisitions “to conduct a comprehensive review of the SEC’s COR program and ensure controls are developed or strengthened to improve the SEC’s contract management activities.”

SEC management agreed with the recommendations.

Meanwhile at the SEC

Release of the audit comes on the back of the news that the SEC was hacked, with an unidentified party gaining unauthorized access to Edgar, the Electronic Data Gathering, Analysis, and Retrieval System.

The breach occurred in 2016, but was only revealed in a single paragraph within a five-page statement on the agency’s approach to cybersecurity by Chairman Jay Clayton. In a statement today, the agency admitted that personal information was accessed.

The SEC was criticized for not following procedure after the incident, including failing to inform many of its most senior members, members of the board, or its outgoing Chief Operating Officer.

“There is treasure trove of information at the SEC and I want people to understand this isn’t some kind of victimless crime,” Rep. Bill Huizenga (R. Mich), who chairs a House subcommittee that oversees the SEC, told the Wall Street Journal.

“This is an important thing for confidence in the markets.”

The SEC continues to try to improve Edgar, which has been in use since 1983, with the Office of Inspector General separately investigating the progress.

It wrote: “The SEC consistently spends over $14 million a year on the Edgar system, or about 6 percent of the agency’s information technology budget. These costs cover both ongoing operations and enhancements to the current Edgar system. Separately, since fiscal year 2014, the agency has spent at least $3.4 million on efforts to redesign the Edgar system.”

The office found that the SEC’s governance of system enhancements could be improved, that the agency should improve its management of the system engineering contract, and that the Office of Information Technology (OIT) did not consistently manage the scope of Edgar system releases or consistently implement system enhancements in compliance with federal and SEC change management controls.

News of the hack, and similar issues at companies like Equifax, have caused some to be concerned about the SEC’s plans to push forward with increasing levels of data retention.

The SEC called for the establishment of the Consolidated Audit Trail (CAT) - which tracks the lifecycle of every single order in the stock and options markets, alongside personal information about customers, including Social Security numbers and dates of birth - after the 2010 Flash Crash.

It took regulators months to determine why the Dow Jones Industrial Average fell by nearly 1,000 points for a few minutes on May 6, 2010, because neither the SEC nor Commodity Futures Trading Commission had access to such data (turned out it was down to someone using a spoofing algorithm). But while CAT may have made the postmortem a lot quicker, fears remain over the safety of the information.

“Now is the time to ask not can CAT be killed, but can we rethink what we’ve delivered thus far?” Chris Concannon, president and chief operating officer of CBOE Holdings, told WSJ.

“I am not comfortable with CAT going live unless we are certain the industry’s data will be protected.”

In July, the US government’s watchdog, the Government Accountability Office, said that it had identified weaknesses in the SEC’s information security controls that put its systems “at unnecessary risk of compromise.”