Amazon Web Services (AWS) engineers still are not entirely sure what caused a massive outage on December 7.
While the company has publicly said that the prolonged downtime at its us-east-1 was due to "network device issues," leaked internal messages show much more confusion and concern amongst staff.
Business Insider reports that the company traced the root cause to "a problem with several network devices within the internal AWS network," where the "devices are receiving more traffic than they are able to process, which is leading to elevated latency and packet loss for the traffic traversing them."
But why they were getting the surge in traffic is not clear. The issue "specifically impacted" Amazon's internal DNS, with one AWS employee speculating it was an "orchestrated DNS attack."
An internal note seen by BI said that "firewalls are being overwhelmed by an as of yet unknown source," with the company working on "blocking the traffic from the top talkers/offending hosts at the firewall."
A separate Slack message said that the issue could be down to Amazon's real-time digital advertising auction.
AWS' vice president of infrastructure, Peter DeSantis, led a 600-person internal call about the then-ongoing outage, where some said it was likely an internal issue, and others pointed to more nefarious possibilities.
"We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired," AWS said on its status page.
"With the network device issues resolved, we are now working towards recovery of any impaired services."