Update June 13, 2023: Another AWS outage impacting its US-East-1 region is currently underway. More details here.
An outage at Amazon Web Services' us-east-1 cloud region impacted customers globally on December 6.
Amazon subsidiaries like IMDb and Ring went down, as did games like Player Unknown's Battlegrounds, Valorant, Clash of Clans, Destiny 2 and Dead by Daylight, amongst others. The issue began at around 10:45AM ET.
Update: AWS has detailed the cause of the outage here.
Report: AWS uncertain about true cause of massive outage, internal note warns of firewall "overwhelmed by unknown source"
Not clear if it was an internal mistake, or something more nefarious
As it happened:
Update at 2:04 PM PST: Amazon said: "We have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. We are continuing to closely monitor the health of the network devices and we expect to continue to make progress towards full recovery. We still do not have an ETA for full recovery at this time."
2:43 PM PST: Amazon: "We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. We are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. We continue to work toward full recovery for all impacted AWS Services and API operations. In order to expedite overall recovery, we have temporarily disabled Event Deliveries for Amazon EventBridge in the US-EAST-1 Region. These events will still be received & accepted, and queued for later delivery."
3:03 PM PST: Amazon: "Many services have already recovered, however we are working towards full recovery across services. Services like SSO, Connect, API Gateway, ECS/Fargate, and EventBridge are still experiencing impact. Engineers are actively working on resolving impact to these services."
4:35 PM PST: Amazon: "With the network device issues resolved, we are now working towards recovery of any impaired services. We will provide additional updates for impaired services within the appropriate entry in the Service Health Dashboard." Users are still reporting problems. Some sites and services that went down due to the outage may have issues for longer than the outage itself as the returning traffic breaks other systems.
Leaked reports show that Amazon may not know why the outage happened, with an internal note saying that the firewall was being "overwhelmed by unknown source" and the DNS seemed to be targeted.
Original story continues: The cause of the outage is not yet known, and AWS does not currently list any issues on its status page. This is not uncommon for the company - previous major outages have also not been reflected on the status page, or brought down the site entirely.
However, the specific page for the us-east-1 AWS Management Console Home is offline, displaying a 500 Server error instead of information about the Northern Virginia region.
Update: The cloud company has acknowledged the outage, saying: "We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/"
On its updated status page, it said that Amazon Connect (N. Virginia) has degraded contact handling, that Amazon DynamoDB and Amazon Elastic Compute Cloud have increased API error rates, and the AWS Management Console and Support Center have increased error rates.
Other companies impacted include stock and crypto platform Public.com, NFT site RECUR, adult webcam site CB Explorer, tech news website Protocol, audio and video program ReelCrafter v2, software platform ConnectWise, Splunk's Rigor, M1 Finance, Duolingo, and more.
Users of Disney+, League of Legends, Tinder, Instacart, Venmo, Robinhood, Roku, Kindle, and the McDonalds app are also reporting problems.
Warehouse workers at Amazon's e-commerce business were also impacted.
The company's Northern Virginia region had issues with EC2 instances in September after resource contention problems. The same region also had issues in November 2020 after the Kinesis service experienced API errors and caused a number of AWS offerings to struggle.
This story is developing, we will update it as we learn more.
