Amazon Web Services is suffering performance issues around its EC2 and EBS storage services in its Northern Virginia region, with issues continuing from Sunday into Monday morning.
According to an AWS status pages, at 8:11 PM PDT yesterday the company said it was investigating “degraded performance for some EBS volumes” in the USE1-AZ2 Availability Zone at its US-EAST-1 Region in Northern Virginia. The company said some new EC2 instance launches within the affected Availability Zone were also impacted.
The company later confirmed EC2 instances within the affected Availability Zone that use EBS volumes may also be impaired due to stuck IO to the attached EBS volume(s), while newly launched EC2 instances may fail to launch due to the degraded volume performance.
The issue has been isolated to a subsystem within the EBS service: “A subsystem within the larger EBS service that is responsible for coordinating storage hosts is currently degraded due to increased resource contention.”
The company said it made “several changes” to address the increased resource contention within the subsystem responsible for coordinating storage hosts with the EBS service, but while the changes led to some improvement, affected EBS volumes hadn’t fully recovered.
As of 1:15 AM PDT of September 27, the company said performance has returned to normal levels for “the majority” of the affected EBS volumes within the affected Availability Zone, but the company also saw recovery slow down some affected EBS volumes as well as some degraded performance for a "small number of additional volumes" in the affected Availability Zone.
“We have investigated the root cause and mitigations are underway to complete the performance recovery of the affected EBS volumes.”
Signal, Xero, and Nest were amongst those company said they were affected with outages.
Update: AWS has said the issue is fully resolved and the service is operating normally, but it is also continuing to recover some services, including RDS databases and Elasticache clusters.
"While almost all of EBS volumes have fully recovered, we continue to work on recovering a remaining small set of EBS volumes. We will communicate the recovery status of these volumes via the Personal Health Dashboard."
