Amazon apologizes for impact of massive AWS outage

Amazon Web Services (AWS) has issued a formal apology after a massive internal system failure hit critical platforms – including Snapchat, Reddit and Lloyds Bank. 

The unprecedented disruption, which originated in the US-EAST-1 region in North Virginia, required the cloud giant to acknowledge the severe consequences for its vast customer base.

“We apologize for the impact this event caused our customers,” the company stated. “We know how critical our services are to our customers, their applications and end users, and their businesses. We know this event impacted many customers in significant ways.”

The outage’s effects were both immediate and far-reaching. Over 1,000 sites and services were reported down, with popular online games like Roblox and Fortnite restored within a few hours. However, critical functions experienced prolonged downtime, including Lloyds Bank, which saw customer issues persist until mid-afternoon, and US payments app Venmo.

The chaos extended even to household tech, with the event reportedly disrupting the sleep of smart bed owners. One manufacturer confirmed some mattresses overheated or got stuck in an inclined position after losing connection, highlighting the unexpected scope of AWS’s dominance.

In a detailed summary, Amazon pinpointed the technical cause to a cluster of data centers in US-EAST-1. The critical failure occurred when processes governing the Domain Name System (DNS) records – the internet’s “address book” – fell out of sync.

This glitch triggered a “latent race condition,” unearthing a dormant bug activated by an unlikely sequence of automated events in the early morning hours.

Dr. Junade Ali, a software engineer and fellow at the Institute for Engineering and Technology, attributed the core problem directly to “faulty automation.” He stressed that the incident underscores the intense reliance of the digital world on a market largely cornered by AWS and Microsoft Azure.

Dr. Ali warned that companies need to be more resilient, emphasizing the need to diversify cloud providers so systems can move over to another provider if theirs fails. Those with a single point of failure in the impacted region were highly susceptible to being taken offline.

Moving forward, Amazon has pledged to “do everything we can” to learn from the incident and improve the availability and resilience of its infrastructure, committing to protecting the applications and businesses that rely on its services.