What caused the AWS outage?
The root cause of the AWS outage on Monday 20th October has been identified by Amazon as appearing to be primarily “related to DNS resolution of the DynamoDB API endpoint in US-EAST-1.” Amazon has also stated that the issue affected other services in the US-EAST-1 region.
US-EAST-1 is one of the data centre regions used by AWS to host customer sites, applications and data. Located in Virginia, the data centre region is one of Amazon’s primary cloud hubs.
What is DNS resolution?
Domain Name System (DNS) resolution is the system through which website names are translated into machine-readable IP addresses. Whenever an internet user inputs a domain name in their browser, the device sends a query to a DNS resolver, which undertakes its own queries to find the correct IP address. The resolver then sends the IP address back to your device, allowing it to fetch the site.
The system is foundational to internet navigation, so when there is a malfunction, it causes significant disruption.
Why was the impact of the AWS outage so widespread?
The impact of the AWS outage was widespread – affecting over 1000 companies, and countless service users – due to the volume of customers hosted in the US-EAST-1 data centre cluster.
Virginia has the largest data centre market in the world, hosting over 300 individual data centres. There are multiple reasons for this concentration – Northern Virginia is in close proximity to the U.S. federal government, and offers local tax incentives for projects with a large capital investment, including data centres. AWS has data centres around the globe, but US-EAST-1 is usually the default due to its scale.
Therefore, while the data centre region is located in the U.S., it hosts customers from around the globe, meaning the impact was felt worldwide.
Further to this, many companies rely heavily on a single hosting provider, in this case AWS. This opens businesses up to cloud concentration risk – if you store all of your critical data and applications in one environment, a single incident, such as this outage, disrupts your entire operation. Many companies host the majority or entirety of their infrastructure with AWS, meaning this incident caused significant issues.
What can companies do to prevent downtime due to outages?
Diversify your infrastructure and mitigate cloud concentration risk
There are several measures you can take to mitigate the risk of cloud concentration, including diversifying your infrastructure, hosting with a vendor-agnostic managed service provider, and building a disaster recovery strategy to ensure you are protected in the case of an incident.
Jake Madders, Co-founder and Director of Hyve Managed Hosting, commented in Capacity Magazine: “The key lies in building resilience into your infrastructure from the outset. Diversifying across multiple cloud providers and geographic regions is essential to ensure redundancy and enable seamless failover when disruption occurs. Just as important is decoupling critical services – such as, for example, identity management, DNS, and core data layers – from any single provider, so that if one ecosystem is impacted, your operations can continue elsewhere.”
Read more details on protecting your infrastructure in our insight “Mitigating Cloud Concentration Risk”.
Consider data sovereignty
The outage highlighted a growing concern – many companies across the UK and Europe are increasingly reliant on U.S. hosting. In this instance, many vital services in the UK, for example the HMRC website, and major UK banks, were disrupted by an outage in the U.S. This is a particular problem for these important services, where any disruption has a widespread and ongoing impact on the UK population, and raises concerns over the safety of the sensitive data they hold.
Data sovereignty – the principle that data is subject to the laws of the jurisdiction in which it resides – is a priority for many such organisations. However, this incident has highlighted that the risk of hosting your data out of your jurisdiction is farther reaching. Madders stated: “For organisations that prioritise data sovereignty, it should also be a key consideration, with local failover options and replication to trusted jurisdictions built into their continuity strategy.”
Plan your incident response
While you can and should put safeguards in place to diversify your infrastructure, and consider data sovereignty from the outset, it is also vital to prepare an incident response plan to prevent an incident like the AWS outage disrupting your entire business. An effective incident response plan should cover early detection through proactive monitoring, a strategy to communicate quickly between key stakeholders to coordinate your response, and automatic failover and disaster recovery processes to allow your operations to continue.
Protecting your business
If you were impacted by the AWS outage, or have concerns about the safety of your infrastructure in case of future incidents, and would like to find out more about mitigating cloud concentration risk, diversifying your infrastructure, or ensuring data sovereignty, you can set up a consultation with one of our experts. Our team can assess your current setup, and advise on how to optimise and protect your platforms.
Fill out our contact form and one of our team will be in touch.