AWS outage Live Updates: Amazon says the underlying issue ‘resolved’ as it works on ‘full recovery’

Amazon has said that it has resolved another issue as they continue working towards full recovery.

“We confirm that we have now recovered processing of SQS queues via Lambda Event Source Mappings. We are now working through processing the backlog of SQS messages in Lambda queues,” the company said in a update.

DownDetector has said that globally so far today, more than 6.5 million outage reports have been recieved, affecting more than 1,000 companies.

BBC reported that Snapchat is up and running again – but some users said that they lost all their friends on the app.

Snapchat was among the platforms experiencing issues this morning amid the AWS outage.

Amazon has issued fresh recomendations as it continues full recovery of the AWS systems.
“We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically,” the statement said.

Monday’s outage stemmed from Amazon’s US-EAST-1 region in Virginia, on the East Coast of the US.

This is the tech giant’s original and largest location for its web services, and there are a lot of data centres there.

It is known to many as a critical centre for the global internet and is the default region for many users.

But because of its age, size and amount of on-demand capacity, it is prone to outages.

This also highlights the huge challenge with so many businesses depending on single cloud computing regions from single cloud computing providers.

The British National Rail has said that it is working with suppliers to resolve the problem.

“Work is ongoing to the systems that are still being impacted following the earlier global IT issues. We are working with suppliers to do this as quickly as possible. Please note, until these systems are fully restored, some of the information displayed on the National Rail Enquiries website / app may still be out of date,” the statement said.

The British National Rail website and app have been affected by the AWS outage.

According to the BBC, the Rail Delivery group says: “Due to technical issues with our infrastructure provider this morning, the National Rail website and app have intermittently been impacted by slower speeds and difficulty loading.”

As many other sites appear to be recovering from the Amazon Web Services issues, Reddit is curiously seeing a spike in reports of problems from users.

Downdetector saw more than 5,000 reports when other sites started getting fixed – with most flagging problems affecting its app.

The New York Times reported that most websites and apps that were experiencing disruption are working normally now, and the underlying issue has been mitigated, though there may be some lingering delays.

Lloyds Banking Group – which includes Halifax and Bank of Scotland – says its services are now coming back online.

Amazon lists two problems it’s working on for full recovery of the AWS system/

“We are continuing to work towards full recovery for EC2 launch errors, which may manifest as an Insufficient Capacity Error. Additionally, we continue to work toward mitigation for elevated polling delays for Lambda, specifically for Lambda Event Source Mappings for SQS,” the latest statement said.

Amazon said in an earlier update that the issue “appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1.”

DNS, which stands for Domain Name System, is often likened to a phone book for the internet.

It effectively translates the website names people use into IP address numeric equivalents that can be read and understood by computers.

This process basically underpins the way we use the Internet, and disruptions to it can cause web browsers to be unable to locate the content they are looking for.

Amazon said the problem appeared to originate in a critical AWS infrastructure zone, the US-East-1 region, located in the northern part of the US state of Virginia.

Amazon has said that the underlying DNS issue that caused Monday’s global AWS outage has been resolved.

“The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events, such as CloudTrail and Lambda. While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances, such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution. If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches,” the latest statement said.

Amazon’s share price is down about a tenth of a per cent in premarket trading in the US, suggesting that investors are not too bothered about the outages.

Downdetector showed that several major mobile phone operators, some UK banks, WhatsApp, and Tinder were also affected in Europe.

DownDetector lists Delta Air Lines and United Airlines as being among the dozens of companies that experienced disruptions.

According to its health status page, Amazon has downgraded the severity of the AWS outage from ‘degraded’ to ‘impacted’.

Amazon has said that the AWS systems that saw a global outage on Monday continue to see recovery.
“We continue to observe recovery across most of the affected AWS Services. We can confirm global services and features that rely on US-EAST-1 have also recovered. We continue to work towards full resolution and will provide updates as we have more information to share,” the statement said.

Some media advocates said that the outage, which caused disruptions to secure communications apps such as Signal and other digital tools, showed how the internet’s reliance on a few major technology companies created risks to free speech.

“When a single provider goes dark, critical services go offline with it,” Corinne Cath-Speth, head of digital for Article 19, a free speech advocacy group, said in a statement.

She added: “We urgently need diversification in cloud computing. The infrastructure underpinning democratic discourse, independent journalism and secure communications cannot be dependent on a handful of companies.”

Amazon’s cloud service is the most popular provider globally, underpinning a large chunk of the internet. It accounts for about a third of the cloud market, meaning any outage has major ripple effects.

Elon Musk, the owner of social media company X, said that his platform continued to work amid several apps seeing outage after AWS systems went down.

“X works,” he said, without commenting further.

The Amazon Web Services (AWS) has said that its system was seeing “massive signs of recovery”, Reuters reported.

Coinbase and Robinhood said that the AWS issue was affecting their services and Perplexity said the disruption had impacted the stability of its website. Other sites, including game sites Roblox and Fortnite and corporate messaging service Slack also had a spike in complaints about service disruptions that might be related to AWS, according to data from Down Detector.

AWS outage Live: Amazon confirmed error rates for multiple AWS services and said it was actively working to mitigate the issue and understand the root cause.

“We can confirm increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause,” AWS said.