Leaked screenshots show Amazon blaming AWS outage on sudden surging traffic from ‘unknown source’

<div style&equals;"text-align&colon;center"><img src&equals;"data&colon;image&sol;gif&semi;base64&comma;R0lGODlhAQABAIAAAAAAAP&sol;&sol;&sol;ywAAAAAAQABAAACAUwAOw&equals;&equals;" fifu-lazy&equals;"1" fifu-data-sizes&equals;"auto" fifu-data-srcset&equals;"https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;75&resize&equals;75&ssl&equals;1 75w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;100&resize&equals;100&ssl&equals;1 100w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;150&resize&equals;150&ssl&equals;1 150w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;240&resize&equals;240&ssl&equals;1 240w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;320&resize&equals;320&ssl&equals;1 320w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;500&resize&equals;500&ssl&equals;1 500w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;640&resize&equals;640&ssl&equals;1 640w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;800&resize&equals;800&ssl&equals;1 800w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;1024&resize&equals;1024&ssl&equals;1 1024w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;1280&resize&equals;1280&ssl&equals;1 1280w&comma; https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1&w&equals;1600&resize&equals;1600&ssl&equals;1 1600w" width&equals;"1920" height&equals;"1080" fifu-data-src&equals;"https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1" class&equals;"attachment-post-thumbnail size-post-thumbnail wp-post-image" alt&equals;"Leaked screenshots show Amazon blaming AWS outage on sudden surging traffic from &OpenCurlyQuote;unknown source’" title&equals;"Leaked screenshots show Amazon blaming AWS outage on sudden surging traffic from &OpenCurlyQuote;unknown source’" srcset&equals;"https&colon;&sol;&sol;i3&period;wp&period;com&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans&period;png&quest;ssl&equals;1 1920w&comma; https&colon;&sol;&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans-300x169&period;png 300w&comma; https&colon;&sol;&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans-1024x576&period;png 1024w&comma; https&colon;&sol;&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans-768x432&period;png 768w&comma; https&colon;&sol;&sol;www&period;awordpresscommenter&period;com&sol;wp-content&sol;uploads&sol;2021&sol;12&sol;read-more-on-captionwire&period;com-log-trans-1536x864&period;png 1536w" sizes&equals;"&lpar;max-width&colon; 1920px&rpar; 100vw&comma; 1920px" &sol;><&sol;div><p>What we know&colon;<&sol;p>&NewLine;<div data-component-type&equals;"content-lock" data-load-strategy&equals;"exclude">&NewLine;<div class&equals;"content-lock-content">&NewLine;<ul class&equals;"summary-list summary-list-variant">&NewLine;<li>Amazon Web Services experienced an outage on Tuesday that caused other apps&comma; like Netflix&comma; Coinbase&comma; and Tinder to see slower services&period;<&sol;li>&NewLine;<li>An internal AWS analysis pointed to traffic congestion across multiple network devices in the Northern Virginia region&period;<&sol;li>&NewLine;<li>But employees are still trying to figure out the root cause of the problem and seem confused by the depth of the outage&comma; which has affected all parts of AWS&&num;8217&semi; global operations in some capacity&period;<&sol;li>&NewLine;<&sol;ul>&NewLine;<p>As Amazon Web Services experienced one of the largest <a href&equals;"https&colon;&sol;&sol;www&period;businessinsider&period;com&sol;amazon-servers-are-down-and-its-affecting-dozens-of-services-2021-12" data-analytics-module&equals;"body&lowbar;link">outages<&sol;a> in company history on Tuesday&comma; more than 600 employees joined an emergency conference call to assess the root cause of the service disruption&period;<&sol;p>&NewLine;<p>The main culprit&colon; a sudden increase in traffic that caused congestion across multiple network devices in Northern Virginia&comma; the biggest region for AWS data centers&period;<&sol;p>&NewLine;<p>The company had initially pegged the &&num;8220&semi;root cause&&num;8221&semi; of the outage on &&num;8220&semi;a problem with several network devices within the internal AWS network&comma;&&num;8221&semi; according to a screenshot of an internal AWS communique from Tuesday morning obtained by Insider&period; &&num;8220&semi;Specifically&comma; these devices are receiving more traffic than they are able to process&comma; which is leading to elevated latency and packet loss for the traffic traversing them&period;&&num;8221&semi;<&sol;p>&NewLine;<p>The problems are ongoing as of Tuesday afternoon&comma; and have resulted in hours of service disruptions across the web&comma; causing some of the world&&num;8217&semi;s biggest online services&comma; including Disney Plus&comma; Netflix&comma; and even Amazon&&num;8217&semi;s own e commerce store to experience widespread glitches and slowdowns&period; The list of companies that saw outages on Tuesday include Spotify&comma; Zoom and Airbnb&comma; to name a few&period;<&sol;p>&NewLine;<section class&equals;"content-recommendations-in-post-module-1"><&sol;section>&NewLine;<p>While the outage was linked to a disruption in Northern Virginia&comma; it has disrupted all parts of AWS&&num;8217&semi; global operations in some capacity&period; Moreover&comma; Amazon&&num;8217&semi;s retail and delivery networks&comma; which rely on AWS&&num;8217&semi; tools&comma; were in some cases thrown into a screeching halt&period;<&sol;p>&NewLine;<p>The outage snarled Amazon&&num;8217&semi;s internal warehousing and logistics operations in the midst of the holiday shopping season&period; Some warehouse workers and drivers were sent home as the company&&num;8217&semi;s internal communications&comma; delivery routing and monitoring systems stalled&period;<&sol;p>&NewLine;<p>The network issue &&num;8220&semi;specifically impacted&&num;8221&semi; Amazon&&num;8217&semi;s internal DNS servers&period; As of 2&colon;04 p&period;m&period; Seattle time&comma; the company did not have an estimate on when the system would be fully operational&comma; according to a message on <a href&equals;"https&colon;&sol;&sol;status&period;aws&period;amazon&period;com&sol;" target&equals;"&lowbar;blank" rel&equals;"nofollow noopener" data-analytics-module&equals;"body&lowbar;link">the public AWS status console<&sol;a>&period;<&sol;p>&NewLine;<p>A separate internal note said &&num;8220&semi;firewalls are being overwhelmed by an as of yet unknown source&comma;&&num;8221&semi; adding that the AWS networking teams are working on &&num;8220&semi;blocking the traffic from the top talkers&sol;offending hosts at the firewall&period;&&num;8221&semi;<&sol;p>&NewLine;<p>Activity from Amazon&&num;8217&semi;s real-time digital advertising auction may be responsible for much of the traffic overwhelming the firewall&comma; according to internal Slack messages seen by Insider&period;<&sol;p>&NewLine;<p>In an email to Insider&comma; Amazon&&num;8217&semi;s spokesperson said&comma; &&num;8220&semi;There is an AWS service event in the US-East Region &lpar;Virginia&rpar; affecting Amazon Operations and other customers with resources running from this region&period; The AWS team is working to resolve the issue as quickly as possible&period;&&num;8221&semi;<&sol;p>&NewLine;<p>Even inside AWS&comma; however&comma; information on the outage remains sketchy&period; As engineers and executives worked decode the issue on a 600-person conference call&comma; led by AWS&&num;8217&semi;s VP of infrastructure Peter Desantis&comma; rumors spread among staff&period; One AWS employee speculated that the outage was caused by an &&num;8220&semi;orchestrated DNS attack&comma;&&num;8221&semi; while another employee downplayed those concerns&comma; saying it was more of an &&num;8220&semi;internal thing&&num;8221&semi; related to networking and firewall saturation&period;<&sol;p>&NewLine;<p>&&num;8220&semi;It&&num;8217&semi;s the fog of war&comma;&&num;8221&semi; said an AWS manager&period;<&sol;p>&NewLine;<p>In a message sent just before 2&colon;00 p&period;m&period; Pacific Time&comma; the company&&num;8217&semi;s internal communications team told employees it was &&num;8220&semi;beginning to see significant recovery for AWS service availability in the US-EAST-1 Region&period;&&num;8221&semi; The division&&num;8217&semi;s &&num;8220&semi;most senior engineers&&num;8221&semi; are continuing to monitor the issue&comma; including &&num;8220&semi;identifying the specific traffic flows that were leading to congestion within these devices&comma;&&num;8221&semi; the note said&period;<&sol;p>&NewLine;<section class&equals;"content-recommendations-in-post-module-fixed"><&sol;section>&NewLine;<&sol;div>&NewLine;<&sol;div>&NewLine;

Sponsored
Sponsored
A WordPress Commenter

Recent Posts

Chasing the Sun: Inside WP Accessibility Day’s 24-Hour Event & What to Expect in 2025

When WP Accessibility Day kicks off next week, someone, somewhere will always be awake. As…

1 hour ago

Performance Chat Summary: 7 October 2025

The full chat log is available beginning here on Slack. WordPress Performance Trac tickets @westonruter…

23 hours ago

WP Engine Accuses Automattic and Matt Mullenweg of Concealing Control of WordPress.org and the WordPress Trademarks

WP Engine has filed a whopping 175-page revised complaint in its ongoing legal fight with…

2 days ago

Woo Outlines AI Plans: MCP Support and Integration With Stripe’s Agentic Commerce Protocol

Woo has shared some early details on what it’s working on in the AI space.…

5 days ago

Ollie’s Menu Designer Lands on WordPress.org, Praised as “Game Changer”

Ollie’s Menu Designer plugin is now available for download on WordPress.org. Originally launched in August…

6 days ago

WordCamp Organizer Banned for Life After Misappropriating Funds

The WordPress Community Team has banned a WordCamp organizer for life after they used $734…

6 days ago