The day was Tuesday, August 12th 2014. I arrived home, only to find an almost unusable internet situation in my home. Some sites such as AnandTech and Google worked fine, but large swaths of the internet such as Microsoft, Netflix, and many other sites were unreachable. As I run my own DNS servers, I assumed it was a DNS issue, however a couple of ICMP commands later and it was clear that this was a much larger issue than just something affecting my household.
Two days later, and there is a pretty clear understanding of what happened. Older Cisco core internet routers with a default configuration only allowed for a maximum 512k routes for their Border Gateway Protocol (BGP) tables. With the internet always growing, the number of routes surpassed that number briefly on Tuesday, which caused many core routers to be unable to route traffic.
BGP is not something that is discussed very much, due to the average person never needing to worry about it, but it is one of the most used and most important protocols on the internet. The worst part of the outage was that it was known well in advance that this would be an issue, yet it still happened.
Let us dig into the root cause. Most of us have a home network of some sort, with a router, and maybe a dozen or so devices on the network. We connect to an internet service provider through (generally) a modem. When devices on your local network want to talk to other devices on your network, they do so by sending packets upstream to the switch (which is in most cases part of the router) and then the switch forwards the packet to the correct port where the other device is connected. If the second device is not on the local network, the packets get sent to the default gateway which then forwards them upstream to the ISP. At the ISP level, in simple terms, it works very similarly to your LAN. The packet comes in to the ISP network, and if the IP address is something that is in the ISP’s network, it gets routed there, but if it is something on the internet, the packet is forwarded. The big difference though is that an ISP does not have a single default gateway, but instead connects to several internet backbones. The method in which internet packages are routed is based on the Border Gateway Protocol. The BGP contains a table of IP subnets, and lists which ports to transfer traffic based on rules and paths laid out by the network administrator. For instance, if you want to connect to Google to check your Gmail, your computer will send a TCP connection to 173.194.33.111 (or another address as determined by your DNS settings and location). Your ISP will receive this packet, and send the packet to the correct port to an outbound part of the internet which is closer to the subnet that the address is in. If you then want to connect to a domain name or website hosting like Anandtech.com, the packet will be sent to 192.65.241.100, and the BGP protocol of the ISP router will then send to possibly a different port. This continues upstream from core router to core router until the packet reaches the destination subnet, where it is then sent to the web server.
- Top 18 Best Web Hosting Services In 2024: Despite there are multiple web hosting providers available in the market, choosing the best one is really a hard nut to crack!! Because almost all service providers are offering their services with the same features and quality.
- How to start your own Crypto currency website: For a cryptocurrency news website, choosing the right Content Management System (CMS) is crucial for managing content effectively, ensuring security, and providing a great user experience. Here are some of the best CMS options tailored for such a site, each with its unique strengths:
- The Evolution of Coffee Brewing Methods: A Historical Perspective: Coffee has been brewed and enjoyed for centuries, with various methods evolving over time to create the perfect cup of joe. From early methods like the Ibrik method and coffee pots for coffee houses to modern technologies like the AeroPress and Hario V60, the world of coffee brewing has come a long way.
- Stop WordPress from Creating Extra Cropped Image Sizes: Whenever you upload an image to your WordPress site through the media library, it automatically creates and stores multiple additional versions of that image. If your site doesn’t utilize these extra image sizes, they will consume valuable storage space and increase the size of your server backups.
- Explain WordPress Portfolio and how to created them: WordPress portfolios are digital showcases for individuals or businesses to display their work, projects, or professional accomplishments. This feature is particularly useful for creatives, freelancers, and agencies looking to exhibit their skills or services in a visually appealing and organized manner. Portfolios in WordPress can be created using various methods, including themes that come with built-in portfolio functionalities, plugins that add portfolio features, or custom coding for a more personalized approach.
- 15 Essential WordPress Plugins For Every Site: Plugins are essential for every type of website, as they provide additional features and customization options that are not available in the core WordPress platform. A study by CodeinWP found that the average WordPress site has 20 active plugins, indicating the importance of plugins in website development.
- SAILING and YACHTING: Awesome Sailing Vlogs for the Enthusiast
- Comparing Google and Microsoft’s Success in Capitalizing on Generative AI: The buzz of interest in AI services helped drive revenue for Microsoft’s biggest unit, cloud services—up by 7 percentage points compared to a year ago—and Microsoft’s overall sales rose 17 percent to nearly $62 billion. It also gained cloud market share, Nadella added. The number of $100 million cloud deals that Microsoft landed increased 80 percent during the quarter compared to the same period a year ago, and $10 million deals doubled.
- The Best WordPress Hosting Solution in Australia: Each of our WordPress hosting solutions are fine-tuned, blazing fast and are ready for you! Starting a WordPress website has never been easier with our free 1-click WordPress installation, enterprise-grade security and an assortment of tutorials and helpful guides to get you started, all backed by our 99.9% uptime guarantee.
- Elementor vs Beaver Builder: A Comparison of Design Flexibility and Performance in WordPress: Elementor and Beaver Builder are two of the most widely recognized options, each providing distinct features that cater to varying user requirements. This article conducts a comprehensive comparison of these tools, examining their design flexibility, performance metrics, and overall user experience.
- The Best Contact Form Plugins for WordPress to Easily Manage User Inquiries: In an effort to improve user interactions on WordPress websites hosting, the examination of contact form plugins becomes essential. This article aims to present an overview of the top 5 contact form plugins available for WordPress, highlighting their features and pricing structures to facilitate an well-considered choices process.
- What are the 20 best Joomla plugins: Joomla plugins are small, task-specific extensions that enhance or modify the core functionality of a Joomla website. They operate as event-driven scripts, listening for specific “events” triggered by Joomla or its components and executing corresponding actions. Plugins are a crucial part of Joomla’s extensibility, allowing developers to add features or customise behaviour without altering the core Joomla code.
- How to design a strong off-page SEO strategy: Lessons learnt from earning over 50,000 contextual links for thousands of websites in the toughest niches. In 2023, virtually every business that has a website is prioritizing their investment in SEO. The reason being, amidst the chaos of the current year, customers are resorting to online channels for safe purchasing, and optimizing their online presence is a surefire way to gain an edge over rivals. While some firms have set up internal teams, and others have hired external agencies to boost their search engine rankings, a documented off-page SEO strategy is still a rarity among most businesses.
- What are WordPress Plugins? WordPress plugins are modular pieces of software that can be added to a WordPress site to extend or enhance its functionality without modifying the core WordPress code. They allow website owners to add features, improve performance, and customise the behaviour of their websites easily, catering to a wide range of needs, from SEO and security to e-commerce and design enhancements.
With the BGP tables being overfilled on certain routers in the chain, packets send to specific routers would then be dropped at some point in the chain, meaning you would not have any service.
The actual specifics of what happened seemed to be that Verizon unintentionally added approximately 15,000 /24 routes into the global routing table. These prefixes were supposed to be aggregated, but this didn’t happen, and as such, the total number of subnet prefixes in the table spiked. Verizon fixed the mistake quickly, but it still caused many routers to fail.
Although you could be quick to jump and blame Verizon for the outage, it has to be noted that Cisco issued a warning to customers explaining that the memory which is allocated for the BGP table would be very close to being full, and gave specific instructions on how to correct it. This warning came several months ago. Unfortunately not all customers of Cisco heeded or received the warning, which caused the brief spike to cripple parts of the internet.
Newer Cisco routers were not affected, because the default configuration for the TCAM memory which is designated for the BGP table allows for more than 512,000 entries. Older routers from Cisco have enough physical memory for up to 1,000,000 entries, assuming the configuration was changed as outlined by Cisco.
The effects of outages like this can be quite potent on the internet economy, with several online services being unavailable for large parts of the day. However this outage doesn’t need to happen again, even though the steady state number of entries in the BGP table will likely exceed magic 512,000 number again. Hopefully with this brief outage, lessons can be learned, and equipment can be re-configured or upgraded which will prevent this particular issue from rearing its head again in the future.
