AWS spent years rebuilding its routing control plane without taking the network down
Every AWS API call, CloudFront video stream, and Route 53 lookup crosses the same infrastructure, which AWS calls its border network. It now runs on a routing system rebuilt from scratch over several years.
The system that tells traffic where to go
The scale AWS reports, current as of the post: 39 regions, 123 availability zones, over 750 points of presence on six continents, peering with more than 5,000 outside networks, hundreds of terabits per second.
Running underneath is the routing control plane, which keeps every device supplied with an accurate account of what destinations exist and how to reach them. AWS reaches for an air traffic control analogy: gather position reports, work out safe routes, broadcast instructions.
That picture has to be rebuilt whenever a link fails or a new path appears, a process engineers call convergence. Until it finishes, parts of the network can disagree about the best path, and packets can arrive late, go the wrong way, or vanish. AWS points to financial trading, telemedicine, and live media as cases where a few seconds of inconsistency can mean failed transactions or interrupted service.
Why AWS had a problem
The network grew in stages, each upgrading hardware and software. One consequence: different parts of it came to run their own control planes, independent of one another but interconnected. Each was reliable alone. Holding them in agreement across hundreds of locations was not.
They converged at different speeds. In the gaps, one part of the network could still be sending traffic down a path another had already written off. AWS says the result can be a routing loop, packets going in circles until they expire, or traffic aimed at a path that no longer exists. Customers get packet loss and higher latency from a routing state that has not settled.
“We recognized that adding more coordination between independent systems would only increase complexity. The more systems that needed to agree, the more overhead was required to keep them in sync. Knowing that simplicity scales, we decided to replace the independent control planes with a single unified system that provides one consistent view of the network to every device, converges faster, and contains the scope of any single change,” AWS experts explained.
Three changes
First, route sharing runs one direction only. Conventionally routers both learn from and advertise to their neighbors, so any device’s picture is secondhand. AWS split the halves apart. Collection nodes take routes from locally attached devices and push them outward; distribution nodes take in remote routes and pass them to local devices. Neither re-advertises anything learned elsewhere, so every device gets each route from its origin.
Second, tunneling. The original packet is wrapped in an outer packet carrying the routing instructions, which intermediate devices follow without inspecting the contents. Traffic then rides the path the control plane chose, insulated from devices still working off stale information.
Third, all of it went network-wide, replacing the independent systems with a single source of truth.
End-to-end tunnels between fabrics (Source: AWS)
What it took, and what came of it
The migration spanned years and thousands of devices on a live network, reaching internet-facing routers, the backbone, Direct Connect, AZ-facing nodes, and edge services including CloudFront and Route 53. AWS says availability held throughout, helped by a validation system that analyzed configuration changes before they reached production.
The unified system carries several times more compute for routing decisions than what it replaced, which AWS says contributed directly to faster convergence. Convergence time on some fabrics improved by up to 96 percent. AWS also reports fewer retries and more predictable latency for customers, and one architecture for engineers to troubleshoot. The entire border network now runs on it.
Download: The ultimate guide to network operations management
