Serious Times — Serious Leaks
At 17:13 UTC on March 31, 2020, the AS50048 (NEWREAL-AS) leaked, in total, 2658 IPv4 network prefixes to the Tier-2 transit provider Transtelecom. Those prefixes included Orange, Akamai, Rostelecom and more than 300 other companies’ networks.
This route leak, in particular, is significant because of its spread. There were several waves, containing separate subnetworks. In fact, the total amount of separate IP addresses (approximately 13.5 million) was even higher than in the largest wave! Due to the periodic nature of the session's capture (and keeping in mind that each wave contains mostly unique prefixes), we suppose that Transtelecom has a route rate limit set with each customer. This can be one of the reasons why this incident didn't transform into an epic disaster — just the simple rate limit.
Compared to the day-to-day BGP operations, such a route leak could constitute a critical threat to the functioning of computer networks all over the world, while those networks are already under stressful conditions by almost worldwide quarantine.
Right now, we can only guess why exactly this particular route leak occurred and spread so far. The owner of the AS50048 has made some mistakes in network configuration, although it's still unclear how NEWREAL-AS upstream AS20485 could accept and further propagate those prefixes. We know that the NEWREAL-AS connection with this provider has been established long ago; however, during this incident, we observed several session restarts due to congestion.
As we've already written, due to the — almost worldwide quarantine measures against the coronavirus pandemic, there is a forceful redistribution of vast amounts of traffic between users endpoint connections. Quite simply, most of the people have already moved away from offices and their corporate networks. They are working from home, which causes significant traffic redistribution, changes in the traffic profile, and the overload of transit and peering facilities, globally.
In such a situation, the majority of ISPs and operators would be working sleepless nights, holding things together with a piece of chewing gum and string, quickly fixing and re-architecting their networks and peering facilities. This inevitably leads to human errors and, given the current sad state of the BGP security, could cause and, as we can see, already caused disrupting consequences.
We strongly warn everyone that this would certainly not be the last serious incident during the coronavirus crisis.
Update: Diverted Netherlands to Brasil traceroute example
Before: trace from Amsterdam to Brazil at 07:46 Mar 30, 2020 1 * 2 22.214.171.124 NL-AMSIX-PEERINGLAN1 Amsterdam Netherlands 0.786 3 126.96.36.199 nl-sar.nordu.net Amsterdam Netherlands 1.416 4 188.8.131.52 uk-hex.nordu.net Amsterdam Netherlands 5.556 5 184.108.40.206 linx.lon.seabone.net London United Kingdom 88.265 6 220.127.116.11 TI Sparkle Seabone POP Fortaleza Brazil 257.636 7 18.104.22.168 TI Sparkle Seabone POP Fortaleza Brazil 253.064 8 22.214.171.124 T-NET WIRELESS E INFORMÁTICA Itaitinga Brazil 253.902 During: trace from Amsterdam to Brazil at 17:15 Mar 31, 2020 1 * 2 126.96.36.199 NL-AMSIX-PEERINGLAN1 Amsterdam Netherlands 12.617 3 188.8.131.52 kzn06rb.transtelecom.net Kazan Russia 71.279 4 184.108.40.206 Chuenkov-gw.transtelecom.net Voskresensk Russia 71.38 5 220.127.116.11 Russia Moscow, GOLDEN Telecom Moscow Russia 68.692 6 * 7 18.104.22.168 TI Sparkle Seabone POP Fortaleza Brazil 249.881 8 22.214.171.124 T-NET WIRELESS E INFORMÁTICA Itaitinga Brazil 246.094
"Active measurement helps to illustrate the traffic misdirection that takes place when a routing leak occurs. Traceroutes show the impact on latency and the geographic detour that simply can’t be understood from BGP alone. In this example, traffic from AMSIX to Brazil is inadvertantly misdirected through central Russia as a result of AS50048's leak."
Quote provided by Doug Madory, Director of Internet Analysis, Oracle Internet Intelligence.