Skip to content
Register

Anycast experiment: three continents, one /32, total chaos

Networking by prague1983 7 replies 513 views
prague1983

Member OP

btw I use Arch

#1

Last month I decided anycast DNS was "easy" and deployed a /32 across three providers:

  • HostHatch: Amsterdam, $3/month
  • CloudCone: Los Angeles, $4/month
  • RackNerd: Amsterdam, $5/month

All running BIND, same zone, BGP announced to my upstream. What could go wrong?

Everything. CloudCone leaked my /32 to a peer in Los Angeles who preferred it over their own customer route. Los Angeles went unreachable from half of the US West Coast because CloudCone's upstream deaggregated poorly. Amsterdam worked fine, which was the worst part—I kept thinking I understood the problem.

I spent 72 hours in routing tables. Traceroutes that looped. DNS responses from Los Angeles to European users with 340 ms latency because some Tier 1 decided that was "optimal."

Then I checked my metrics. 40% of users had lower latency than my old unicast setup. The chaos found paths I never would have engineered. Los Angeles leak? East Asia saw 15 ms improvement. US West's mess? Southern California got better peering.

I kept it running. I hate it. It works.

https://bgp.tools

oops: 0000 [#1] SMP
pam3

Member

Latency Witch

#2

Observed routing instability with scientific interest. The Hong Kong leak—observed path length decrease of 2.3×10^0 hops versus expected, with latency reduction of 1.5×10^2 ms.

RFC 4786[1] discusses anycast deployment considerations. Your experience confirms Section 4.2 regarding unintended path attraction.

[1] RFC 4786, "Operation of Anycast Services"

100% packet loss at hop 3
Berlin1

Member

1 #3

This is someoneelsesproblem engineering at its finest. You did not Design the chaos, you Inherited it. The capitalized Nouns make it sound Intentional.

My own anycast attempt with three providers ended when one upstream decided to blackhole the entire /24 because of a "routingpolicydispute" that was actually them forgetting to update their IRR. Footnotes[1]

[1] never trust a provider whose lookingglass runs on http without redirect

minhdock

Member

#4

Prague1983 san

1. You use BIND, very old
2. I use knot, much better
3. I try anycast one time, my upstream say "no" and nullroute
4. You lucky, very lucky
5. I have question: how you get /32? My provider give only /128 for IPv6, no IPv4 anycast

I want try chaos too, very much

PetraSuper

Member

network boot believer

#5

Hej, I see you have malý server in Amsterdam. I try anycast too, on server into RackNerd, but no work. "no" I mean, route not propagate. I have friend on OVHcloud, he say "use BGP community string" but I not understand. Into config I put nothing and expect magic.

Your chaos is better than my silence. I keep malý server for now.

boot anything, anywhere, anytime
CR

Member

#6

Which bind version, 9.16 or 9.18

WO

Member

#7

€3/month for BGP session?

Hel

Member

Bare Metal Purist

#8

Had the same hong kong leak once.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft