Skip to content

My anycast experiment: same city, 3 providers, 12ms spread

Networking by minh1987 17 replies 3K views
#11
minh1987 said:
Costs 40% more

Forty percent for not breaking your communities. That's not premium routing, that's protection money. I run solar-powered setup in Lisbon and even my diesel backup doesn't markup that hard.

Announce the /33 or /34 more specific through working providers. Make their aggregate look like the long path it is. Worked for me with a provider in Amsterdam that wanted to backhaul to Frankfurt.

#12
FlowSana said:
Watch for dampening

In Bangalore I see this constantly. Providers here love to dampen /32 and /33 on international sessions. One upstream I use has 15-minute penalty for anything longer than /24.

My workaround: announce /24 covering aggregate with no-export, then /32 with standard communities. The aggregate prevents dampening from blackholing you entirely, no-export keeps it from polluting global tables.

RIPE Atlas is your friend here. I run 50 probes in APAC, cost is negligible for the data quality.

#13
minh1987 said:
Costs 40% more

This is why I self-custody billing. Provider sees you as captive, they extract rent. Monero means they cannot freeze your payment method when you dispute.

But practically: the more-specific route leak is correct answer. Force their hand in global routing tables where their marketing claims meet reality — check https://bgp.tools.

not your keys, not your coins
9 #14

Following this thread closely. Chicago anycast to ORD is usually solid but I have seen backhaul to DEN on one provider during peak. Never got a straight answer why.

Minh1987 what monitoring are you using to catch the 12ms spread? SmokePing? I need something better than my current setup.

#15
WalnutDude3 said:
What monitoring are you using

SmokePing for baseline, but the real tool is ripe-atlas-tools with custom measurements. I have 20 probes in my metro running ping and traceroute every 15 minutes. The path ID tracking is manual from traceroute output—looking for same router names, same IX appearances. https://www.peeringdb.com is useful for cross-checking those IX guesses.

Also wrote small script that parses MTR JSON and alerts when median RTT to same anycast IP from same probe jumps by more than 3ms. Caught this issue within 6 hours of provider change.

phở at 3AM, deploy at 4
#16
minh1987 said:
20 probes in my metro

In Auckland I am lucky to have 3 RIPE Atlas probes total. The density problem is real outside major hubs.

I pay for Vultr instances in Sydney and Melbourne as synthetic probes. Not perfect but better than nothing. Their Sydney PoP is actually in Sydney, unlike some providers who backhaul to Melbourne for "Oceania optimization."

#17
web10 said:
Make their aggregate look like the long path it is

In Frankfurt this works but watch for BGP hijack complaints. I did similar with a provider backhauling to Düsseldorf, announced /34 through DE-CIX direct peers. Their NOC sent automated abuse ticket about "route leak."

Had to explain that my more-specific was legitimately originated from my ASN, not a leak. Took 3 days. Document your RIR registration and LOA before you do this.

#18

Port 22 open to world but worried about 12ms routing asymmetry. Priorities.

airgapped, encrypted, faraday'd, still worried

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft