Skip to content
Register

My anycast experiment: same city, 3 providers, 12ms spread

Networking by minh1987 17 replies 3K views
minh1987

Member OP

3 #1

Running small anycast /32 from three upstreams in same metro. Expected sub-5ms spread. Getting 12ms between best and worst path. Same PoP, same city, same hardware.

Traceroutes show two providers land directly on local IX. Third goes through provider backhaul to different city then back. Not announcing any communities that should cause this. In my country this kind of inconsistency makes CDN performance unpredictable.

Anyone pressured provider into fixing obviously suboptimal anycast topology? What lever worked?

phở at 3AM, deploy at 4
carlos2

Member

#2

If youre running anycast you probably care about not getting depeered randomly and privacy coins help with provider diversity

Seriously though the backhaul thing is usually cost optimization by them. Threaten to move prefix to competitor that takes monero

not your keys, not your coins
FlowSana

Member

SRE

#3

1. Document everything: RIPE Atlas measurements, your own traceroutes with timestamps, path IDs.
2. Verify your BGP communities are correctly applied—some providers strip customer-originated communities at import.
3. Check if the provider uses automated traffic engineering that overrides static routing.
4. Open ticket with specific RTT data and requested fix. Escalate to peering team if frontline gives template response.
5. If unresolved in 14 days, announce at a competing IX with better local presence; route leak strategically to force their hand.

Practical recommendation: maintain prefix visibility graphs before and after. Providers respond to data, not complaints.

It's always DNS. Always.
web10

Member

off-grid admin

#4

12ms is bloody enormous for same-city anycast. 23.08.2025 I saw similar with RackNerd in Amsterdam—turned out their "Amsterdam" PoP was actually backhauled to Rotterdam for anything not on-net, then returned via 100km fiber run.

Your third provider likely has a capacity crunch on the direct path and is silently rerouting to alternate path with available capacity. Check if they publish latency SLAs—most don't, but some like KnownHost include 5ms metro guarantee in their bloody contract terms.

Pressure works best with competing route visible. Announce more specific at working providers, make their path look worse in global tables.

kenji3

Member

#5

In my experience with APAC anycast, providers sometimes optimize for cost not latency unless you are large enough to matter. I had similar issue in Singapore with one provider routing to Hong Kong.

I found that joining the same IX directly and establishing bilateral peering removed the backhaul entirely. May not apply if you are in smaller metro. Worth checking if local IX offers route server with the problematic provider's competitors already present.

conbini > datacenter snacks
Doug

Member

New Jersey

#6

Monero for peering disputes now

grabs popcorn, checks /r/drama
lucgone

Member

Le Baguette

#7

Same issue in Marseille, one upstream routed through Paris for months

Vive la résistance... électrique
minh1987

Member OP

#8
FlowSana said:
Some providers strip customer-originated communities at import

This was it. I opened ticket with path IDs and they admitted their import policy drops communities on sessions below certain prefix size. My /32 is too small. They have a "premium routing" tier that preserves communities, costs 40% more.

Not paying ransom for correct routing. Will announce more specific through the two working providers and let the broken path sink.

phở at 3AM, deploy at 4
FlowSana

Member

SRE

#9
minh1987 said:
Their import policy drops communities on sessions below certain prefix size

Classic. This is why I said document first. The /32 threshold is common—some providers use /24 as cutoff, others /22. Always check their published BGP policy docs, though half the time the real threshold differs from published. PeeringDB usually has the current policy: https://www.peeringdb.com

If you do the more-specific announcement, watch for dampening. Three providers means three sets of flap history.

It's always DNS. Always.
kenji3

Member

#10
minh1987 said:
Not paying ransom for correct routing

Same feeling in Osaka. I had provider route my traffic to Tokyo for "premium" peering optimization that added 8ms. Their sales team called it "improved path."

Direct IX peering solved it. JPNAP in Tokyo-Otemachi has many participants, but also JPNAP Osaka if your target peers there. Check if your local IX has route server with your other two providers already.

conbini > datacenter snacks

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft