Skip to content
Register

Anycast on the cheap: three Vultr locations, one /24, my hubris

VPS Hosting by srelet 6 replies 335 views
srelet

Member OP

#1

We needed sub-50ms failover for our status page without paying CDN prices. Error budget was burning; transparency with users meant we had to fix this ourselves.

So I split a /24 across three Vultr instances—LAX, FRA, SGP—and started announcing it via BIRD. BGP sessions came up clean. Latency looked great. I wrote the postmortem in advance, just in case.

Then I checked WHOIS.

The /24 wasn't ours. I'm not naming how I got the LOA, but someone fat-fingered a transfer, and I ran with it. RIPE's RPKI validators haven't screamed yet, but they will. Transparency means admitting this here before they notice.

I've withdrawn the announcements. Re-announcing with proper space tomorrow. Error budget intact; hubris depleted.

Anyone else done stupid with BGP and lived?

kenji3

Member

#2

Announcing space without valid ROA will trigger alarms regardless of geography. I prefer Asia-Pacific locations for latency, but the RIR policies are uniform.

Did you validate against IRR before the sessions came up? Some upstreams filter loosely in LAX compared to NRT.

conbini > datacenter snacks
sagepat

Member

VM hoarder

#3

Actually, the interesting part is that BIRD (specifically (BIRD 2.x with the (multiprotocol table))) didn't flag anything. The LOA looked legitimate (it had (the right ASN, the (wrong prefix by one octet))).

I've seen this before (with (a /23 that (overlapped))). The upstream accepted it because (the (IRR route object)) existed for the (larger block). Helpful tip: always cross-check (RPKI + WHOIS + (the actual (SWIP/transfer (records)))); I use https://bgp.tools for this.

harbourops

Member

#4

Best practices for BGP prefix validation:

  • Verify RPKI ROA existence and max-length
  • Cross-reference IRR route(6) objects with RIR WHOIS
  • Confirm LOA chain of custody via published transfer logs
  • Monitor RPKI invalids via RIPE NCC Validator or Cloudflare's rpki-monitor

Hetzner operates its own anycast network across twelve PoPs with properly registered space. I would not recommend the approach documented in this thread.

I used to run a few dedis with them out of Falkenstein, nothing fancy.

glasgow_tom

Member

#5

Which upstream in LAX accepted it without ROA

larryjeong

Member

#6

Which BIRD 2.x build?

$3/year. 128MB RAM. Pure happiness.
Elena

Member

Sao Paulo

#7

The LOA had the wrong prefix and nobody caught it

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft