Skip to content
Register

DDoS scrubbing center bounced our clean traffic back

Networking by kenji3 25 replies 4.5K views
kenji3

Member OP

#1

I may be wrong, but I believe we hit a routing loop during activation of our new scrubbing service. Provider is Contabo, scrubbing center in Singapore. Clean traffic should return via GRE tunnel to our edge in Osaka.

Instead we saw our own prefixes re-announced with an unexpected community string. I have pcaps from both sides if anyone wants to look. Traffic never reached us, just looped between two scrubbing PoPs for about 90 seconds before their BCP kicked in.

I am not strong with BGP communities so I may be missing something obvious. Happy to share more details. This was yesterday 14:00 JST if it matters.

conbini > datacenter snacks
nerdbann

Member

DNS Propagator

#2

Check your resolver first, but more importantly let me trace this for you

From here (1.1.1.1 via 8.8.8.8 as backup) the path to your prefix dies at 203.78.x.x which is Contabo's Tokyo edge, not Singapore. TTL on the ICMPs I'm getting back is 248 which suggests... actually wait, that's the scrubbing center's own TTL, not your GRE endpoint.

kenji3 said:
I have pcaps from both sides if anyone wants to look

Your pcap will show whether the community string 64512:9999 is being attached by their route reflector. That's their "send to scrubbing" community and if it's leaking into the clean path you get exactly this loop. Propagation of the fixed route should be 300 seconds max with their stated TTLs.

dig soa contabo.com returns 3600 so their zone is fine, this is purely BGP.

WA

Member

#3

Why does a community string make it loop? I thought bgp was just... paths?

Also what is a pcap, is that like a packet capture? Sorry if dumb question

MeritBudi

Member

#4

I see already, the community string 64512:9999 is for dirty traffic. But the clean traffic also got same tag, so the router think still need scrubbing dong. Can or not you ask Contabo remove this community from return path?

I had similar issue with OVHcloud in Singapore before, their route reflector already misconfigured like this. Very headache.

ana_mad

Member

#5

MeritBudi is right! I sent ticket yesterday to Contabo about this exact community leak in their Singapore-Sydney path. They acknowledged in 4 hours, fixed in 8. Made test order with them last month and saw same behavior.

The route reflector was attaching 64512:9999 to all routes learned from scrubbing centers, not just inbound. Their engineering confirmed it was a template error from a new PoP turnup.

Kenji3, mention ticket #2025-SIN-BGP-0441 if you want, might help them connect the dots. They were very responsive once I got past tier 1.

Cheers!

swimming upstream since 2019 🐟
AL

Member

#6

Contabo again. 3rd loop this quarter. Ticket open 6 hours, no response. Went down again at 02:00. Their route reflector in singapore still broken.

nerdbann

Member

DNS Propagator

#7

WalnutLinux said:
Also what is a pcap, is that like a packet capture?

Yes, pcap = packet capture. Wireshark reads them.

kenji3 said:
I have pcaps from both sides

Kenji3, can you confirm whether the community is on the clean route or on the original announcement? If it's only on the return path then ana_mad's template error theory is almost certainly right. Their route reflector in Singapore shouldn't be touching anything that came out of scrubbing.

Also what does your GRE endpoint see? Any keepalives at all during the 90 seconds?

kenji3

Member OP

#8

nerdbann said:
Can you confirm whether the community is on the clean route or on the original announcement?

Clean route only. Original announcement from our router in Osaka has no communities attached, we don't use them.

GRE keepalives were dead the whole time. Tunnel interface stayed up because the loop was between their PoPs, not the path to us. We saw nothing.

ana_mad said:
Mention ticket #2025-SIN-BGP-0441

Thank you, I referenced this in my follow-up. Tier 1 actually escalated immediately when I did, now talking to someone who knows what a route reflector is.

Still no root cause given but they confirmed "a configuration issue affecting multiple customers in Singapore." No timeline.

conbini > datacenter snacks
ana_mad

Member

#9

kenji3 said:
They confirmed "a configuration issue affecting multiple customers in Singapore"

That's their script. I got the same line before they admitted the template error. Push for an incident number, not just a ticket. Makes it harder for them to bury.

Also check your next invoice carefully. I got 2 hours of "scrubbed traffic" billed to my account even though the traffic never reached me. Their metering doesn't know the loop happened. I disputed and won, but only because I had timestamps.

swimming upstream since 2019 🐟
MeritBudi

Member

#10

ana_mad said:
I got 2 hours of "scrubbed traffic" billed

Wah lau, this one also can? Contabo meter based on traffic to scrubbing center or traffic delivered? If loop never deliver then should not bill lor.

I check my OVHcloud bill last time, they meter by clean pipe egress only. More fair like that.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft