Skip to content

DDoS scrubbing center bounced our clean traffic back

Networking by kenji3 25 replies 4.5K views
#11

Sorry to jump in late, but I'm trying to understand why the community string doesn't get stripped when traffic comes back clean? Isn't that the whole point of a scrubbing center, to remove the bad stuff and then... remove the label too?

Or is the community not about the traffic itself but about the route announcement?

#12

nate_pad said:
Is the community not about the traffic itself but about the route announcement?

Exactly right. BGP communities are route labels, not packet labels. The traffic itself has no idea it was ever marked dirty. The routers decide where to send packets based on the prefix's current label.

So: dirty traffic arrives at your edge, your edge sees 64512:9999 and sends it to Singapore. Singapore scrubs it, sends it back clean. But if the return route still has 64512:9999, your edge says "oh this still needs scrubbing" and sends it to Singapore again. Loop.

The scrubbing center is supposed to strip that community before announcing the clean prefix back. Or their route reflector is supposed to not re-attach it. Someone forgot.

#13

6 hours now. Still no response to my ticket. Kenji3 you got escalated because you quoted ana_mad's number. I quoted it too and got told "that ticket is not available to reference." They are lying or their system is broken or both.

I am moving 2 VPS to OVHcloud Singapore this weekend. Done with Contabo routing — https://contabo.com/en/vps/.

#14

ALPHA16 said:
I am moving 2 VPS to OVHcloud Singapore

Be careful with OVHcloud Singapore if you need GRE. Their Singapore-Sydney path has had MTU issues with GRE for months. Not a loop but tunnel drops under load. Different problem, equally annoying.

I'm in Toronto so I use Beauharnois. No scrubbing there though, just raw bandwidth.

#15

This is why I don't let providers touch my routing. I announce my own prefixes from Manchester, if I need scrubbing I use a third party that just gives me a clean feed and I control the communities myself. More work, fewer surprises.

Contabo is cheap for a reason. You are the product and the QA department.

#16

kenji3 said:
This was yesterday 14:00 JST

I saw something weird from Osaka at that exact time. I run a small monitoring mesh here in Osaka and one of my targets behind Contabo Tokyo went unreachable from 14:02 to 14:04 JST. Only from my Singapore vantage point though, from Japan it was fine.

Probably unrelated but the timing is suspicious. Could their route reflector hiccup have propagated beyond just scrubbing customers?

#17

waqaszhang said:
Could their route reflector hiccup have propagated beyond just scrubbing customers?

Very possibly. We saw the re-announcement with unexpected community from two different Contabo peers, not just our scrubbing path. I assumed it was limited to the scrubbing service but if your Tokyo target was affected, maybe wider.

I don't have visibility into their full customer cone though. Anyone else see prefix weirdness from Contabo Singapore yesterday ~14:00 JST?

conbini > datacenter snacks
2 #18

I did a quick check from Frankfurt. traceroute to a Contabo-hosted site I know was going via Singapore for about 10 minutes around 06:00 UTC, which is 15:00 JST. Normally that path goes London-Nuremberg. So something was definitely shifting routes in their Asia network yesterday.

I don't have an account with them so I can't see more. Just a data point.

5 #19

Following this thread closely. I was about to order Contabo scrubbing for a project in Mumbai. Now I'm hesitating.

ana_mad said:
Check your next invoice carefully

This is the part that worries me most. I can handle a 90-second loop. I can't handle arguing about phantom traffic charges for weeks.

Does anyone know if their Mumbai scrubbing center has the same setup? Same route reflector or separate?

5 #20

shellking5 said:
Does anyone know if their Mumbai scrubbing center has the same setup?

Mumbai is newer than Singapore, turned up late 2024 if I remember. Different hardware generation but I wouldn't trust the templates. They probably copy-paste.

If you do go ahead, test with a /24 you don't care about first. And set up your own monitoring with something like Pingdom or UptimeRobot, don't trust their status page. I learned that the hard way.

swimming upstream since 2019 🐟

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft