Skip to content

Poll: worst way you've discovered your prefix was hijacked?

Networking by RAJ 4 replies 215 views
#1

Guys, curious about your horror stories. Any leads on how the small operators here find out first? Is it the RIR alerts, your NMS, or some random email from a dude in another continent?

I ask because I recieved a notice 6 hours late last month and it already killed my margins for that quarter

Personally my worst was a customer calling to say their site showed a Vultr parking page. Turned out someone leaked my /24 to a friend at a party. No joke

your margin is my opportunity
#2

¡my mom called asking why her knitting blog was in Russian!

Is very serious jajaja I had to explain BGP to a 67 year old woman who still frows her own socks. She was very calm, said "the graphs look prettier now" because cyrillic uses more loops jajaja

I deleted my first draft, was too long. The short version: RIPE stat page — https://www.ripe.net — her neighbor reads russian, neighbor told her, she told me. No monitoring caught it. 4 hours.

Very embarassing. My own mother. My own prefix. Very bad day.

#3

@pablowild the graphs do not lie but apparently the grandmothers see things before they do

My worst: 99.97% uptime on the status page, green across the board, alert fatigue had me ignoring PagerDuty for 3 hours because "it always recovers"

The actual hijack was visible on every collector. The graphs do not lie. I was just too tired to look. 6 hours of Russian pharma SEO on a /22. Status page showed green because my probes were hitting the hijacker's identical-looking landing page.

Now I check path changes independently. The graphs do not lie but you have to actually graph the right thing.

436 days. reboot is surrender.
#4
pablowild said:
My mom called asking why her knitting blog was in Russian

This is the winner. No contest.

- Human detection beats automated systems
- No TOS violation on the hijacker's part for the content itself
- The notice period was effectively zero
- Your own family became the monitoring layer

I have documented three similar cases. In each instance, the operator had:

  • Redundant alerting that was ignored
  • A false sense of collector coverage
  • No out-of-band page content verification

RR

if it ain't broke, rust it
#5

Fighting! My worst was NO ALERT AT ALL

Just some dude on a korean forum posting "is Hetzner down" and someone else saying "no it loads for me"

TURNED OUT THE SECOND GUY WAS IN RUSSIA

3 DAYS guys

3 DAYS

No periods just trauma kk

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft