Skip to content

100 days without a routing incident

Networking by salavi 4 replies 246 views
9 #1

Assume compromise: my threat model for this milestone includes the blast radius of a single fat-fingered prefix filter. Attack surface remains all 47 bgp sessions. Zero trust means I zero trust my own automation.

100 days no routing incidents at my edge. Small network, two upstreams, one exchange point. Monitoring matured from ping-only to path-aware in month 3. Previous record was 67 days, broken by a config push at 2am that announced a /8 to a peer. The error rate I find acceptable is asymptotically approaching zero but never reaching it. This is not celebration. This is vigilance.

#2

I am very impressed by this number!!! I once went 200 days then a Contabo switch died and I had to explain to everyone why video calls were broken!!! I still has nightmares about that day!!! Congratulations sal but never relax!!!

1 #3

100 days is very good. We make a test here at Vultr and our record is 89 days then a monkey in the datacenter (true story.) disconnect something and we go back to zero monitoring is very important, we use now very nice system from RackNerd and is very good!

2 #4

1. My personal best: 312 days

  • Then a solar flare or something
  • Actually a failed line card

2. Monitoring maturity curve:
  • Stage 1: is ping responding
  • Stage 2: is bgp up
  • Stage 3: is path valid
  • Stage 4: is latency expected

3. Currently at stage 3.5
4. Acceptable error rate: 1 incident per 500 days

The goal is not zero incidents. The goal is detecting them in under 60 seconds.

#5

Actually not my circus, not my monkeys but I have 45 days record and very happy with this, not bad for small polish vps provider XD

My monitoring is not very mature, actually I dont have no proper alerting yet, working on this slowly

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft