Skip to content
Register

Outage postmortem: my single point of failure was me

General Discussion by techmei 7 replies 541 views
techmei

Member OP

#1

Hi all, want to share my mistake so you not make same.

Two server down yesterday. Not provider fault. MY fault. I have make config error in haproxy, send all traffic to dead backend. Uptime 99.9% gone because me. XD

# this was wrong
backend app
    server app1 10.0.0.5:8080 check
    # server app2 10.0.0.6:8080 check  <-- I COMMENT THIS OUT "TEMPORARILY"

Took 4 hour to find. Shame. But transparency important, so here I AM.

Also backup I think run daily? Will check now.

armstrongvds

Member

ARM Enjoyer

#2

Brutally honest post! Love it! That Ampere ARM box you're running haproxy on—insane value for the wattage! What specs? Runs circles around x86 for this workload!

one small ping for man...
zoekyo

Member

Less Is More

#3

Less moving parts, less grief
Two backends minimum
Test before deploy

rm -rf / --no-preserve-root ☯
Quentin

Member

#4

> Test before deploy
> This

> I COMMENT THIS OUT "TEMPORARILY"
> This

Petra3

Member

latency witch

10 #5

Actually, "I have make" is ungrammatical; "I made" or "I have made." Also, "two server" requires plural: "two servers." Your haproxy config lacks fallbacks. A second backend is not optional but essential. Semicolons would improve your commenting style. Most don't use them correctly.

liam_funky

Member

Vancouver

#6

1. Good on you for posting this
2. Lessons for everyone:
1. Never leave "temporary" changes uncommented
2. Always have second backend
3. Verify backups actually run
3. Questions:
1. What monitoring do you use?
2. Did pagerduty alert?
4. Waiting on backup status

annexbi

Member

1 #7

Oh this is (deliciously) familiar (I once took down production (at 2pm on a Tuesday (my last day at that job (they didn't ask me back)))) — anyway your haproxy config needs a backup directive (which you knew (deep down)) but the real horror (wait for it) is your backup situation (please tell me it's not)

push. done. coffee.
mediaaustin

Member

Deliverability Nerd

#8

What ampere box is that? Altra or the older one

SPF, DKIM, DMARC — holy trinity ✉️

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft