larryjeong
Member
OP
- Joined:
- Jun 2024
- Posts:
- 179
- From:
- Texas, US
So I was ssh'd into both boxes, prod and staging, same tmux color scheme on both cause I was too cheap to pay for a fancy terminal theme. Was cleaning up old logs and typed rm -rf /opt/app/* and watched it happen on the WRONG PANE. $5/month vps for staging, $200/month dedi for prod, both looked identical. Heart stopped for like 5 seconds. Then I remembered the ancient rsync backup to that old Contabo storage I never cancelled cause it was only €2/mo. 3 year old script. Saved my butt. Meant to replace it with something modern like 18 months ago. Still havent. Maybe wont now lol
$3/year. 128MB RAM. Pure happiness.
uma
Member
99.99% or bust
- Joined:
- Jun 2024
- Posts:
- 324
- From:
- Dublin, IE
I've got three terminal color schemes mapped to severity:
- Green for staging
- Amber for prod
- Red for anything customer-facing
Status page went amber at 14:03 UTC. Backup restoration RTO was 4 hours 17 minutes against an
SLA of 4 hours. Alert fatigue is real but this is why I keep the pagerduty escalation path to three people minimum.
436 days. reboot is surrender.
lee_mcr
Member
- Joined:
- Sep 2024
- Posts:
- 242
- From:
- Manchester, UK
Right then, proper nightmare that. We've all been one tab away from disaster. I once rebooted a live hypervisor thinking it was the test node. Cheers lads for the honesty, most hosts won't admit they nearly rm -rf'd their entire platform. 🦡 The £2 backup you forgot about is always the one that saves you.
Honey badger don't care... about downtime
Richard7
Member
Pentium III Enjoyer
- Joined:
- May 2024
- Posts:
- 129
- From:
- Norfolk, UK
Ancient rsync backup to that old Contabo storage
No cloud backups, no snapshots, just a DAT tape drive I barely trusted. Took six hours to restore. They don't make them like that anymore. Your Contabo rsync script is basically that DAT tape. There's something warm about old backup systems that just keep working while you forget them.
SPARCstation 20, still serving HTTP
