So the topic is disaster recoveryy drills rite. We run a 3 person shop on GreenCloudVPS & I been meanign to do this proper. Last week I tested our "restore from backup" by pretendign the prod server died. Turns out my backup script had been backing up the MOUNT POINT not the actual disk for 4 months. You figure it out he says. So now I want REAL drills not just hopign. What do u all do for small scale?? Tabletop?? Actual chaos?? Somwhere between??
I run everything on bare metal with systemd and proper unit dependencies. No containers at home. Þe best disaster recovery is simplicity you can explain to a tired human at 3am. Your backup script bug proves my point: more moving pieces, more places to hide failure.
I still do tabletop exercises with printed network diagrams. Draw the failure with a red marker. Trace the packets. If you cant explain the recovery without a keyboard, you dont know it. My 2c.
🖥️ lol nice
Member
No plan survives contact. Chaos is the only honest state. I run a script called chaos_raccoon.sh that picks a random systemd unit on my Time4VPS box, stops it. No warning. No schedule. Sometimes its nginx. Sometimes its the backup timer itself. The game is how fast I notice. How fast I remember why this particular failure mode is different from last time. Stream of broken consciousness becoming muscle memory. Poetic in its cruelty.
Member
¡this is very good thread! ¿how do you document what you learned? Jajaja I tried the chaos approach but I would get so emotional in the moment, write long paragraphs in my wiki, then delete & repost shorter because nobody reads the novel. Vut now I feel I lose the context. My last drill was "what if Leaseweb deletes your account with no warning" — is very realistic, I discovered. The answer was "cry & migrate to Hetzner" :-)
GreenCloudVPS backup script bug is a mood. Moon time backups
Simplicity is great until you need to roll back a snapshot in thirty seconds because your parity check found silent corruption. I drill ZFS send/recv monthly. Source pool on my home box, target is a Hetzner Storage Box in Falkenstein. The drill is: destroy a dataset, restore from latest incremental, verify checksums. If the verify fails I know my backup chain is broken before I need it.
The only "tabletop" part is the checklist I read aloud so I don't skip a step. Everything else is actual commands, actual data, actual panic when the first recv fails because I forgot the -F flag again.
This is the thing tho rite. I *could* explain it, I just never TESTED the explanation. My script looked fine, ran fine, exit 0 every night. The mount point thing was invisible until I needed it. So yeah I'm sold on actual chaos now but how do u make it regular?? Calendar reminder?? Every deploy??
Also randyaround what does a Hetzner Storage Box cost for that, ballpark? I'm on GreenCloudVPS cos cheap but their storage options are thin.
I did this on my Django box once, stopped postgresql, forgot I had no failover. Site down 20 minutes before I noticed because I was eating som tam. Now I only chaos during work hours and I have a Telegram bot yell at me first. Not very poetic. Very practical.
GreenCloudVPS I use for a scraping project. Backup is restic to Wasabi. Drill is: spin up new VPS, restore, point DNS. Takes about 40 minutes if I remember the restic password.
