Skip to content
Register

What do you use for disaster recovery drills?

General Discussion by tomhider 17 replies 1.3K views
norafilch

Member

#11
NUT said:
Eating som tam

This is why I run chaos_raccoon at 2pm on a Tuesday. The hospital NOC is staffed, I'm caffeinated, and the pager is loud enough to hear over the HVAC. Muscle memory requires repetition, not heroics at midnight.

tomhider said:
How do u make it regular

Calendar is too easy to dismiss. I tied mine to an actual business event: every time we rotate the tape offsite (weekly), we also fail over the PACS viewer to its secondary. The DR drill is now part of a real process with a real person accountable. Took six months to get there.

mediaaustin

Member

Deliverability Nerd

#12
tomhider said:
Also randyaround what does a Hetzner Storage Box cost for that, ballpark?

I feel this in my soul. Also: mail deliverability is a DR drill nobody thinks about until your SPF record is wrong on the failover IP and now your invoices are in spam. Ask me how I know.

Hetzner Storage Box starts around 4 EUR for 100 GB, scales to 20-something for 5 TB. Wasabi is cheaper per GB but egress will eat you if you restore often.

SPF, DKIM, DMARC — holy trinity ✉️
Prague

Member

#13

I run my drills on the actual hardware because it's all I have. Core 2 Duo in Prague, backup to another Core 2 Duo at my mother's flat. The drill is: unplug the network cable, walk to her place, boot from USB, rsync back. Takes two hours if the tram is running. If it's not running, that's also the drill.

Tabletop would be faster. I do not care. The machine is real, the failure is real, the walk is real.

iron core, iron will
MARIA3

Member

#14
Prague said:
Core 2 Duo at my mother's flat

Now that's true love for infrastructure. I've got three cats that walk across the keyboard, that's my chaos engineering. Once they rebooted my nephew's Minecraft server. Now everything's got passwords and the cables are in conduits.

For DR I keep it basic: snapshots on Btrfs, copy to an old NAS at my sister's place in Valencia. The drill is one Sunday a month, coffee, and "let's see if this thing boots".

siesta first, deploy later
reykjavik_doc

Member

container skeptic

#15
randyaround said:
ZFS send/recv monthly

Monthly is too long. You discover a corrupted snapshot at month three and you've been backing up garbage since. I run weekly scrub, weekly send. The incremental is small, the verification is immediate, and I sleep better.

Prague said:
Walk to her place

This is not a drill, this is pilgrimage. Respect.

MeritBudi

Member

2 #16

I have 7 VPS and no sleep so my DR drill is mostly "can I afford to lose this one." The answer is usually yes. For the two that matter: GreenCloudVPS in Singapore, backup to Hetzner Helsinki — https://www.hetzner.com/cloud — via rclone. The drill is automated, I just check the rclone log for ERROR strings every morning while eating instant noodles. If I see ERROR I fix it. If I don't see ERROR I assume it's fine. This is probably bad.

tomchan

Member

#17
MeritBudi said:
Assume it's fine

Famous last words. I used to assume until I found rclone had been silently skipping files with colons in names for six months. Windows target, Linux source, nobody warned me.

Now my drill includes a random sample restore: pick 10 files, verify they exist, verify they match checksum. Caught three different bugs in two years. The WAF in my shed is less paranoid than my backup verification.

works on my bench ¯\_(ツ)_/¯
pablowild

Member

3 #18
tomchan said:
Windows target, Linux source

Jajaja the colon problem! I had this with my seedbox, same thing. Now I use rclone check --one-way and I read the output like tea leaves. Very stressful.

My "cry and migrate to Hetzner" drill I did again last month but for real, Leaseweb raised prices and I moved a storage VPS. The migration WAS the drill. Took 6 hours, most of it waiting for rsync. I documented nothing, I was too tired. Now I have a server I don't remember how to configure. Chaos forever :-)

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft