Skip to content

Automation question: how do you test your backups actually work?

General Discussion by sofialund 21 replies 1.5K views
#1

I have the server red with 200 days uptime, and the backups automatic to Hetzner. But I never do the restore test, no? ¡I am afraid that when the disaster comes, the backup is corrupted! My friend from RackNerd says he lost a database because the backup had the checksum bad.

How often you do the fire drill of restore? ¿You automate the test? I use vos system with ansible for deploy, but for backup I have the blind faith. ¡I want to change this!

What tool you recommend for verify the integrity without affect the production? The restore to VM isolated is sufficient, or you test in hardware real?

Thanks in advance!

hot air, steady hand, magic smoke
#2

Can lah

I do random restore every 2 week leh, spin up container in lab already

No fancy tool lor, just cron + script, pick backup from 7/30/90 day ago randomly, restore, run checksum, destroy

Last year found 3 backup corrupt out of 50 test, sibeh lucky

#3

Random 7/30/90 day pick is smart, I do fixed schedule only

SPF, DKIM, DMARC — holy trinity ✉️
#4

Hetzner silent corruption, same headache. Seen it a dozen times.

...
5 #5

This is why I keep a warm spare on a different provider entirely. Hetzner to RackNerd, RackNerd to Hetzner. Belt and suspenders.

grabs popcorn, checks /r/drama
#6
digitalsearch said:
No fancy tool lor, just cron + script

What language you write the script in? Bash or something else? I been meaning to automate mine but keep doing it manual like a caveman.

SPF, DKIM, DMARC — holy trinity ✉️
#7

¡Thank you for the replies! I am looking now the restic, this seems very complete. But I have the question: the prune operation is safe for the production data? I read that can be heavy in IO.

Also, the warm spare of Doug is interesting but the cost double. For homelab is maybe too much.

hot air, steady hand, magic smoke
#8

Restic prune is safe but yes, heavy IO. I run it on Sunday 3 AM when nothing else happens. My Hetzner box in Falkenstein has SSD so it's 20 minutes for 400 GB. On spinning rust would be pain.

For verify without restore, I use `restic check --read-data-subset=10%` every week. Reads random 10%, not full restore but catches corruption early. Full restore test still monthly to VM.

POLISH SERVERS. LOUD FANS. GOOD PRICE.
#9

I just pipe everything to /dev/null and hope for the best. Has worked so far.

/dev/null: full of good ideas
3 #10
sofialund said:
The cost double

Not necessarily. My warm spare is the smallest RackNerd VPS, 1.5 GB plan, $18/year. Just enough to hold encrypted restic repo. If main dies I can restore anywhere, not just back to Hetzner.

The real cost is the time to set it up, not the money.

...

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft