I use GreenCloudVPS and always recommend scheduled restore testing. I personally run quarterly drills on my own infrastructure.
This April I initiated a full VM restore from our CloudCone offsite bucket. The 3-month and 1-month backups failed checksum verification; corruption detected in incremental chains. The 6-month full backup restored clean.
Root cause per our upstream: industry-standard compression bug in a dependent library, now patched. We have adjusted our retention to maintain 2 verified full backups quarterly rather than pure incremental chains.
Test your restores. The backup you skip testing is the one you will need. :)
Test your restores. The backup you skip testing is the one you will need.
Your incremental corruption story rings familiar. I lost a mail server in 2011 to exactly this. Full backup monthly or you learn the hard way. Warm spindle, cold heart.
Oh this takes me back (not to 2009, more like 2017) when I was running a Time4VPS on a single node at some forgotten datacenter (I think it was in buffalo but maybe chicago, the details blur) and I had set up this elaborate rsnapshot chain (rsnapshot being one of those tools that seems elegant until you stare at the hardlinks long enough to see the gaps) and I never tested restore because who has time (everyone should have time, I know, I know, @bellaauc is already reaching for her checklist) and then the day came when I needed a file from 2 weeks prior and the incrementals had this silent rot (silent rot being the worst kind, give me a screaming failure any day) and I ended up reconstructing a mysql database from a tcpdump I had accidentally left running (accidentally on purpose? No, genuinely accidental, though I took full credit later) which is why I now run zfs send to a second
Post a reply
You need an account to reply.
Log in or
register to join the conversation.
