What do you use for automated backups that actually restore?

Rsync -avz, they said. Simple, they said. Set it and forget it. Found out last week my "backup" has been silently failing for 6 months. Cron email went to root@localhost. Root mailbox was on a volume that... also wasn't backing up. Beautiful. The kicker? I only found out because I needed a file from March. The backup directory had timestamps from March. File sizes were all zero. rsync exited 0 every time. SSH key had rotated, destination was accepting connections and immediately closing them. Zero bytes, happy exit code, happy cron. So. What do you actually use? And have you tested a restore this year? Be honest.

---
- borg:
- deduplication: yes
- encryption: yes
- NOTE: append-only mode for ransomware protection
- restic:
- same backend support
- WARNING: prune is slow
- kopia:
- newer
- GUI exists
- NOTE: I do not trust GUIs
---
- testing restores:
- monthly: automated
- quarterly: full disaster simulation
- WARNING: most people skip this
- NOTE: you are most people
---
- your rsync problem:
- exit code 0 on SSH disconnect is criminal
- NOTE: wrap in --stats, parse output
- or stop using rsync for backups

Guys, this is why I just sell the backup service to clients. Let them deal with it. Any leads on whitelabel backup panel? Recieved request from big client last week, their current guy failed restore test.

Margins on backup reselling are solid if you find right provider. But which ones actually work? I tried one last year, restore took 14 hours for 200GB. Client was not happy.

Anyone using GreenCloudVPS or Hetzner for backup storage? Need API, need reliable.

I use Borg with HostHatch storage, works good for me. I make test restore every month, is important.

Regards

---
Oh no, I read again your post. 6 months silent fail? Is nightmare. I did same with rsync long time ago, now I always check log with script that send alert if no new snapshot.

You should try Restic, has built-in check command. Or Kopia, is easy to make server up with docker compose.

Regards