borgbackup vs restic vs kopia: my 3-year data survived none

3-year backup test. All three tools: corrupted restores. Bitrot in borg, restic, kopia. Prices only go up, trust only goes down.

I maintained separate repos on the same VPS. Quarterly test restores. Last week I needed a 2023 file from each. All three returned garbage on the same 47 files. Different garbage, but garbage.

Not a credentials issue. Not a repo lock issue. Underlying ext4 on KnownHost KVM. /24 will cost you a kidney, but data integrity shouldn't.

Where do I even start looking?

Hey folks

Gonna be honest this is giving me anxiety about my own setup

Super stoked you caught it in testing tho, that's the fire move right there

I run borg on Hostinger, now I'm scared. Gonna run test restore tonight. Thanks for the heads up, seriously.

Same generation hardware across all three repos? Or spread around?

I've seen RAM errors masquerade as filesystem corruption. DDR4-2400, early production runs, bit flips under load. Run memtest86+ on that box before you blame the software. Also check your power draw—undervolted DIMMs do weird things.

If it's a single VPS: ask KnownHost what node you're on — https://status.knownhost.com. Supermicro X11 generation had a bad batch. I shipped three pallets of those back in 2019.

Ran fio and stress on my KnownHost box last month:

disk: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=78.3MiB/s,w=0KiB/s][r=20.0k,w=0 IOPS][eta 00m:00s]
   read: IOPS=20.5k, BW=80.1MiB/s (83.0MB/s)(4802MiB/60001msec); 0 zone resets
    slat (nsec): min=1500, max=81000, avg=4500.00, stdev=1200.00
    clat (usec): min=120, max=45000, avg=3100.00, stdev=2800.00
     lat (usec): min=125, max=45100, avg=3105.00, stdev=2801.00
cpu          : usr=5.2%, sys=22.1%, ctx=1234567, majf=0, minf=9
IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%

fsync/fdatasync/sync_file_range:
  sync (usec): min=3, max=450, avg=45.00, stdev=30.00
  sync percentiles (usec):
   |  1.00th=[    5],  5.00th=[    8], 10.00th=[   12], 20.00th=[   20],
   | 30.00th=[   28], 40.00th=[   35], 50.00th=[   42], 60.00th=[   50],
   | 70.00th=[   58], 80.00th=[   68], 90.00th=[   85], 95.00th=[  120],
   | 99.00th=[  250], 99.50th=[  350], 99.90th=[  400], 99.95th=[  420],
   | 99.99th=[  450]

Decent disk IO, meh network

But look at those clat stdev spikes. 28ms average, 45ms max. That's not normal for local SSD. Could be noisy neighbor, could be something worse.

$ btrfs scrub start /backup
$ dmesg | grep -i "error\|correction\|ata"
[ 1234.567890] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 1234.567891] ata1.00: failed command: READ FPDMA QUEUED

Wait—that's from my actual box. Not a benchmark, live system. ATA errors on the host. KnownHost using consumer drives in RAID0? Maybe.

What borg version and which python

I had the same clat spikes on KnownHost, ata errors in dmesg too

Same 47 files? That's a pattern, not random bitrot.