Your worst outage story — go

Quinnbizz's 9-hour opacity is why I left OpenVZ behind entirely. Virtualization tax is real but at least with KVM I can snapshot and evacuate.

My worst: 2023, running a small hosting coop on KnownHost KVM instances. Kernel 5.15.0-76 on host, 5.15.0-88 on guest. Guest cgroup memory limits were being enforced incorrectly after a host kernel minor update — memory.max was being interpreted in pages instead of bytes for one specific kernel config combination. Guests with 2GB limits were getting 8MB effective. OOM killer every 45 minutes, no pattern, no useful logs.

Took 11 hours to bisect: boot older guest kernel, stable. Boot same guest kernel on different host, stable. Only that specific host kernel + guest kernel + cgroup v2 + memory controller compiled as module. The virtualization tax includes debugging kernel interactions at 3AM.

I now pin host kernels for 6 months minimum and tes

Anyway idk why I do this

3am, 2024, I'm trying to make a "high availability" minecraft server for my friends using two $3 NAT boxes and wireguard. One is in germany, one in singapore, shared state over sqlite on a third $2 box because I'm cheap

The german box had a "scheduled migration" at 2am their time. No email, just gone. My "ha" setup had a 90 second health check interval. In those 90 seconds, the singapore box promoted itself, wrote a chunk, the german box came back with old state, and I had two diverging worlds

My friends lost 6 hours of building a castle. I lost their trust. I still have the conflicting region files, they're in my "shame" directory

$ ls ~/shame/minecraft_ha_fail/
world/  world_nether/  world_the_end/
README.txt  # "never again"

The real lesson is don't do distributed consensus at 3am for minecraft

Hello, very sorry to hear your castle is gone

My worst outage is small story. I made VPS last month at RackNerd, they have very good price for Indonesia location. I use for my family shop website. Then last week, I cannot access. I make ticket, no answer for 3 days. I make second ticket, they say "network issue, please wait"

I wait 5 days more. Total 8 days. I have no backup, I not know about backup before I read this forum. My family shop lose many order, my mother is angry to me

Now I learn 3-2-1 rule from @bellaauc, thank you very much. I setup backup to my old laptop and to Vultr. I test restore last night, it is working. I am very grateful for this forum 🙏

8 days is long time but I am lucky to find you all

It's not a Windows vs Linux thing. My worst was actually at a previous job running Windows Server 2019 on a dedicated box at Hetzner. WSUS had a bad patch Tuesday in 2023, KB5026370, caused boot loops on specific Supermicro BIOS versions. My box was one of them. 14 hours to get IPMI console, boot WinRE, uninstall the patch. I had tested the patch on a VM, but the VM was VMware, and the bare metal had a specific I40e driver interaction that the VM didn't surface. The "test on identical hardware" rule gets violated constantly because identical hardware is expensive. I now maintain a sacrificial physical box for patch testing. It costs $90/month to sit idle 29 days a month. The one time it saves me, it pays for three years. I don't have a dramatic vani