kate3
Member
- Joined:
- Jul 2024
- Posts:
- 208
- From:
- Utrecht, NL
Live patching is a band-aid and you know it
The virtualization tax on these budget KVM nodes is already bad enough without adding another kernel module doing runtime trampolines. OpenVZ was worse obviously because you couldn't patch the host kernel without killing every container, but at least the overhead was near-zero. With KVM you're paying the full virtualization tax on context switches, TLB flushes, and now livepatch is doing stop_machine() on your vCPUs.
I run a mix of Ryzen 5950X and EPYC 7313 dedicated boxes. Reboot quarterly at 04:00 local unless there's an active exploit in the wild. The 5 minutes of downtime beats months of accumulated scheduler weirdness from livepatch stack pivoting.
To be honest if your provider's network is decent the boot time is under 45 seconds. Just eat the cost.
virsh list --all | wc -l: 47
haroldgsm
Member
Grumpy Old Sysadmin
- Joined:
- May 2024
- Posts:
- 329
- From:
- Ohio, US
20 years in this business and I have seen every "zero downtime" fad come and go.
Back when we ran bare metal on Pentium 4s in a colo that flooded twice a year, you rebooted when the kernel panicked or never. These kids these days with their live patching and their kexec and their "virtualization tax" have never held a screwdriver while a RAID controller beeps at 3 AM because the BBU failed.
I reboot monthly. First Sunday, 02:00. Mark my words: one of these days a livepatch is going to corrupt page tables on a heavily loaded database box and the recovery time will dwarf any planned outage. The industry will swing back to "reboot and verify" within five years. It always does.
My current fleet is 8x Dell R730xd with E5-2680v4, running Proxmox. 45 seconds to login prompt. I spend more time waiting for ZFS scrub to finish than the actual boot.
IPv4, IRC, and irssi — fight me
pieter_rtm
Member
- Joined:
- Jul 2024
- Posts:
- 195
- From:
- Rotterdam, NL
As said in the other thread, our maintenance windows are:
- standard nodes: first Tuesday 03:00-05:00 UTC, quarterly
- NVMe tiers: rolling reboots across hypervisor pairs, monthly
- legacy SATA: whenever we remember, to be honest
Specs on our current Ryzen 9 7950X3D fleet:
- 128 GB DDR5-4800
- 2x 3.84 TB NVMe (disk ~1450 MB/s seq read on KVM)
- 2x 10 Gbit LACP to each node
- Geekbench 6 single: ~2650
We give 72 hours notice via status page. Livepatch is available as a $2/month add-on but most customers skip it. The ones who do buy it are the same ones who open tickets asking why their 512 MB NAT VPS is "slow" during a kernel update.
Dry humor: our most rebooted node this year was restarted 47 times by a customer who wrote a cron job to apply security updates at midnight. We suggested unattended-upgrades. They switched providers.
Containers before it was cool
olespete
Member
Trust No One
- Joined:
- Jun 2024
- Posts:
- 270
- From:
- Unknown
I reboot INSTANTLY!!!
What could go wrong if you wait:
1. Attacker already has your box by the time you read the CVE
2. Livepatch doesn't cover the networking stack and your firewall rules are bypassable
3. Your provider's "maintenance window" is when THEY get owned and pivot to YOUR vm
4. fail2ban won't save you from a kernel-level rootkit
I run 12 NAT boxes across 6 providers including Oracle free tier. Every one has custom iptables, firewalls, AND I reboot on ANY kernel update within 15 minutes. My uptime is terrible. My security posture is EXCELLENT
WARNINGS: if you trust livepatch you are trusting a CLOSED SOURCE SIGNING KEY from a vendor who got owned in 2019 and didn't disclose for 6 months!!! 👀
The graphs don't lie but the vendors do
airgapped, encrypted, faraday'd, still worried
uma
Member
99.99% or bust
- Joined:
- Jun 2024
- Posts:
- 324
- From:
- Dublin, IE
99.99% or bust My status page has 47 subscribers and I will not disappoint them. Current stats:
- OVHcloud KVM 2GB: 99.97% over 18 months (lost 0.03% to their quarterly maintenance, not my fault)
- Vultr dedi: 99.995% over 24 months (one reboot for kernel, took 127 seconds)
- Contabo NAT: 99.91% (their network, not mine) I livepatch where available, reboot everything else during my own 04:00 Sunday window. Alert fatigue is real: I have 23 monitoring checks and PagerDuty goes off if ICMP drops for 30 seconds. But my blood pressure does. I tried kexec once. Kernel crashed. 23 minutes of downtime. Never again. https://www.kernel.org
436 days. reboot is surrender.
pieter_rtm
Member
- Joined:
- Jul 2024
- Posts:
- 195
- From:
- Rotterdam, NL
What's the monthly cost on that Vultr dedi
That 7950X3D fleet is bare metal, not dedi in the old sense. Vultr calls it "Cloud GPU-adjacent" but it's just regular bare metal with fast provisioning. List was around $120/month last I checked but we commit to quarterly so we get a break. The real cost is the NVMe overages if a customer actually uses the disk.
To the actual thread: we tried livepatch for six months. Three tickets about "stuck processes after canonical pushed an update." Not worth the $2 for us. Customer can buy it or reboot. We document the window.
Containers before it was cool