Skip to content

How often do you actually reboot for kernel patches?

General Discussion by FlowSana 25 replies 3.1K views
#1

1. We run approximately 340 KVM instances across three budget providers (OVHcloud, Vultr, Contabo).
2. Current policy: critical CVEs (CVSS 9.0+) within 72 hours, everything else monthly during scheduled maintenance windows.
3. Live patching via Canonical Livepatch on Ubuntu 24.04 LTS covers roughly 60% of kernel CVEs without reboot.
4. Remaining 40% require full restart. Downtime budget is 99.5% SLA, so we batch these.

Question for the room: do you reboot immediately on every kernel update, or do you queue them? I am particularly interested in how smaller shops handle this without redundant nodes.

My current recommendation: deploy live patching, maintain a 30-day maximum patch lag, and never let a known exploited vulnerability sit beyond 24 hours regardless of maintenance window.

It's always DNS. Always.
#2

Live patching is a band-aid and you know it

The virtualization tax on these budget KVM nodes is already bad enough without adding another kernel module doing runtime trampolines. OpenVZ was worse obviously because you couldn't patch the host kernel without killing every container, but at least the overhead was near-zero. With KVM you're paying the full virtualization tax on context switches, TLB flushes, and now livepatch is doing stop_machine() on your vCPUs.

I run a mix of Ryzen 5950X and EPYC 7313 dedicated boxes. Reboot quarterly at 04:00 local unless there's an active exploit in the wild. The 5 minutes of downtime beats months of accumulated scheduler weirdness from livepatch stack pivoting.

To be honest if your provider's network is decent the boot time is under 45 seconds. Just eat the cost.

virsh list --all | wc -l: 47
#3

Just use Arch and run linux-hardened 🐧

Livepatch? Skill issue. Real admins compile their own kernel with -ck patches and KSPP enabled. I reboot whenever I feel like it which is basically never because I read every commit between 6.10 and 6.11 and if Linus didn't mess up the merge window AGAIN I just kexec

> Reboot quarterly at 04:00
This is why we can't have nice things. Back in MY day we patched with ksplice and ORACLE killed it because they hate fun

Anyway my uptime is 847 days on a $3 NAT box from RackNerd. The graphs don't lie except when they do

https://www.racknerd.com

oops: 0000 [#1] SMP
1 #4

20 years in this business and I have seen every "zero downtime" fad come and go.

Back when we ran bare metal on Pentium 4s in a colo that flooded twice a year, you rebooted when the kernel panicked or never. These kids these days with their live patching and their kexec and their "virtualization tax" have never held a screwdriver while a RAID controller beeps at 3 AM because the BBU failed.

I reboot monthly. First Sunday, 02:00. Mark my words: one of these days a livepatch is going to corrupt page tables on a heavily loaded database box and the recovery time will dwarf any planned outage. The industry will swing back to "reboot and verify" within five years. It always does.

My current fleet is 8x Dell R730xd with E5-2680v4, running Proxmox. 45 seconds to login prompt. I spend more time waiting for ZFS scrub to finish than the actual boot.

IPv4, IRC, and irssi — fight me
#5

As said in the other thread, our maintenance windows are:

- standard nodes: first Tuesday 03:00-05:00 UTC, quarterly
- NVMe tiers: rolling reboots across hypervisor pairs, monthly
- legacy SATA: whenever we remember, to be honest

Specs on our current Ryzen 9 7950X3D fleet:
- 128 GB DDR5-4800
- 2x 3.84 TB NVMe (disk ~1450 MB/s seq read on KVM)
- 2x 10 Gbit LACP to each node
- Geekbench 6 single: ~2650

We give 72 hours notice via status page. Livepatch is available as a $2/month add-on but most customers skip it. The ones who do buy it are the same ones who open tickets asking why their 512 MB NAT VPS is "slow" during a kernel update.

Dry humor: our most rebooted node this year was restarted 47 times by a customer who wrote a cron job to apply security updates at midnight. We suggested unattended-upgrades. They switched providers.

Containers before it was cool
#6

I reboot INSTANTLY!!!

What could go wrong if you wait:

1. Attacker already has your box by the time you read the CVE
2. Livepatch doesn't cover the networking stack and your firewall rules are bypassable
3. Your provider's "maintenance window" is when THEY get owned and pivot to YOUR vm
4. fail2ban won't save you from a kernel-level rootkit

I run 12 NAT boxes across 6 providers including Oracle free tier. Every one has custom iptables, firewalls, AND I reboot on ANY kernel update within 15 minutes. My uptime is terrible. My security posture is EXCELLENT

WARNINGS: if you trust livepatch you are trusting a CLOSED SOURCE SIGNING KEY from a vendor who got owned in 2019 and didn't disclose for 6 months!!! 👀

The graphs don't lie but the vendors do

airgapped, encrypted, faraday'd, still worried
#7

99.99% or bust My status page has 47 subscribers and I will not disappoint them. Current stats:
- OVHcloud KVM 2GB: 99.97% over 18 months (lost 0.03% to their quarterly maintenance, not my fault)
- Vultr dedi: 99.995% over 24 months (one reboot for kernel, took 127 seconds)
- Contabo NAT: 99.91% (their network, not mine) I livepatch where available, reboot everything else during my own 04:00 Sunday window. Alert fatigue is real: I have 23 monitoring checks and PagerDuty goes off if ICMP drops for 30 seconds. But my blood pressure does. I tried kexec once. Kernel crashed. 23 minutes of downtime. Never again. https://www.kernel.org

436 days. reboot is surrender.
#8

I had same alert fatigue on PagerDuty—23 checks is brutal

#9

What's the monthly cost on that Vultr dedi

#10
ZhangElder3 said:
What's the monthly cost on that Vultr dedi

That 7950X3D fleet is bare metal, not dedi in the old sense. Vultr calls it "Cloud GPU-adjacent" but it's just regular bare metal with fast provisioning. List was around $120/month last I checked but we commit to quarterly so we get a break. The real cost is the NVMe overages if a customer actually uses the disk.

To the actual thread: we tried livepatch for six months. Three tickets about "stuck processes after canonical pushed an update." Not worth the $2 for us. Customer can buy it or reboot. We document the window.

Containers before it was cool

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft