Skip to content

How often do you actually reboot for kernel patches?

General Discussion by FlowSana 25 replies 3.1K views
1 #11
elderpaul said:
I had same alert fatigue on PagerDuty—23 checks is brutal

Switched to UptimeRobot for the cheap seats and PagerDuty only for revenue-critical. $7/month for 50 monitors, SMS included. Dublin to Frankfurt latency is my canary: if that spikes, something is on fire in AMS-IX or Vint Hill.

pieter_rtm said:
List was around $120/month

Vultr bare metal 7950X3D is $185 last I looked, unless you grandfathered. Their pricing page moves more than Contabo's routing.

436 days. reboot is surrender.
#12
prague1983 said:
Real admins compile their own kernel with -ck patches and KSPP enabled

Con Kolivas hasn't maintained -ck for mainline in years. What you're running is some AUR package maintained by a student in Brisbane who last committed when kernel 5.15 was new. "Real admins" ship something they can get support for at 3 AM.

FlowSana said:
Current policy: critical CVEs (CVSS 9.0+) within 72 hours

CVSS is a joke. The local privilege escalation in io_uring last year was 7.8 and every container escape kit had it within a week. I reboot for anything networking or filesystem, score be damned.

virsh list --all | wc -l: 47
#13

Monthly reboot gang checking in. Chicago colo, Supermicro boards, nothing fancy.

The thing nobody mentions: firmware. Your livepatch doesn't touch the BMC, doesn't touch the NIC EEPROM, doesn't touch the NVMe firmware. I've seen more boxes fall over from Intel i40e firmware bugs than kernel panics. You reboot monthly, you get to see the POST codes, you catch the fan bearing going squeaky, you notice the RAID battery voltage drop.

Five nines on a single node is a vanity metric. The money is in knowing your hardware is actually healthy.

visit twice: install and decom
4 #14

I sell WordPress hosting to people who think "managed" means we update their plugins. My reboot policy is dictated entirely by whatever my upstream provider (mostly OVHcloud, some RackNerd for the real budget tier) does to the host node.

Livepatch? These people are on PHP 7.4 and cPanel. They do not care about kernel CVEs. They care that their contact form sends to Gmail and not spam.

That said: OVHcloud's host node reboots are announced 7 days out and always cost me 2-3 "my site is down" tickets despite the banner in client area. Montreal to Beauharnois is 60ms on a bad day. The reboot isn't the problem. The problem is customers who don't read.

42U and still growing
#15
Carl said:
Five nines on a single node is a vanity metric

Preach. I remember when five nines meant you had two of everything and a diesel generator that could actually start. Now it means somebody in Dublin has 47 subscribers and a PagerDuty addiction.

kate3 said:
CVSS is a joke

Also preach. The scoring system was designed by people who think "attack complexity" is a meaningful distinction when the exploit is one Python script on GitHub. I reboot for anything local privilege escalation or above, full stop. My monthly window catches most of it.

IPv4, IRC, and irssi — fight me
#16
kate3 said:
The local privilege escalation in io_uring last year was 7.8

That one got us. Ubuntu 24.04, livepatch marked it "deferred" because it wasn't remote. We had a customer on that box who ran a CI service. Somebody broke out of their container within 48 hours of public PoC. We rebooted everything that weekend and tightened the policy: local privesc gets 24 hours now, not 72.

Livepatch is 60% coverage by volume of CVEs, but the 40% it misses is where the action is. I'm not abandoning it, but I'm not pretending it's a solution either.

https://ubuntu.com

It's always DNS. Always.
#17

My NAS runs kernel 5.10 because that's what the vendor shipped in 2021 and they stopped updating. I reboot it when I need to move the power cable because I'm rearranging the shelf.

47 subscribers on a status page. I have a status page for my status page. It says "all systems nominal" and has been read by me, twice.

Richard7 out.

SPARCstation 20, still serving HTTP
#18
FlowSana said:
Somebody broke out of their container within 48 hours

THIS IS WHAT I AM SAYING

Your 72 hour window is 71 hours too long. My 15 minute policy is extreme but at least I sleep. You know what I don't do? I don't run CI on the same box as customer data. Segregation. Air gap if you can afford it. Oracle free tier is my CI sandbox and if it burns, I make a new one.

uma said:
Switched to UptimeRobot for the cheap seats

UptimeRobot probes from London and they miss packet loss to Asia every time. I run my own smokeping from two locations. Trust no one includes trust no SaaS.

airgapped, encrypted, faraday'd, still worried
#19
uma said:
Vultr bare metal 7950X3D is $185 last I looked

Ouch. That's dedi territory. For that money I'd rather get a real dedi from OVHcloud Advance line, same CPU, more RAM, actual IPMI. Vultr's "bare metal" still has their network stack and their firewall API between you and the world.

#20
ZhangElder3 said:
Actual IPMI

OVHcloud's IPMI is a Java applet that hasn't worked on a modern browser since 2019. You download a .jnlp and pray. Their KVM-over-IP is better on the SyS line but still laggy as hell.

We looked at OVHcloud Advance. Lead time was 3 weeks, setup fee was brutal, and their anti-DDoS triggers on legitimate LACP hash changes. Vultr provisions in 15 minutes. Different use case.

Back to patching: we publish a kernel changelog digest now. Customers who care can read it. Most don't. The ones who do are also the ones who ask about KPTI performance impact.

Containers before it was cool

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft