uma
Member
99.99% or bust
- Joined:
- Jun 2024
- Posts:
- 324
- From:
- Dublin, IE
I had same alert fatigue on PagerDuty—23 checks is brutal
Switched to UptimeRobot for the cheap seats and PagerDuty only for revenue-critical. $7/month for 50 monitors, SMS included. Dublin to Frankfurt latency is my canary: if that spikes, something is on fire in AMS-IX or Vint Hill.
List was around $120/month
Vultr bare metal 7950X3D is $185 last I looked, unless you grandfathered. Their pricing page moves more than Contabo's routing.
436 days. reboot is surrender.
kate3
Member
- Joined:
- Jul 2024
- Posts:
- 208
- From:
- Utrecht, NL
Real admins compile their own kernel with -ck patches and KSPP enabled
Con Kolivas hasn't maintained -ck for mainline in years. What you're running is some AUR package maintained by a student in Brisbane who last committed when kernel 5.15 was new. "Real admins" ship something they can get support for at 3 AM.
Current policy: critical CVEs (CVSS 9.0+) within 72 hours
CVSS is a joke. The local privilege escalation in io_uring last year was 7.8 and every container escape kit had it within a week. I reboot for anything networking or filesystem, score be damned.
virsh list --all | wc -l: 47
Carl
Member
Rack & Stack
- Joined:
- May 2024
- Posts:
- 227
- From:
- Chicago, US
Monthly reboot gang checking in. Chicago colo, Supermicro boards, nothing fancy.
The thing nobody mentions: firmware. Your livepatch doesn't touch the BMC, doesn't touch the NIC EEPROM, doesn't touch the NVMe firmware. I've seen more boxes fall over from Intel i40e firmware bugs than kernel panics. You reboot monthly, you get to see the POST codes, you catch the fan bearing going squeaky, you notice the RAID battery voltage drop.
Five nines on a single node is a vanity metric. The money is in knowing your hardware is actually healthy.
visit twice: install and decom
haroldgsm
Member
Grumpy Old Sysadmin
- Joined:
- May 2024
- Posts:
- 329
- From:
- Ohio, US
Five nines on a single node is a vanity metric
Preach. I remember when five nines meant you had two of everything and a diesel generator that could actually start. Now it means somebody in Dublin has 47 subscribers and a PagerDuty addiction.
Also preach. The scoring system was designed by people who think "attack complexity" is a meaningful distinction when the exploit is one Python script on GitHub. I reboot for anything local privilege escalation or above, full stop. My monthly window catches most of it.
IPv4, IRC, and irssi — fight me
Richard7
Member
Pentium III Enjoyer
- Joined:
- May 2024
- Posts:
- 129
- From:
- Norfolk, UK
My NAS runs kernel 5.10 because that's what the vendor shipped in 2021 and they stopped updating. I reboot it when I need to move the power cable because I'm rearranging the shelf.
47 subscribers on a status page. I have a status page for my status page. It says "all systems nominal" and has been read by me, twice.
Richard7 out.
SPARCstation 20, still serving HTTP
olespete
Member
Trust No One
- Joined:
- Jun 2024
- Posts:
- 270
- From:
- Unknown
Somebody broke out of their container within 48 hours
THIS IS WHAT I AM SAYING
Your 72 hour window is 71 hours too long. My 15 minute policy is extreme but at least I sleep. You know what I don't do? I don't run CI on the same box as customer data. Segregation. Air gap if you can afford it. Oracle free tier is my CI sandbox and if it burns, I make a new one.
Switched to UptimeRobot for the cheap seats
UptimeRobot probes from London and they miss packet loss to Asia every time. I run my own smokeping from two locations. Trust no one includes trust no SaaS.
airgapped, encrypted, faraday'd, still worried
pieter_rtm
Member
- Joined:
- Jul 2024
- Posts:
- 195
- From:
- Rotterdam, NL
OVHcloud's IPMI is a Java applet that hasn't worked on a modern browser since 2019. You download a .jnlp and pray. Their KVM-over-IP is better on the SyS line but still laggy as hell.
We looked at OVHcloud Advance. Lead time was 3 weeks, setup fee was brutal, and their anti-DDoS triggers on legitimate LACP hash changes. Vultr provisions in 15 minutes. Different use case.
Back to patching: we publish a kernel changelog digest now. Customers who care can read it. Most don't. The ones who do are also the ones who ask about KPTI performance impact.
Containers before it was cool