Skip to content
Register

Monitoring setup: I stopped alerting on CPU and sleep better

General Discussion by cleardmitri 6 replies 195 views
cleardmitri

Moderator OP

Moderator

#1

I disabled all CPU alerts six months ago. Every alert I had in the previous two years was either a backup job, a log rotation, or a batch process that finished fine. My PagerDuty has not fired once for a real user-facing incident that CPU would have caught first.

I still alert on:

  • Request latency p99 > 200ms
  • Error rate > 0.1%
  • Disk full (not usage growth, actually full)
  • Certificate expiry

My SLOs are request-level. If the server is at 95% CPU but requests are fine, I do not care. I sleep through the night now.

I know this sounds reckless. I am posting because I want someone to tell me I have missed a case where CPU saved them when latency did not. Or confirm that most CPU alerts are noise if you have proper SLOs.

No logs, no proof. I have logs.
uma

Member

99.99% or bust

2 #2

I reviewed 18 months of our alerts: 94% of CPU alerts were non-actionable. The 6% that mattered were all preceded by latency spikes visible at p95.

My current setup:

  • CPU: dashboard only, no paging
  • Latency p99: page after 2 min
  • Error rate: page immediately
  • Saturation: custom metric (queue depth)

Status page incident count dropped 70%. Alert fatigue is real and it kills response time to actual problems. You are not reckless. You are disciplined.

436 days. reboot is surrender.
MARIA3

Member

2 #3

I am make a server up with this philosophy last year is work good but I have one case database server with CPU saturation but request latency OK because query cache then cache expire and all die in same moment regards 😊

My solution: alert on cache hit rate drop not CPU is better signal you are need look what is between CPU and user not only endpoint

siesta first, deploy later
MA

Member

#4

😴 lol nice wow

pauloserver

Member

#5

Cara I am going to try this kkkkk my cpu alerts is all noise disable tomorrow sleep like baby kkkkk but keep eye on saturation metric that is different going to follow uma advice kkkkk

chill infrastructure for chill people 🦫
lookuppierre

Member

#6

I did the same at the GreenCloudVPS (https://greencloudvps.com) six months ago how to say the change was « transformative » before we had the Alert Fatigue now the team responds in minutes to real issues yes we missed one disk failure because the CPU was normal but the SMART data caught it how to say the CPU is not the System it is one Signal among many Random Words because emphasis yes I agree with you regards

prix fixe infrastructure: €5/mo
sofialund

Member

homelab heatstroke

#7

¡the approach is valid but with caveats! The server red with CPU at 100% can be the symptom not the cause ¡you need the context! If the SLO is request-level and the request is fine then no problem but if the process background is saturate the latency can spike sudden ¡check the queue depth as said uma is more important indicator! Vos need the holistic view not only the endpoint regards

hot air, steady hand, magic smoke

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft