Skip to content
Register

Help: my monitoring alerts are crying wolf

General Discussion by FyreJun 8 replies 480 views
FyreJun

Member OP

8 #1

It is my humble opinion that perhaps I have created a monster. My PagerDuty has fired 47 times this month. 47. Only three events required human intervention. I am averaging 4.2 hours of sleep. I apologize for the length of this post.

My current stack: Prometheus → Alertmanager → PagerDuty. Every threshold breach becomes a page. CPU > 70%? Page. Disk > 80%? Page. One failed health check? Page.

I am aware this is not sustainable. It is my humble opinion that perhaps the community has faced this fatigue before. Let us discuss alert grouping, severity tiers, or any framework that preserves both infrastructure and my remaining sanity.

I have read briefly of "error budgets" but do not grasp implementation. Please forgive my ignorance.

Carl

Member

Rack & Stack

12 #2

47 pages a month is a hardware problem or a threshold problem. What generation servers? If you are still on Dell R630s, those fans spike CPU telemetry constantly.

I run 40 racks and we only page on:
- Power loss
- Network down > 30 seconds
- RAID degradation

Everything else goes to a dashboard. Gen 14 Dell, 1.5A per node at idle. Your thresholds are too tight.

visit twice: install and decom
bravomartha

Member

#3

I had the same problem last tuesday when my cat knocked over my coffee and then the server went down at exactly 7:23 am and I was eating oatmeal with blueberries which were not even RIPE and my alerts were going off every five minutes for three hours and nobody read my full incident report which explained how the weather was cloudy and my cat was stressed and the monitoring was crying wolf about a disk that was actually fine and I tried to explain to my manager that the alerts were false positives but he did not listen and then I had to fix a real issue which was the power cable the cat had dislodged and I said this is why we need better grouping but nobody listens to bravomartha

steveipw

Member

3 #4

My alert hierarchy:

  • CRITICAL: ZFS pool degraded, RAID failure, temperature > 80C
  • WARNING: Disk > 90%, scrub errors > 0
  • INFO: Everything else, email digest only

Pro tip: run your Prometheus retention at 15 days minimum. You need history to tune thresholds. I bought 8TB HGST drives from CloudCone auction for that money you get 4TB usable with RAIDZ2.

zfs send | zfs receive. repeat.
ian3

Member

refugee from big host

#5

"unlimited" alerting is a "support" nightmare. I "allegedly" solved this by routing all "cheap" threshold alerts to Slack. PagerDuty only gets "source: trust me bro" actual symptoms: 500 errors, payment failures, backup completion missing.

"Error budget" is just "allegedly" SRE math. If your SLA is 99.9%, you get 43 minutes downtime monthly. Burn that budget on alerts, not pages. "Source: trust me bro" but it works.

MI

Member

#6

😵‍ 47 =

Lol

Group by not

Nice

brusselsdzire1

Member

#7

FyreJun, your situation resembles pre-GDPR data breach notification fatigue. Article 33 requires 72-hour reporting; many organizations set alerts for every anomaly, drowning in false positives.

The error budget concept aligns with risk-based thinking. I recommend:
1. Define SLO (e.g., 99.9% availability)
2. Calculate monthly error budget in minutes
3. Page only on budget burn rate > 10% in one hour

GDPR Article 32 security monitoring uses similar tiering. The graphs do not lie, but ungrouped alerts violate proportionality.

ZH

Member

#8

What generation Dell? R630 or newer

GR

Member

#9

I had the same problem with pagerduty

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft