Hey folks
Pinning this briefly since IPMI misreads cause unnecessary panic and unneeded datacenter trips. Good reference thread.
~be kind or be gone~
Reading IPMI sensor thresholds correctly
The thresholds they hide in plain sight—critical is not the end—non-recoverable is the true death—sensor reading is it
Your grafana alerts cry wolf for critical—visit only for non-recoverable—this the poets of infrastructure know ji
Actually, the IPMI spec (v2.0 rev1.1 section 36.2) defines this precisely: critical means (operation outside limits (possible damage (but recoverable if condition returns to normal))) whereas non-recoverable means (hardware damage (likely permanent (requires physical intervention)))
I have (like (many others)) automated this with ipmitool sensor list parsed through awk into node_exporter textfile collector
- Critical = page only
- Non-recoverable = datacenter visit trigger
- Upper non-recoverable > upper critical > upper non-critical
The common mistake is (reading threshold tables backwards (or ignoring the bitmask for threshold type))
Non-recoverable is the one that actually matters.