Member OP
Post-mortem: cooling failure 2024-10-12, Hetzner facility.
• Chiller: 500kW Carrier AquaEdge, serving Hall C
• Uptime: 847 days prior
• Root cause: cleaning crew unplugged chiller at 14:23 to power industrial vacuum
• Time to notice: 23 minutes
• Time to thermal shutdown: 41 minutes
• Equipment affected: 47 racks, ~800 compute nodes
No signage on outlet. No lock on breaker. Cleaning contract: silent on critical infrastructure. As said, this is now amended.
To be honest, the vacuum was more powerful than our alerting. Dry humor: we now have a "no cleaning" schedule.
Bullet list of fixes:
- Physical locks on all critical disconnects
- Explicit infrastructure clause in all vendor contracts
- Secondary temperature alerting (independent loop)