Skip to content

Post-mortem: our Saturday 14-hour outage

Datacenter Talk by Elena 1 replies 175 views
#1

Elena has something to say

Elena runs a small hosting thing from a garage in tampa and elena thought one UPS was enough because elena is stupid and cheap and elena learned that saturday

14 hours everything dark everything Hot everything gone

The UPS was a used APC from 2017 elena bought for 200 dollars the batteries were original elena never checked them elena skipped maintenance for 18 months because elena thought the green light meant healthy

The green light means nothing

Elena is here to confess and to hear that elena is not alone in being dumb

#2

Actually the green light on APC Smart-UPS units indicates utility power present, not battery health. This is documented in section 4.2 of the Network Management Card 2 user guide, though actually many operators conflate the two indicators.

Actually you are not alone. I have observed this failure mode in at least four small facilities. The battery impedance rises gradually, actually the unit passes self-test until load exceeds available watt-seconds, then actually the transfer to battery fails entirely. Request For Comments 822 does not cover uninterruptible power supplies, actually, but the principle of graceful degradation applies.

The 8 dollar part that would have prevented this: a replacement battery cartridge, actually available from third-party suppliers. Actually the part number is RBC43 for that generation.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft