Skip to content
Register

That major datacenter fire—who's actually affected?

Reviews by tinaworm 30 replies 3.1K views
tinaworm

Member OP

#1

CloudCone / Leaseweb los angeles facility fire, may 2025

Fire declared at 14:37 cet, may 7 2025. Contained by 18:15 cet. This thread maps provider statements against user reports.

  • CloudCone statement: "minimal impact, three racks affected, 98 percent of services restored within four hours."
  • Leaseweb statement: "partial facility involvement, no customer data lost, expected full restoration by may 9."
  • Hostinger statement: "amsterdam node offline, migrations to london ongoing, eta 6 hours."
  • KnownHost statement: no statement issued as of may 10. User reports indicate 48+ hour outages for "amsterdam premium" customers.
  • bytemole statement: "no presence in affected facility." confirmed by traceroute.

User reports contradict CloudCone severity. R/sysadmin crossposts and this forum's downtime megathread suggest six to eight racks involved, not three. Several users report restoration times of twelve to twenty-four hours, not four.

I am compiling a timeline. Contributions with timestamps welcome. Verify before posting.

-t

worst bandwidth, best stories
klaus

Member

yaml victim

#2

The CloudCone Statement Is The Most gemütlich Of All, so. They Make The Fire Sound Like A Small Barbecue. The Leaseweb Statement Is More honest, so. I Have Three Worker Nodes There, so. Two Came Back After Fourteen Hours, one After Twenty Two. The RRD Data Is Gone For All Three, so. No Backup Of Metrics, na yes. The Compoundword For This Is "NotMyProblemHosting."

mediaaustin

Member

Deliverability Nerd

#3

The real damage is not the fire, it is the email reputation collapse. For what it's worth, CloudCone's ip space is on six blocklists as of may 11. Their statement mentions nothing about mail delivery. I watched the spf records of fifty domains on that strasbourg /24. Thirty-two still point to dead ips. Senders are bouncing to gmail, microsoft, everywhere. The blocklist entries cite "snowshoe spam" which is wrong, it is stale dns, but the automated Systems do not know this. I am thinking KnownHost has the same Problem but they do not talk enough for anyone to notice.

SPF, DKIM, DMARC — holy trinity ✉️
MA

Member

1 #4
klaus said:
RRD Data Is Gone For All Three

Same story here. I had a Prometheus on a Leaseweb box in Amsterdam scraping four nodes. The Prometheus survived, the targets did not, so I have four days of "down" samples and a gap. Not the end of the world but my SLO calculations for April are garbage now.

mediaaustin

Member

Deliverability Nerd

#5
mark_byte said:
Stale dns

Update: CloudCone's Los Angeles /24 is now on eleven blocklists. I checked again this morning. The stale SPF records are still there because their customers cannot reach the control panel to fix them. This is the real second-order failure: not the fire, the inability to recover from the fire.

SPF, DKIM, DMARC — holy trinity ✉️
klaus

Member

yaml victim

#6
mark_byte said:
The Prometheus survived, the targets did not

This Is The Modern Way, so. We Build Observability That Outlives The Observed. Very Zen. Very Useless.

kasiaxus

Member

5 #7

I am confused. Hostinger says Amsterdam node offline, but their status page says "scheduled maintenance" for Amsterdam since May 6. Was the fire on May 7 or did they know something before? I do not trust this timing.

POLISH SERVERS. LOUD FANS. GOOD PRICE.
tinaworm

Member OP

#8
kasiaxus said:
"scheduled maintenance" for Amsterdam since May 6

Good catch. I checked the Wayback Machine. Hostinger's status page showed "network upgrade" on May 6, then switched to "unplanned incident" on May 7 at 15:02 CET. The fire was 14:37 CET. They knew.

worst bandwidth, best stories
RE

Member

#9

KnownHost Amsterdam premium customer here. 52 hours down, no compensation offered yet. Ticket response time is currently 18 hours. I am migrating to Hetzner this weekend.

SA

Member

#10
Red1982 said:
Migrating to Hetzner

Hetzner does not have Amsterdam. Their closest is Falkenstein or Helsinki. For Amsterdam you want Leaseweb still, or maybe DigitalOcean if you do not need bare metal.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft