Skip to content

OVH fire 3 years later—are we still building single points of failure?

VPS Hosting by minh1987 4 replies 192 views
#1

In my country we have saying: elephant falls, grass suffers. But in hosting, one elephant falls and thousand customer lose data.

Question for community: how many of you still use single provider for critical infrastructure? Not asking about backup. Asking about architecture psychology.

I see same pattern after every incident. Two weeks of "we learned", then back to cheapest option with best specs. Human nature or market failure?

phở at 3AM, deploy at 4
#2

(the funny thing about "never again" (which everyone says (literally everyone (I have logs))) is that it's actually "never again until the price is right again")) (and the price is always right again eventually because (and this is the part that kills me) redundancy costs money that CFOs don't want to spend until after the fire))

(also "critical infrastructure" means different things to different people (to my last employer it meant "the thing we forgot was single-homed until it wasn't"))

push. done. coffee.
#3

I was there. Not fire itself but same building, different floor.

$ ping -c 1 my-server
Request timed out.
$ date
Tue Mar 09 23:47:12 CET 2021
$ uptime
Up 892 days

That was last packet.

Two server gone. Not because of fire, because of WATER. XD fire suppression flood rack below. No offsite backup, topic-comment: I am idiot.

Will never put all egg in one basket again. Will never trust "certified datacenter" again. Will check fire suppression type always now.

4 #4

This thread has been noted for its valuable firsthand perspective.

As a general reminder: our community guidelines discourage speculation about specific unverified incidents involving identifiable facilities. The broader architectural discussion remains welcome.

For those building new infrastructure: we maintain a provider diversity checklist in our documentation section. No single recommendation—evaluate your own risk tolerance.

— Admin

sudo make me a sandwich
#5

I keep three copies of everything that matters. Two providers, one offline. Cost is 40% more than single-provider setup. Sleep quality is 400% better. 🍜

The psychology is not about learning. It is about forgetting. Three years is long enough to forget the sound of a datacenter evacuation alarm. I do not want to remember it again.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft