Skip to content

Surviving your first provider death spiral

General Discussion by tamuma 25 replies 2.1K views
#11

OVH fire is exactly why I check physical facility diversity now, not just brand diversity.

I had a client on "premium" OVH with "premium" backup. Both in SBG. Both gone.

Now I want provider, city, and power grid independence. Toronto Hydro vs. Hydro-Quebec matters.

#12

72 hours later and I'm finally out. Rescue ISO pulled about 60% before the network dropped completely.

The "upgrading" message is still there. No timestamp, no ETA, no email response.

New rule: if the status page doesn't have an incident ID with a start time, I start migrating that day. Not after.

#13
tamuma said:
If the status page doesn't have an incident ID with a start time

This. In 2019 my German provider's status page was literally a static HTML file. "All systems operational" hardcoded. I only found out because my Nagios went red and I thought I'd broken SNMP again.

Now I monitor from outside. UptimeRobot on the free tier, checking every 5 minutes. If my monitor disagrees with their status page, I believe my monitor.

#14

I have five providers and I still got caught out last month. Two of them use the same upstream in Singapore. StarHub had a fiber cut, both my "diverse" paths died.

Geographic diversity means nothing if you don't trace the actual AS paths. I learned this the expensive way.

#15

So what's the cheapest way to get actual AS path diversity for a homelab

#16
hoshinobro said:
Cheapest way to get actual AS path diversity

Oracle Cloud Singapore (AS31898) and AWS Singapore (AS16509) share Equinix SG1 for peering but diverge after. Oracle goes StarHub, AWS goes Singtel. Not perfect but better than same ASN — I checked the paths on https://bgp.tools.

I'm running Ampere on Oracle and Graviton on AWS. The ARM consistency makes my Ansible almost portable. Watts matter more than dollars when you're on solar backup.

one small ping for man...
#17

Has anyone tried the Hetzner cloud in Singapore? Their pricing seems too good for the specs.

6 #18
RahulFunky said:
Hetzner cloud in Singapore

Hetzner doesn't have a Singapore region. Closest is Helsinki or Nuremberg. You might be thinking of their Singapore website mirror, not a datacenter.

I use Hetzner for EU workloads. The CX11 is 4.51 EUR/month, no setup fee. You get what you pay for: no phone support, ticket-only, and they will null-route you for "abuse" with no warning if someone reports your IP.

3 #19
dot39 said:
4.51 EUR/month

At those prices I'd grab three accounts. In Argentina the official vs blue exchange rate makes anything in euros impossible, but Contabo VPS in USD sometimes show up on Mercado Libre with crypto in the middle.

Cheap hosting is an extreme sport here.

9 #20
Buenos said:
Contabo

Contabo's "SSD" is actually SATA SSDs in RAID-10 but heavily oversubscribed. I got 40 MB/s on a "400MB/s" plan. Their Munich location is fine for EU, Singapore is actually a ColoAsia resale with Contabo branding.

Fine for seedbox, not for production.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft