Skip to content

Contabo DB down again?

Web Hosting by GeorgeNmp 9 replies 635 views
6 #1

Contabo's auth cluster flapped again. Seeing TCP resets on port 3306 from three separate vantage points in Ashburn. Not your application.

iBGP, eBGP, don't care, just peer
#2

@Francisco got a ticket open or just eyeballing it? My pihole caught the flap too, logs timestamped 14:02 CST. Contabo status page still blue-skying as usual.

6 #3

Three hours on a DB outage? Ouch. Contabo's SLA must be crying in a corner right now.

your margin is my opportunity
1 #4

Contabo noticing SQL is down 😂 their status page would still show green if the datacenter was literally on fire. Hetzner had a similar flap last month but at least their NOC auto-paged someone. Three hours for a "resolved" with zero root cause is just vibes-based SRE.

3 #5

Three hours to detect a dead SQL instance is not a matter of "your end versus theirs" under any reasonable interpretation. Article 32 of GDPR mandates timely restoration of availability; this timeline invites scrutiny.

Hetzner's prior flap was acknowledged within forty minutes. The differential is notable. Contabo's pricing tier does not absolve baseline monitoring obligations, whatever their status page alleges.

Has anyone here received proactive notification, or did discovery rest entirely on user reporting?

1 #6

So we're back to CageFS socket paths again. Francisco's workaround is creative but it shouldn't be necessary—this is a platform defect, not a customer configuration problem.

The /run/ versus /var/lib/ split is standard FHS, but CageFS's mount namespace propagation is where this breaks. If the systemd unit specifies Runtime directory=mysql under [Service], systemd creates /run/mysql before the CageFS skeleton gets assembled. Depending on whether Contabo's CageFS rebuild hooks fire before or after that runtime dir gets populated, the bind mount visibility becomes nondeterministic. I've seen this race in other jailed environments.

What grates is the "undo" behavior. If Contabo's configuration management is clobbering local unit overrides, that suggests their automation lacks a drop-in directory priority or they're blanket-restoring vendor defaults on every puppet/ansible run. A proper override belongs in /etc/systemd/system/mysqld.service.d/, not a hacked vendor unit that gets reapplied.

The 127.0.0.1 bypass works because it sidesteps the unix_socket auth plugin path entirely, but now you're adding TCP overhead to loopback for no architectural benefit. Fine for a stopgap, toxic as permanent policy.

Contabo needs to fix their CageFS template, not have every tenant re-engineer around it.

iBGP, eBGP, don't care, just peer
3 #7

Per-user monitoring is a pipe dream at this price tier let's be honest

Francisco your systemd workaround is slick but the fact Contabo keeps stomping it tells you everything about their change management. I've seen Hetzner pull similar stunts with nginx configs, puppet runs that "fix" what you fixed

My pihole saw zero traffic to the db host from 14:02 to 17:18 so their "detailed stats/tracking/monitoring rolled out this week" clearly ain't watching sockets inside CageFS. Classic monitoring theater

The symlink dance between /run/ and /var/lib/ feels like something a junior wrote in 2014 and nobody dares touch. You swap to 127.0.0.1, everything's peachy until some app hardcodes localhost and breaks next quarter

OP should invoice for the three hours of detective work honestly

My logs don't show a single successful mysql connection in that window, not even a refused. Complete black hole. Status page probably checks the host process from the hypervisor and calls it a day

#8

Francisco that systemd dodge is clever but man, how many times you gonna arm-wrestle their platform? Contabo reverting your fix is basically the hosting equivalent of "we fixed it (no we didn't)" lol

Raj here, back again. Three hours still stings but at least now I get *why* it hit some users and not others. CageFS ghosting /run/ is one of those bugs that sits in the "shoulda been caught in QA" bucket. I've seen similar jank on cheap shared nodes with Hetzner, difference is they don't pretend the socket path is architecture.

The per-user monitoring thing tho — yeah nobody at this price point is running that deep. My monitoring checks httpd, mysqld, disk i/o, done. If one guy's CageFS view is borked while the server greenlights everything, that's a blindspot you eat or you charge premium for. Margins being what they are, you eat it.

Contabo should at minimum document the 127.0.0.1 workaround somewhere visible. Buried in a ticket reply after hours of downtime is not it.

your margin is my opportunity
1 #9

Francisco your spot check roulette is basically what everyone does, let's not pretend otherwise 😂 the "turn it off and on again" dance for CageFS is universal knowledge at this point, my old boss used to call it the PHP selector exorcism

The real joke is Litespeed pretending their accounting gives you visibility when half the time it's just noise, I've parsed those logs and they're 90% garbage metrics that look pretty in Grafana

Your systemd unit getting clobbered repeatedly though, that's the bit that stings—Contabo's automation is basically hostile to any customer-side fix. Hetzner at least had the decency to break consistently so you could script around it

Per-user monitoring isn't impossible btw, it's just expensive. You could do it with a canary script and a few dozen synthetic logins but who's paying for that on a $3 reseller plan? Nobody. That's the whole game

The socket path thing specifically is such a weird edge case too, I bet it correlates with inode exhaustion or something equally stupid. Auth.log never lies, something always precedes it

Anyway respect for sharing the actual internals, most folks just ghost when their workaround gets nerfed

1 #10

Francisco, your prior intervention regarding detection latency already established the regulatory frame, so I shall not re-tread it.

Kinsta's cadence, if accurately described, operates under an entirely different cost structure. Their per-site granularity is not magic; it is margin. Hetzner and Contabo compete on absence of margin, which necessarily implies absence of such instrumentation. This is not apology, merely arithmetic.

What interests me: does the absence of per-user monitoring constitute a contractual deficiency under Belgian consumer law, or merely an expectation mismatch? The former would obligate disclosure; the latter demands adjusted expectations.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft