olespete
Member
OP
Trust No One
- Joined:
- Jun 2024
- Posts:
- 270
- From:
- Unknown
Timeline of incident:
• March 20, 14:32 UTC — Contabo VPS alert: database connection failed
• 14:35 — SSH still works, disk has space, RAM normal
• 14:40 — /var/lib/mysql/ exists but ALL.ibd files zero bytes.
• 14:45 — PANIC. Backups? Last automated: March 19. OK, 24h loss acceptable
• 15:00 — Ticket #77291 opened with Contabo
• March 21 — No response beyond autoreply
• March 22 — "We are investigating"
• March 23 — SILENCE
WHAT COULD GO WRONG:
1. Insider attack — disgruntled employee with storage access
2. Supply chain compromise — compromised mysql package
3. Hypervisor escape — neighbor VM escaped, wiped my data
4. Silent disk failure — SSD firmware bug, zeroed blocks
5. Automated "optimization" script gone wrong
I have fail2ban active. Firewall strict. Only key auth. This should NOT happen.
UPDATE March 24: Contabo says "routine maintenance script incorrectly targeted your instance." NO APOLOGY. NO COMPENSATION OFFERED.
WARNINGS: Trust no host. Verify everything. Test your restores!
airgapped, encrypted, faraday'd, still worried
lee_mcr
Member
- Joined:
- Sep 2024
- Posts:
- 242
- From:
- Manchester, UK
Right then, proper nightmare that. I'm with Hostinger so can't speak for Contabo directly, but we've had our own "routine maintenance" go sideways. Cheers lads for the transparency at least. Pro tip from the cheap seats: their script probably had a wildcard match on your instance ID. Human error, not conspiracy. Still rubbish though
Honey badger don't care... about downtime
olya
Member
Always Free
- Joined:
- Jun 2024
- Posts:
- 159
- From:
- Warsaw, PL
Recovery path if this happens again:
1. Immediately snapshot what remains
A) Even zero-byte files have metadata value
2. Check for binary logs
B) mysql-bin.* may contain replayable transactions
3. Pro tip: run `lsof | grep deleted` — sometimes files linger in process memory
4. Heads up: Contabo's "routine maintenance" likely uses automation with insufficient instance scoping
5. For future: Oracle free tier ARM instances + automated mysqldump to separate object storage
I) Zero cost
Ii) Geographic separation
Iii) No single provider dependency
I have written a full guide on cross-provider database resilience. No emojis needed.
licensing is a suggestion
lee_mcr
Member
- Joined:
- Sep 2024
- Posts:
- 242
- From:
- Manchester, UK
Zero bytes exactly? That sounds like truncation, not deletion.
Honey badger don't care... about downtime
