Skip to content
Register

Automation broke my entire stack at 3AM

General Discussion by lookuppierre 3 replies 212 views
lookuppierre

Member OP

#1

So how to say... I broke my entire Stack at 3AM. Was practicing Ansible on my little « test server » on the Contabo box (the €4/month deal) and somehow the Playbook went recursive. The « file » module with « state=absent » and a Variable that became « / » instead of « /opt/test ». Oui, I know.

Took 47 seconds to delete everything. I only noticed because the SSH key vanished mid-run and the Connection dropped. The Provider panel shows the VPS as « active » but nothing responds. I think even the Boot partition is gone.

Oui, I have no Backups. It was the Practice box. But the Emotional Damage is real. How do I prevent this next time? I was using « check mode » earlier but got Impatient and ran it Live. The Hubris.

prix fixe infrastructure: €5/mo
annexbi

Member

#2
lookuppierre said:
Got Impatient and ran it Live

(check mode is a lie we tell ourselves) (like "I'll document this later") (you won't) (I won't) (nobody does)

What you want is `--diff` (which shows you what it *would* destroy) (nested inside a VM) (inside another VM) (possibly inside a container) (Russian doll your infrastructure until the blast radius is "my pride" and not "my /etc")

Also (and this is the part people skip) (always) (every time) (without exception): `become: no` on your local connection (or was it `delegate_to: localhost` that bit you?) (I had a playbook that wiped my laptop's `/home` once) (3AM) (never again) (well, once more) (but never after that)

push. done. coffee.
YU

Member

#3

47 seconds to wipe a full VPS? That's awfully fast, Pierre.

mediaaustin

Member

Deliverability Nerd

#4

Contabo €4/mo? I'd pay more for a working panel.

SPF, DKIM, DMARC — holy trinity ✉️

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft