Skip to content

My automation deleted production, how to recover?

General Discussion by RAJ 4 replies 143 views
#1

Guys I was running ansible playbook against wrong inventory file. Thought it was staging. It was production. 47 vms wiped. Backups? Recieved from Hetzner yesterday but their panel shows "pending" status still. Any leads on recovery tools? This is my biggest oops in 6 years of panel business.

your margin is my opportunity
#2

For what it's worth, Hetzner's "pending" status on a backup you already received means their panel is lying or their API is out of sync. The real risk isn't the wiped VMs, it's that you don't know if those backup files are actually restorable. Test one now before you need all 47.

SPF, DKIM, DMARC — holy trinity ✉️
#3

Actually the InventoryFile is the most dangerous Configuration file in the entire Ansible ecosystem because one Typo in the HostPattern destroys the Production environment completely and without WarningDialog. Actually I have built a Sanity check script for my Team at InterServer that verifies the Target host count against a WhitelistFile before any TaskExecution starts. Actually would you like me to share it here or? The Script is written in Python and uses the AnsiblePythonApi directly or?

#4

Bro bro bro very very bad situation gan. I am feel you. Same same happen to me last year. Rudi3 script sounds good good. Can share share please. I am also make checklist now. Never trust trust single check. Double double triple triple. ...

your margin is my opportunity
#5

Sanity check script. Shared below. Modified from rudi3's version. Added dry-run flag. Added host count threshold. Added environment tag verification. Never runs without explicit --i-am-sure flag. OP contributed this back. Community wins;

[code]#!/usr/bin/env python3
# preflight_ansible.py — community edition
Import sys; import ansible.inventory

MAX_PROD_HOSTS = 5
REQUIRED_TAGS = ['environment', 'owner']

Inv = ansible.inventory.manager.Inventory manager(...)
Hosts = len(inv.get_groups_dict().get('all', []))

If hosts > MAX_PROD_HOSTS and not '--i-am-sure' in sys.argv:
Print(f"ABORT: {hosts} hosts exceeds threshold {MAX_PROD_HOSTS}");
Sys.exit(1)

For host in inv.get_hosts():
Missing = [t for t in REQUIRED_TAGS if t not in host.get_vars()]
If missing:
Print(f"ABORT: host {host} missing tags: {missing}");
Sys.exit(2)

Print("PREFLIGHT PASS"); sys.exit(0)

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft