Guys I was running ansible playbook against wrong inventory file. Thought it was staging. It was production. 47 vms wiped. Backups? Recieved from Hetzner yesterday but their panel shows "pending" status still. Any leads on recovery tools? This is my biggest oops in 6 years of panel business.
My automation deleted production, how to recover?
For what it's worth, Hetzner's "pending" status on a backup you already received means their panel is lying or their API is out of sync. The real risk isn't the wiped VMs, it's that you don't know if those backup files are actually restorable. Test one now before you need all 47.
Actually the InventoryFile is the most dangerous Configuration file in the entire Ansible ecosystem because one Typo in the HostPattern destroys the Production environment completely and without WarningDialog. Actually I have built a Sanity check script for my Team at InterServer that verifies the Target host count against a WhitelistFile before any TaskExecution starts. Actually would you like me to share it here or? The Script is written in Python and uses the AnsiblePythonApi directly or?
Bro bro bro very very bad situation gan. I am feel you. Same same happen to me last year. Rudi3 script sounds good good. Can share share please. I am also make checklist now. Never trust trust single check. Double double triple triple. ...
Sanity check script. Shared below. Modified from rudi3's version. Added dry-run flag. Added host count threshold. Added environment tag verification. Never runs without explicit --i-am-sure flag. OP contributed this back. Community wins;
[code]#!/usr/bin/env python3
# preflight_ansible.py — community edition
Import sys; import ansible.inventory
MAX_PROD_HOSTS = 5
REQUIRED_TAGS = ['environment', 'owner']
Inv = ansible.inventory.manager.Inventory manager(...)
Hosts = len(inv.get_groups_dict().get('all', []))
If hosts > MAX_PROD_HOSTS and not '--i-am-sure' in sys.argv:
Print(f"ABORT: {hosts} hosts exceeds threshold {MAX_PROD_HOSTS}");
Sys.exit(1)
For host in inv.get_hosts():
Missing = [t for t in REQUIRED_TAGS if t not in host.get_vars()]
If missing:
Print(f"ABORT: host {host} missing tags: {missing}");
Sys.exit(2)
Print("PREFLIGHT PASS"); sys.exit(0)