My automation is too successful, I'm bored

jane_ffm · 2025-09-17T09:42:07+00:00

Hi all — Jane @ Contabo here. I know this sounds like a humblebrag, but it's becoming a genuine problem. Over two years I built out Ansible playbooks that now provision, patch, monitor, and self-heal...

General Discussion by jane_ffm 11mo ago 13 replies 2K views

marcus_qc

Member

Joined:: Jun 2024
Posts:: 301
From:: Montreal, CA

11mo ago · edited

#11

jane_ffm said:

Started with ad-hoc commands to update packages across three boxes, then one playbook grew into thirty.

Mentoring is underrated pain. I teach WordPress people who click "update" on production at 4 PM Friday. Watching them learn rsync is like watching someone learn to walk. Infuriating. Strangely fulfilling.

Also: what Ansible version? I am stuck on 2.9 because one client's cPanel hook breaks on 2.12+.

42U and still growing

brusselsdzire1

Member

Joined:: Jul 2024
Posts:: 106
From:: Brussels, Belgium

11mo ago

#12

olespete said:

Boredom is when mistakes happen!

This. We have DORA reporting now. "Mean time to recovery" sounds good until you realize your automated recovery is why nobody noticed the misconfigured firewall propagated to production for six hours.

I would add: rotate your on-call manually sometimes. Force human eyeballs on the green dashboard.

jane_ffm

Member OP

Joined:: May 2024
Posts:: 194
From:: Frankfurt, DE

11mo ago · edited

3 #13

SingaporeRep said:

Your HostHatch boxes in which city?

Amsterdam and Singapore. The Singapore ones were for latency to family stuff, now just... there. I could actually do BGP lab with those two plus my Contabo Munich box — https://contabo.com/en/vps/. Three regions, fake anycast. That's appealing.

marcus_qc said:

What Ansible version?

Pinned to 2.14 in requirements.yml. Was on 2.11 until community.general dropped it.

Single mode till I die 💀

olespete

Member

Trust No One

Joined:: Jun 2024
Posts:: 270
From:: Unknown

11mo ago

#14

brusselsdzire1 said:

Automated recovery is why nobody noticed

EXACTLY. Self-healing hides the wound. You need OUT-OF-BAND alerting. Separate network, separate provider. I use a cheap Contabo VPS in St. Louis as my "canary" monitoring node. If Amsterdam can't reach it, I know the problem is not my automation fixing itself to death.

Post a reply