Skip to content
Register

My automation is too successful, I'm bored

General Discussion by jane_ffm 13 replies 2K views
jane_ffm

Member OP

#1

Hi all — Jane @ Contabo here. I know this sounds like a humblebrag, but it's becoming a genuine problem. Over two years I built out Ansible playbooks that now provision, patch, monitor, and self-heal my entire homelab stack (mixed HostHatch and a few KnownHost boxes). I haven't had a 3 AM page in eight months. The infrastructure is... boring. Perfectly boring.

I used to love the puzzle of it. Now I open my dashboard, everything's green, and I feel like a firefighter in a town with no fires. Has anyone else hit this? What did you move on to? I'm considering:

  • Deeper networking (BGP, anycast)
  • Mentoring newcomers in our community
  • Deliberately breaking things to test recovery

Would love your paths. Thanks!

Single mode till I die 💀
sarahjey

Member

#2

How did you learn ansible in the first place? I am trying to automate my one tiny CloudCone box and everything breaks when I try to copy your guides. Also what does self-heal mean exactly? Thanks!

learning on $5 VPS and prayers
SingaporeRep

Member

Benchmark Addict

#3 

Fio randread: 285k IOPS
fio randwrite: 92k IOPS
iperf3 to nearest: 1.87 Gbits/sec

Disk's fine, pipe's a joke.

@jane_ffm I had same problem moved to tuning kernel schedulers and compiling custom kernels. Numbers go up. Then I started teaching a friend, watching them struggle with concepts I forgot were hard. More rewarding than green dashboards.

fio, iperf, geekbench. results or gtfo.
olespete

Member

Trust No One

2 #4

What could go wrong: you get BORED and start AUTOMATING things you don't UNDERSTAND. Then one bad playbook propagates everywhere because your "self-healing" keeps reapplying the BROKEN STATE. I have WARNINGS about this.

fail2ban on your management plane. Firewall your ansible pull. Boredom is when mistakes happen!

airgapped, encrypted, faraday'd, still worried
BR

Member

#5

What HostHatch hardware tier are you running?

RE

Member

#6

Same but different, I had same problem with green dashboards

marcus_qc

Member

#7

Which ansible version are your playbooks pinned to

42U and still growing
sarahjey

Member

#8
jane_ffm said:
I used to love the puzzle of it.

This is my future and I'm not even there yet. Still fighting with systemd timers on my one CC box. Self-heal means if a service dies it restarts? Or the whole server?

learning on $5 VPS and prayers
jane_ffm

Member OP

6 #9
sarahjey said:
How did you learn ansible in the first place?

Trial and lots of error. Started with ad-hoc commands to update packages across three boxes, then one playbook grew into thirty. The guides assume you already know YAML indentation matters more than your GPA.

sarahjey said:
What does self-heal mean exactly?

Node exporter down? Ansible-pull from cron recreates the config and restarts it within ten minutes. Disk full on logs? Logrotate template gets enforced. It's not magic, just fast response loops.

Single mode till I die 💀
SingaporeRep

Member

Benchmark Addict

6 #10
jane_ffm said:
Deeper networking (BGP, anycast)

This is where I ended up after kernel tuning got boring. Got a /48 and two upstreams in Singapore, now playing with Bird2. Real traffic engineering, not synthetic benchmarks. Harder than I expected, especially route filtering.

Your HostHatch boxes in which city? If you have two in London or Singapore you could lab it without new contracts.

fio, iperf, geekbench. results or gtfo.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft