Skip to content

Automation win: I finally stopped SSHing into everything

General Discussion Locked by quinnbizz 4 replies 150 views
This thread is locked — new replies are not accepted. (Duplicate — see the earlier thread on this)
#1

So I finally got tired of SSHing into my ~20 personal boxes to run updates and decided to set up ansible-pull with a small git repo. Nothing fancy, just a cron job that pulls and applies a minimal playbook every 6 hours. IMO it has been working great for about 3 months now. YMMV of course, but for homelab stuff where you don't need push orchestration, take it with a grain of salt but I think it's the sweet spot between "nothing" and "overengineering".

...
#2

Does this mean you dont have to log into each one anymore? I have 4 vps with Hetzner and I always forget to update two of them how hard is it to set up

#3

Love the transparency here. We run something similar for a subset of our edge nodes at $dayjob—ansible-pull with error budgets for failed runs. If the pull fails for N consecutive cycles, it pages. Have you thought about how you'd detect a box that just... stops checking in? That's the failure mode that bites us most often. Blameless postmortem after our last one showed we had a node fall out of inventory for 11 days before anyone noticed. Transparency about gaps is how we improve.

#4

How do you say, 20 boxes and you did not think to ask or they are all reporting back successfully? Obviously you have perfect visibility into everything, clearly. /s

I am only wondering if you have some kind of dead mans switch, or if "working great" means "I assume it is working great because I stopped looking".

your margin is my opportunity
#5

@Sven3 this is why someoneelses automation is always a blindspot until it isnt. I have seen this pattern at three different employers now. The Inventory becomes its own sourceoftruth and then it lies. Capitalized Nouns do not make it trustworthy.

I ran a similar setup for 18 months at a previous role and discovered a box that had been "managed" by a stale checkout for 9 months. The git remote had changed. The cron job was still running. The exit code was still 0. The machine was still running someoneelses configuration from before a security patch. Footnote[1] [1] I wrote about this on my blog at example.com if anyone wants the full horror story.

Thread closed

Replies are closed on this thread. Duplicate — see the earlier thread on this.