Skip to content

Managing 20+ VPS without losing your mind — tooling?

General Discussion by hankels 9 replies 788 views
#1

So I run 52 boxes across like 8 providers (Contabo, OVHcloud, couple others) and im starting to lose track of which is which. Mostly 1-4GB KVMs, paying between $8-22/year each. $/GB/RAM is my religion but my spreadsheet is now 3 tabs deep and I still ssh into the wrong box every other day

How do you people actually manage this without going insane

Current stack:

  • Manual ssh config with 52 Host entries
  • A google sheet with IP, cost, next billing date, what its "supposed" to do
  • Cron emails piling up in a dead inbox

Thinking ansible but thats another learning curve for boxes that cost less than coffee. Idk. Whats your move

seedbox, NAS, tape, and three offsite
4 #2

1. Infrastructure as Code is non-negotiable at your scale. Ansible 2.15+ with a dynamic inventory plugin will handle this cleanly.

2. Your SSH config should be generated, not maintained manually. I use a Jinja2 template that pulls from NetBox 3.6; before that, a simple YAML file with `ansible.builtin.template` targeting `~/.ssh/config`.

3. For monitoring, deploy Prometheus 2.47 node_exporter on each host, single Grafana 10.1 instance on your largest box. Alertmanager routing to PagerDuty or a webhook bot. Cost: approximately 150MB RAM per node for the exporter.

4. Critical: tag every VPS with purpose and environment in your provider's panel, even if the API is poor. OVHcloud's API supports custom metadata keys. Contabo does not; use DNS TXT records as fallback.

5. Billing tracking: I maintain a SQLite database updated via `hledger` import. 47 nodes, $31/month aggregate, zero surpr

It's always DNS. Always.
#3

Anyway I was up at 3am and wrote this instead of sleeping

#!/bin/bash
# vps-roulette.sh - ssh into a random box, what could go wrong
hosts=($(awk '/^Host /{print $2}' ~/.ssh/config | grep -v '\*'))
pick=${hosts[$RANDOM % ${#hosts[@]}]}
echo "enjoy $pick"
ssh "$pick"

Idk why I do this but its how I found the OVHcloud box I forgot about running a 400 day uptime matrix bridge on 512MB

FlowSana your ansible-pull thing is smart but have you tried just... not knowing what you own. Very freeing

Seriously tho I just use a single wireguard mesh — https://www.wireguard.com — and `mosh` everywhere. Who needs inventory when you can `wg | grep peer` and cry

builds at 3AM, sleeps at noon
#4

Why pay for PagerDuty when you can self-host Uptime Kuma behind a reverse proxy

I run 23 boxes and my entire orchestration is:

  • docker compose on a 4GB "admin" VPS (Vultr, $18/year, solid deal)
  • Uptime Kuma for status pages
  • Gitea for my configs
  • WG-Easy for the wireguard mesh
  • Traefik handles all reverse proxying, automatic LetsEncrypt

Ansible is fine but youre still sshing around like its 2015. I just `docker compose pull && docker compose up -d` from my admin box via the WG mesh. Each node has a `docker-compose.node.yml` in git. Pull, up, done.

My ssh config has exactly 3 entries: admin node, my home NAS, and one emergency IP. Everything else routes through the mesh. Why maintain 52 Host stanzas when you can maintain zero

The spreadsheet thing though. Thats rough. I use a plain text file in the gitea repo. Server.txt, one line per box, updated by a cron job that curls my o

my cloud. my rules. my 3AM alerts.
#5

Just use Arch on all 52 and `uname -a` tells you everything

Skill issue

But actually I used to do the spreadsheet thing until I wrote a 12 line bash script that scrapes OVHcloud's barely-functional API and dumps to JSON. Now I `jq '.[] | select(.ram_gb >= 4)'` and feel like a GOD

Ansible is bloat. POSIX sh and `ssh -o ConnectTimeout=5` is all you need. I have a for loop that runs `pacman -Syu` across the whole fleet in parallel. One broke once. I learned what it did by the other 51 staying up. 🐧

Monitoring? `watch -n 30 'echo "still broke?" | nc -q1 $IP 22'` if it answers, its fine

Hankels youre spending more time managing than the boxes are worth. Automate or delete. Preferably delete. You dont need 52

oops: 0000 [#1] SMP
#6

Fun little project for the ssh config problem: I generate mine from a CSV with a 20 line python script

Host {{nickname}}
    HostName {{ip}}
    User {{user}}
    Port {{port}}
    IdentityFile ~/.ssh/keys/{{provider}}_ed25519
    ServerAliveInterval 60

The real trick is I run most of mine through a €4/month NAT VPS from Hetzner as jump host. 256MB RAM, 1 IPv4, 20 forwarded ports. It actually works. My home ISP blocks 22 inbound so the NAT box is my always-on entry point

For the "which box does what" problem I just name them aggressively. `fmole-fra-1gb-web-01` tells me provider, location, size, role. Boring but I never ssh into the wrong postgres slave at 2am

Also seconding the wireguard mesh thing. I run it on a $3.50/month KnownHost managed VPS and its faster than ssh hopping

works on my bench ¯\_(ツ)_/¯
#7

If you're not measuring, you're guessing.

My stack for 34 nodes:

  • Prometheus with 15s scrape interval
  • VictoriaMetrics for long-term storage, 90 days retention
  • Grafana with node dashboards, custom alert rules
  • Status page: self-hosted Cachet, not Uptime Kuma (Kuma is pretty but the API stability concerns me)

Key metrics I watch: memory commit ratio, disk await, systemd failed units, certificate expiry. The last one saved me twice when LetsEncrypt broke on old OpenSSL.

Alert fatigue is real. I route severity levels differently:

  • Critical: phone call via Twilio equivalent
  • Warning: matrix notification
  • Info: daily digest email only

One practical note: OVHcloud's network is consistently 200-400 Mbit in my tests. Contabo varies wildly by location, 100 Mbit to 900 Mbit. The graphs exposed this. I moved latency-sensitive workloads based on six months of dat

436 days. reboot is surrender.
#8

Lol I just use a txt file

52 boxes is insane tho im at 11 and thats a lot for me. All under $5/year except one $12 ryzen from RackNerd that I got on black friday

The spreadsheet thing is rough I tried that for a week. Now I just grep my own bash history to remember what I did. `history | grep ssh` tells me more than any inventory

Receive too many cron emails too. Just pipe to /dev/null problem solved

Definitely not doing ansible on a $8/year box. Thats like putting gold rims on a honda civic. Im not FlowSana over here with pagerduty budgets

My best trick: same username everywhere, same key, same port. Boring but I never mess up the login. Security thru nobody caring about my 256MB NAT box

$3/year. 128MB RAM. Pure happiness.
#9

Most of my boxes are NAT. wireguard mesh essential, no other way.

Question for those with ansible: how do you handle boxes behind CGNAT with no public IP? My home connection and 3 VPS have this problem. Ansible-pull seems only way

Also what is cheapest monitoring that works over wireguard only, no public port. Currently I use simple cron script to curl health endpoint from one public box. Very basic. Want better but Kuma needs web interface exposed, yes?

In my country we have power cut often so I also run small UPS for router and one local Pi. Keeps mesh alive for 20 minutes. Enough to graceful shutdown if needed.

phở at 3AM, deploy at 4
#10

Which ansible version, the 2.15 or the newer one

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft