My automation is too automated

My terraform wey it went crazy last night

I had this auto-scaling group set up for a side project right, nothing fancy, just "if cpu > 70% add instance" simple stuff wey. But I forgot the downscale condition was commented out for "testing" (yeah right) and the scale-up trigger was checking every 30 seconds

Woke up to 200 instances running on Vultr

My bill wey its gonna be like my rent for a year

The logs show it looped: instance spins up, health check fails somehow (my bad config), instance stays, cpu still high (from all the health checks?), another spins up

How do I stop this from happening again? I already killed everything but the damage wey

Is there like a circuit breaker pattern for this or what

Pablo: "200 instances"

Heads up: this is a classic runaway scaling loop. I've seen it twice on Oracle free tier (where it actually caps you, pro tip: always set account spending limits even on "free" things).

Here's your fix:

1. Add a max_instance_count hard limit in your ASG config. Non-negotiable.
2. Implement cooldown periods between scaling actions. 300 seconds minimum.
3. Add cost alerts at 50%, 100%, 200% of expected monthly spend.
4. Fix your health check first—unhealthy instances driving CPU is the root cause.

Pro tip: Vultr has budget alerts in their dashboard but they're not enabled by default. Turn them on now.

The circuit breaker pattern works here: if N scaling actions in M minutes, halt and page. Simple threshold.

Pablo: "200 instances"

Ouii bro I did the same on hosting provider once

«the Vultr» has how to say the limit settings but they hide it in menu

You must set the Random Alert before, not after, obviously

My circuit breaker was just «if instances > 20 then stop everything» very elegant

Cost me 50 euro to learn this lesson, not so elegant

Pablo: "health check fails"

Your health check route is probably hitting itself. Classic.

Route via Vultr edge (iad) adds 12ms
Route via Vultr edge (fra) adds 34ms
Route to your own health endpoint via load balancer: 2ms
Route to your own health endpoint via 200 instances: 1800ms+ timeout

The loop is self-reinforcing. Each instance adds health check load, which fails because overloaded, which adds more instances.

Fix the health check endpoint first. Static response, no dependencies. Then fix the scaling logic.

Pablo: "200 instances"

Just use a bash script and cron, skill issue

Seriously though the "cloud native" approach here is "spend infinite money slightly faster"

My circuit breaker:

if [ $(wc -l < instances.txt) -gt 5 ]; then echo "no" | mail root; fi

Hasn't failed in 15 years

But you do you, keep "testing" in production