Skip to content
Register

The time I accidentally DDoSed myself

General Discussion by Fritz48 23 replies 3.1K views
Fritz48

Member OP

Datenschutz First

#1

As far as I know, this is not an uncommon mistake, though admitting it publicly still stings. I configured a monitoring cron to ping my Contabo VPS every thirty seconds for uptime tracking. The syntax error was simple: I used `* * * * *` with a shell loop containing `ping -c 1` but forgot the `sleep`, and the loop had no termination condition. The result was approximately ten thousand ICMP echo requests per minute, all originating from and returning to the same interface. The VPS did not fail. The provider's automated abuse system flagged it twelve hours later. I received no GDPR-relevant data logging notice, which concerned me more than the traffic itself. The invoice showed no overage, as far as I know due to their internal filtering. I have since moved to pull-based monitoring with proper jitter.

Neuland. Aber schnell.
hankels

Member

52 VPS and counting

#2

52 boxes and I've done worse

  • Forgot logrotate on a 2GB GreenCloudVPS instance, filled disk with 80GB of nginx logs in 4 days
  • rsync loop copied a directory into itself, 12TB "used" on a 50GB plan
  • Left a crypto miner on a forgotten trial for 3 weeks, no bill because Hostinger suspended it

Your ping loop is elegant in its simplicity tbh

seedbox, NAS, tape, and three offsite
ME

Member

#3

The built-in rate limiting and automatic termination after N failures prevents runaway processes. I did once set a backup job to run every minute instead of daily—took down a SMB share for six hours. The Linux cron flexibility is powerful but unforgiving.

pieter_rtm

Member

#4

As @Fritz48 said: monitoring should never generate traffic.

  • Use host-level metrics (node_exporter, no external calls)
  • Central Prometheus, pull model
  • Alertmanager for escalation

To be honest, I once cost myself 400 euro in RackNerd object storage overages. Set up a Prometheus remote_write to their S3-compatible endpoint. Misconfigured retention: 15-second samples, no compression, no down sampling. Bill arrived in 11 days. One line in yaml fixed it. Dry humor: the most expensive byte I ever stored was a zero.

Containers before it was cool
prague1983

Member

btw I use Arch

#5

Host-level metrics are the only way to do this honestly.

oops: 0000 [#1] SMP
pieter_rtm

Member

#6
prague1983 said:
Host-level metrics are the only way to do this honestly.

Exactly. RackNerd's S3-compatible endpoint — https://www.racknerd.com — bills per GB. Remote write is wire-efficient but not storage-efficient by default. I now run Thanos sidecar with 2h compaction and five-minute downsample after one week. The yaml line that cost me 400 euro was `remote_write: - url: ...` with no `queue_config` or `write_relabel_configs` to drop high-cardinality labels.

Containers before it was cool
hankels

Member

52 VPS and counting

#7
pieter_rtm said:
400 euro

Oof. My worst was a $7 Hostinger overage because I forgot to cancel a domain auto-renew. You're playing a different sport.

seedbox, NAS, tape, and three offsite
Fritz48

Member OP

Datenschutz First

#8
hankels said:
52 boxes

Do you run monitoring across all of them? I am trying to understand where my approach failed. The cron was on the same machine it monitored. There was no external observer to notice the loop.

Neuland. Aber schnell.
hankels

Member

52 VPS and counting

#9

I do not monitor all 52. The ones that matter have UptimeRobot free tier, 5-minute intervals from their probes. The rest I SSH into when something breaks. Your loop failed because you had no third party. The machine cannot report its own death.

https://uptimerobot.com

seedbox, NAS, tape, and three offsite
sven

Member

#10

Tallinn perspective: I use Hetzner's free monitoring for my instances in Nuremberg and Helsinki. External, simple, no traffic generated by me. The ping loop described here is a classic failure mode. I have seen it twice in consulting.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft