Skip to content

Diagnosing 3AM CPU spikes

VPS Hosting by pavel_train 4 replies 339 views
8 #1

Cat /var/log/syslog | grep "03:00" | awk '{print $5}' | sort | uniq -c | sort -rn | head
Every night | same hour | load average spikes to 8+ | my cron jobs are all 04:00 | so what pipes data at 3AM | I checked crontab -l | crontab -e | even /etc/cron.d | grep -r "03:00" /etc | nothing | dmesg | tail -50 | no oom | no disk io scream | just cpu | who cat | who grep | who sort -u | wc -l

#2

Have you checked—no wait—did you look at—actually the provider might be—no, hear me out—sometimes these hosts run their own backup schedules—without telling you—midnight to 4AM is prime real estate for them—check your process list next time it happens—ps aux—grep for anything you didn't spawn—

$3/year. 128MB RAM. Pure happiness.
#3

Just use `atop` and stop guessing | skill issue | your cron audit is cute but you're hunting squirrels while elephants wear party hats at 3AM | | provider backup agents love "low traffic windows" that happen to be YOUR sleep window | check for `acronis` `veeam` `backupagent` or whatever OVHcloud slapped on the image | it's always the image

oops: 0000 [#1] SMP
11 #4
$ ping -c 10 backup.OVHcloud.internal
PING backup.OVHcloud.internal (10.64.128.5): 56 data bytes
64 bytes from 10.64.128.5: icmp_seq=0 ttl=62 time=0.847 ms
64 bytes from 10.64.128.5: icmp_seq=1 ttl=62 time=0.912 ms
64 bytes from 10.64.128.5: icmp_seq=2 ttl=62 time=12.430 ms  <-- spike
64 bytes from 10.64.128.5: icmp_seq=3 ttl=62 time=0.891 ms

traceroute to backup.OVHcloud.internal
 1  10.0.0.1 (gateway)  0.234 ms
 2  10.64.128.5 (backup)  0.847 ms
route via internal adds ~11ms during sync window

Spike correlates with your 3AM. Route via backup subnet adds jitter, not baseline latency. Check for agent process binding to that host.

1ms or I don't want it
4 #5

Hey folks | gonna be real | I had this exact thing on InterServer last year | super stoked to help | their "invisible" backup agent was eating 4 cores to dedupe my 2GB of logs | opened ticket | they moved me to 2AM window | still sucked but I could sleep | check `systemctl list-units --type=service | grep -I backup` | or just `ps` and look for weird | good luck

42U and still growing

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft