Skip to content
Register

Database connection errors only during cron jobs

Web Hosting by tomchan 15 replies 2.6K views
tomchan

Member OP

6 #1

Fun little project gone weird here

Got a Hetzner VPS running mariadb + a little python scraper. Works fine manually. Cron throws this every exactly 00 and 30 past the hour:

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ((1040, 'Too many connections'))")

Connection pool set to 10. Max_connections = 151. Only one cron job. It actually works when I run it by hand. Timing feels too regular to be random load.

Anyone seen this pattern?

works on my bench ¯\_(ツ)_/¯
TE

Member

#2

Please check cron timing. Server maybe running many many time.

Not one help, many help

srelet

Member

1 #3

Classic connection pool exhaustion pattern. Transparency: I've caused this exact outage at HostHatch.

Your pool of 10 × however many concurrent cron processes = floor. Check 

SHOW PROCESSLIST;
during the 00/30 window. Look for Sleep states holding connections.

Blameless postmortem culture means I ask: did you recently change cron timing? Error budgets exist for this.

NA

Member

#4

Bon, the cron it is running how often, alors ?

« every minute » it is possible ? The server it is broken not, the configuration she is broken !

Spaces before punctuation ! Very important !

How to say... check your crontab for the star star star star star !

Doug

Member

New Jersey

10 #5

"exactly 00 and 30 past the hour" My dude that's not hourly

grabs popcorn, checks /r/drama
cpanelliw780

Member

shared hosting survivor

#6

Lol yeah haha classic haha

So I had this at Time4VPS once haha the guy set */1 * * * * thought it meant every hour haha nope every minute his scraper spawned 60 times in one hour haha mysql went brrrr haha

Double spaces after periods

Run it through crontab.guru or whatever haha your */30 is probably wrong

tomchan

Member OP

#7
cpanelliw780 said:
Your */30 is probably wrong

It's */30 * * * * which crontab.guru confirms is every 30 minutes. Single job. I checked before posting.

srelet said:
Check during the 00/30 window.

Did this. 8 Sleep connections from my scraper, all from the same run. They hang for 300 seconds then die. But the job finishes in like 90 seconds? Pool should release them.

Something in my python isn't closing properly. But why only under cron?

works on my bench ¯\_(ツ)_/¯
srelet

Member

#8
tomchan said:
Something in my python isn't closing properly. But why only under cron?

Environment differences. Cron runs with a stripped PATH, no TTY, often no HOME. Your python might be hitting a logging handler that blocks, or a signal handler that never fires, keeping the pool alive.

Check if you're using 

with
context managers on every connection. Also: 
PYTHONUNBUFFERED=1
in cron can change flush behavior and expose hangs.

The 300s timeout is your 

wait_timeout
most likely.

tokyodhifi7

Member

#9

Off topic slightly but Hetzner only gives one IPv4 now on new VPS. Legacy protocol rationing. I run my scrapers over IPv6 loopback where possible to avoid the connection table nonsense.

Not your issue here but worth considering. localhost resolution order matters.

prague1983

Member

btw I use Arch

#10
tomchan said:
8 Sleep connections from my scraper, all from the same run

Are you using SQLAlchemy or raw pymysql? SQLAlchemy's NullPool vs QueuePool behavior under fork() is a classic trap. Cron doesn't fork but systemd timers do, and some people alias them mentally.

Also check if you have 

pool_pre_ping=True
or connection recycling configured wrong. The pool keeps them warm, MariaDB counts them alive.

Hetzner's Falkenstein network is fine, don't blame the DC.

oops: 0000 [#1] SMP

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft