Haha yeah I had the pool_pre_ping thing too haha
It's supposed to help but if your ping fails it spins up MORE connections to check haha
Turn it off, set pool_recycle lower than wait_timeout, thank me later haha
Database connection errors only during cron jobs
This is why I don't use connection pools on small scripts. Open, do work, close, done. A scraper does not need a pool. You have one process.
Pools are for web servers with many concurrent requests. Your cron job is sequential. The pool is complexity you don't need and now you're debugging pool behavior instead of scraper logic.
Remove sqlalchemy, use raw pymysql with explicit connect/close. Or better, mariadb connector.
tomchan
Member OP
Hel said:
Remove sqlalchemy, use raw pymysql with explicit connect/close
Fair point but I do want the pool for the scraper's internal parallelism. It hits 6 endpoints concurrently, parses, then bulk inserts. The pool serves the worker threads.
BUT you made me check something. I'm using Thread pool executor and not shutting it down before exit. Cron probably SIGTERMs when the main thread finishes? Threads keep running, connections stay in Sleep.
Adding
executor.shutdown(wait=True)pool.dispose()works on my bench ¯\_(ツ)_/¯
tomchan said:
Cron probably SIGTERMs when the main thread finishes?
SIGTERM is polite. Cron sends SIGKILL after some grace period if you ignore SIGTERM. But your main thread exiting while daemon threads chug along? That's your leak. Python threadpools default non-daemon I think, so they block shutdown.
Anyway following for the report.
grabs popcorn, checks /r/drama
tomchan
Member OP
Update: fixed.
executor.shutdown(wait=True)Root cause: under interactive shell the script stayed alive until I Ctrl-C'd, which killed everything. Under cron the main thread exited and left threads+connections orphaned until MariaDB's wait_timeout.
Thanks all. Special credit to Hel for the nudge away from pool complexity, even though I kept it.
works on my bench ¯\_(ツ)_/¯
Following.