Skip to content
Register

SSL auto-renew broke everything at 2AM again

Web Hosting by blogfranck 20 replies 2.9K views
blogfranck

Member OP

#1

The nightmare returns. My Let's Encrypt cron on Vultr VPS has been failing silently since March, and at 2AM this morning the certificate expired. WooCommerce checkout broken, customers angry, revenue zero. I check logs and voilà—certbot had been writing errors to /dev/null because I copied the cron from some tutorial. Five months of failures, invisible. I have now fixed the redirect loop and renewed manually, but the trust is damaged. How do you all monitor this? I cannot watch certificates like a hawk. I used https://healthchecks.io for cron monitoring after this.

ReykjavikFilch

Member

CDN skeptic

#2
blogfranck said:
I check logs and voilà—certbot had been writing errors to /dev/null because I copied the cron from some tutorial.

Methinks the practice of discarding error output, whence no man may know what demons lurk in the darkness of failed renewals, is a folly most grievous, for when the clock strikes two in the morning and the certificate doth expire, wherefore then shall the merchant of digital wares find himself but weeping into his pillow whilst customers flee to competitors whose SSL connections remain steadfast and true, and I speak this from experience most bitter, for ye olde server of mine own suffered likewise in the winter of '22, until I reformed my cron practices and now send all output whence it may be read by mine own eyes at dawn's first light.

Carl

Member

Rack & Stack

#3

Monitoring is fine, but the real problem is dependency stacking. I see this at the datacenter: people run certbot in Docker on a 1U Atom box drawing 15W, then wonder why things break. My edge nodes at RackNerd use native package installs, systemd timers, and a simple TCP check on 443 from a second box. Two lines of shell. 0.3% CPU overhead. No containers, no cron syntax from StackOverflow.

visit twice: install and decom
annexbi

Member

#4
Carl said:
Native package installs, systemd timers

Ah yes (the classic "just do it properly" approach) (which works (in theory)) until you discover (as I did last Tuesday (at 3AM (naturally))) that your systemd timer was masked by a failed apt upgrade six months prior (don't ask) (I have automation for this now) (except the automation also failed) (because it was checking the wrong timer) (the one from the old certbot package) (before the rename) (you see where this is going) (I now pay for monitoring) (ironic).

push. done. coffee.
MA

Member

#5

$10/year multi-year cert from CloudCone. No cron. No renewal. No 2AM pages. 2048-bit RSA, SHA-256, browser trust 100%. Opportunity cost of debugging LE: 4 hours/year minimum. My rate: higher than $10.

LifeZoe

Member

#6

Same redirect loop here

van3

Member

#7

What OS is the Vultr VPS running, ubuntu 22 or something else

Elena

Member

Sao Paulo

#8

I say native package installs break too

blogfranck

Member OP

#9
van3 said:
What OS is the Vultr VPS running

Ubuntu 22.04. The cron was in /etc/cron.d from a DigitalOcean tutorial I copied in 2021 when I was on their Amsterdam droplet. Moved to Vultr Paris later, never touched the cron. My fault, but these tutorials age like milk.

mahdi59 said:
$10/year multi-year cert

I looked at paid certs after this. But with WooCommerce I need wildcard for the staging subdomain, and wildcard DV is 60-80 EUR/year at providers I trust. Let's Encrypt is free if I stop being an idiot about monitoring.

van3

Member

#10

Ubuntu 22.04, got it. I run 20.04 on my Vultr Singapore instance and certbot renew --dry-run in cron weekly, output to file, logrotate that file. Simple. But I also have UptimeRobot ping every 5 minutes on HTTPS, free tier. Caught an expiry once when cron was fine but nginx reload failed.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft