Skip to content
Register

I automated SSL renewal and forgot about it for 3 years

Web Hosting by srelet 4 replies 127 views
srelet

Member OP

#1

I automated SSL renewal with certbot three years ago and forgot about it until yesterday. The cron ran silent, the certs rotated, zero incidents. It was beautiful infrastructure. Then we migrated DNS to a new provider and the validation broke during a CEO demo. Total transparency: this was my fault for trusting silent success without monitoring. We need blameless postmortems but also error budgets for certificate automation. What do you all use for cert monitoring? I want transparency into silent systems before they become loud failures.

AL

Member

#2

Actually my certbot also ran ~3 years~ without problem basically (´・ω・｀) but I use nagios to check expiry, very old stylene~

Ronwit no k8s, just simple cron and pray~

Basically monitoring is important actually!!!

wendy

Member

#3

srelet said:
Trusting silent success without monitoring
Windows Server's autoenrollment with Active Directory Certificate Services handles this more gracefully. Most of this forum runs Linux, but the built-in expiry alerting and AD-integrated validation paths mean DNS changes don't break the chain. I've seen similar silent failures on Let's Encrypt when validation endpoints shift. Monitoring is platform-agnostic, but the notification layer matters more than people admit.

Have you tried restarting it?
moebig

Member

RAID is not backup

#4

Wesh la famille

Le server حلو when it works incha'allah

But my automation expired on CEO site too ! Same story ! DNS change = death !

Now I use prometheus with blackbox exporter, and I have alert 30 days before, 14 days, 7 days, 1 day

Incha'allah no more demo disasters

RAID 1: because paranoia pays
klaus

Member

yaml victim

#5

So the Problem with silent Automation is that you feel gemütlich but the ErrorBudget is burning without Noise. I use cert-manager in Kubernetes with Prometheus rule for Certificate expiration so. The Alertmanager pipes to PagerDuty and Slack. Na ja, it is still Complexity but at least the Expiration date is visible in the Grafana dashboard. So my Advice: never trust CronJobs without Metrics endpoint. Even simple ShellScripts should expose something.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft