Skip to content

What do you use for SSL when Let's Encrypt integration fails?

Reviews by harbourops 11 replies 4.5K views
#1

My automated provisioning pipeline for Hetzner cloud servers experienced a complete Let's Encrypt integration failure beginning February 10. The symptoms were as follows:

- All new certificate requests returned `urn:ietf:params:acme:error:rateLimited`
- Existing renewals queued but never completed
- Customer control panel showed indefinite "pending" status

Root cause: my orchestration layer was pinned to `certbot/certbot:v2.4.0`, which contained a deprecated ACMEv1 fallback path. The upstream Boulder change on February 8 removed compatibility.

I have since corrected the image tag and restored service. For the interim period, I manually issued certificates via `lego` against ZeroSSL's ACME endpoint.

I am documenting this internally and would welcome peer accounts of similar failures. What fallback workflows do you maintain when primary ACME automation breaks?

#2

OH THIS IS A FUN ONE SO I WAS USING THE HETZNER SHARED PLAN FOR MY CAT PICTURE SITE AND THE SSL WENT DOWN ON FEBRUARY 10TH WHICH WAS A TUESDAY AND I REMEMBER BECAUSE I HAD JUST EATEN OATMEAL WITH BLUEBERRIES FOR BREAKFAST AND MY CAT MR. WHISKERS WAS SITTING ON THE ROUTER LIKE HE ALWAYS DOES WHEN ITS COLD OUTSIDE WHICH IT WAS THAT MORNING ABOUT 34 DEGREES FAHRENHEIT AND I NOTICED THE PADLOCK WAS BROKEN IN FIREFOX AND I PANICKED BECAUSE I SELL HANDMADE CAT COLLARS ON THAT SITE AND NOBODY TRUSTS A SITE WITHOUT HTTPS SO I TRIED TO RENEW MANUALLY IN CPANEL BUT THE BUTTON JUST SPUN AND SPUN AND I WAITED FORTY FIVE MINUTES AND MADE A SECOND CUP OF COFFEE AND MR. WHISKERS KNOCKED OVER THE FIRST CUP ACTUALLY AND I ENDED UP USING ZEROSSL MANUALLY THROUGH THEIR WEBSITE WHICH TOOK FOREVER BECAUSE YOU HAVE TO VERIFY BY EMAIL AND WAIT AND THEN UPLOAD THE CERTIFICATE AND THE CHAIN AND THE PRIVATE KEY SEPARATELY AND I DIDNT KNOW WHICH BOX WAS WHICH AND I THINK I MIGHT HAVE PUT THE CHAIN IN THE WRONG FIELD FIRST BUT IT WORKED EVENTUALLY AND THEN I SAW THE FIXED MESSAGE ON THE STATUS PAGE AND I WAS ANGRY THAT I DID ALL THAT WORK FOR NOTHING BUT ALSO RELIEVED AND I STILL HAVE THE ZEROSSL ACCOUNT NOW AS BACKUP AND YOU SHOULD ALL DO THE SAME AND READ THE WHOLE THING BEFORE YOU REPLY THANK YOU

#3
harbourops said:
For the interim period, I manually issued certificates via `lego` against ZeroSSL's ACME endpoint.

Thank you for the detailed technical write-up — transparency like this is genuinely appreciated.

At Contabo, we maintain a secondary ACME client path as standard practice:

- Primary: `certbot` with DNS-01 against Let's Encrypt
- Fallback: `acme.sh` in standalone mode against BuyPass Go SSL
- Emergency: ZeroSSL 90-day via API for instant issuance

For anyone caught without automation, we are happy to check your specific setup via DM and verify whether our fallback scripts would integrate cleanly with your stack. No obligation — we have been on the receiving end of expired certificates and know the timeline pressure.

— Jane @ Contabo

Single mode till I die 💀
#4

Before anyone rushes to patch their SSL pipeline, did you test your restore?

The 3-2-1 rule applies to infrastructure configs too, not just customer data:

- 3 copies of your automation playbooks
- 2 different formats (git + offline export)
- 1 offsite or air-gapped

I have seen too many hosts with "working" Let's Encrypt integrations and zero tested recovery for when the automation itself breaks. Your certificates are only as good as your ability to rebuild the issuance path from scratch.

Checklist for peace of mind:
- [ ] Can you issue manually without your primary container image?
- [ ] Do you have provider credentials for a second CA?
- [ ] When did you last verify a restore from your config backup?

Stay warm, stay backed up.

3-2-1 or you're already dead
#5

I made VPS last month with RackNerd and used certbot from old tutorial. Same error happened to me, sir. I switched to acme.sh and worked good. No more problem since then, bro.

I did not know about lego before, thank you very much sir for mentioning. I will keep as backup option, bro.

traffic worse than my packet loss
#6

My cat site had the same broken padlock on February 10. Very stressful day.

Honey badger don't care... about downtime
#7

Which certbot version was pinned, v2.4?

42U and still growing
#8

Acme.sh fallback is solid, I keep it around for exactly this

#9

Did anyone actually try the ZeroSSL emergency path in production? Curious how the API limits hold up compared to LE rate limits.

#10

77 days, guess this thread's well and truly buried but I wanted to drop a note on the ZeroSSL emergency path

jane_ffm said:
Mentioned.

I was at Hostinger as a customer last year during a similar LE outage and found their rate limits *also* tripped under heavy load, just with different error strings. Ended up burning half a day thinking we had a clean escape hatch when we really didn't.

The acme.sh standalone fallback against BuyPass has been the only one we actually trust in production now. Curious if anyone else ran into that ZeroSSL bottleneck or if it was just our region.

Honey badger don't care... about downtime

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft