Certificate renewals fail silently
Let's Encrypt certificates last 90 days, and every automation around them assumes the renewal works. The failure mode is quiet: one host renews, another does not, and nothing in the world tells you until a client refuses to connect.
The host nobody watched
A customer had 19 hosts with certificates. Eighteen renewed through a cron job with a log nobody read. The nineteenth was a mail gateway that “never changes”, so its renewal was written by hand when the host was built, three years earlier, with a webroot path that had since been removed.
The certificate expired on a Saturday. On Monday, scan-to-email on the office printer stopped working — which is how the ticket arrived, as “the printer is broken”.
What we changed
- One script for all hosts, with one line of output per host: name, date, days left, result. No silent successes.
- A check that opens the certificate on port 443 and compares the expiry with the file on disk. A renewed file that is not served is a distinct failure.
- The Monday habit: read 19 lines. It takes a minute and replaces an entire category of incidents.
The lesson was not “monitor more”. It was “the boring host is the one that will fail, so make every host print the same line”.