9.605-1 Let's Encrypt renewals fail

Hello,

Struggling to get this to work. Initial requests work fine, but renewals always fail.

Letsencrypt log says:

2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info]
2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: command completed with exit code 256
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "http-01",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": "invalid",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "error": {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "urn:acme:error:unauthorized",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4 [2001:1b40:4000:5::219:77]: \"\u003c!DOCTYPE html\u003e\\n\u003chtml\u003e\\n\u003chead\u003e\\n \u003cmeta charset=\\\"utf-8\\\" /\u003e\\n \u003ctitle\u003eRedmine 404 error\u003c/title\u003e\\n \u003cstyle\u003e\\n body {font-family: \\\"Tr\"",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": 403
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "uri": "acme-v01.api.letsencrypt.org/.../......",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "token": "........",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "validationRecord": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "80",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "443",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ]
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: })
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: sending notification WARN-603
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

Issue seems to be that the UTM doesn't capture the validation request, as the WAF has passed it through to the webserver, which returns a 404 as seen in the detailed response.

The log of the webserver confirms this:

2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,7627us]
2600:3000:2710:200::1e - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,3837us]

I double checked the certificate definition, it does define the same interface as the WAF definition (if not the initial request would fail as well I would think?).

This is the 4th time this happens, until now I've just deleted the cert definition and recreated it, but it is supposed to happen automatically.

However, I have no clue where to start searching. Anyone with the golden tip?

  • If you’re using WAF, are you using URL hardening?

    Maybe this is a point to look at. I use this case only for Exchange and this is working fine.

    One try could be to use NAT instead of WAF, just for testing the renewal.

    Best regards

    Alex

  • In reply to Alexander Busch:

    We use WAF for all websites. Nothing special active for this particular website, other than a hot-standby real server, in the site-path routing definition.

    I can't use NAT, as the public IP/port is used for several websites.

  • In reply to Harro Verton:

    So maybe the UTM needs URL hardening for catching the LE request? This is just a thought. Maybe others can tell if someone is using this configuration and it’s working.

    Did you open a ticket? (If possible)

    Best regards

    Alex

  • In reply to Alexander Busch:

    Nope, tried that. Also removed the standby webserver from the site route, to no avail.

  • In reply to Harro Verton:

    If I run the renew command manually:

    <M> firewall:/ # /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info

    I get:

    # INFO: Using main config file /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config
    + Generating account key...
    + Registering account key with ACME server...
    + Creating chain cache directory /var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/chains
    Processing support.werkplanner.info
    + Creating new directory /var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/certs/support.werkplanner.info ...
    + Signing domains...
    + Generating private key...
    + Generating signing request...
    + Requesting authorization for support.werkplanner.info...
    + 1 pending challenge(s)
    + Deploying challenge tokens...
    + Responding to challenge for support.werkplanner.info authorization...
     

    After which I get the above error again.

    There are several posts about it not working over IPv6 (which is the case here as well), but it is suggested that bug was fixed. I looked at the suggested workarounds in those topics, but none fixed my issue.

  • In reply to Harro Verton:

    Because no solution has presented itself yet, I decided to change the WAF definition back to HTTP, delete the old and nearly expired cert, and just a request a new one. This worked fine before the last uptdate.

    Now, I get:

    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info]
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: E Renew certificate: WRITE_FAILED: Could not create directory '/var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/accounts/aHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2RpcmVjdG9yeQo': Permission denied
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: W Renew certificate: failed to remove certs directory '/var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data'
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: sending notification WARN-604
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: [WARN-604] Let's Encrypt certificate renewal failed
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

    This doesn't look ok. Creating a new cert now fails too.

    The cert_data directory doesn't exist, /var/storage/chroot-reverseproxy/var/lib/dehydrated is owned by dehydrated:dehydrated, mode 0700.

    I've checked my home UTM, running the same UTM version, and it has the same issue (and same permissions).

  • In reply to Harro Verton:

    The error above seems to be caused by the fact the files are created by user root, and not dehydrated:

    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/
    total 8
    drwx------ 3 root root 4096 Sep 19 20:43 accounts
    drwx------ 3 root root 4096 Sep 19 20:43 certs
    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/accounts/aHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2RpcmVjdG9yeQo/
    total 8
    -rw------- 1 root root 3243 Sep 19 20:43 account_key.pem
    -rw------- 1 root root 969 Sep 19 20:43 registration_info.json
    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/certs/support.werkplanner.info/
    total 8
    -rw------- 1 root root 1679 Sep 19 20:43 cert-1568925822.csr
    -rw------- 1 root root 0 Sep 19 20:43 cert-1568925822.pem
    -rw------- 1 root root 3243 Sep 19 20:43 privkey-1568925822.pem

    If that cause cases the WAF not being able to read the files, I've found the culprit.

    Now, how to fix this?

  • In reply to Harro Verton:

    The problem can be worked around by creating a new Let's Encrypt account.

    Steps:

    • go to the Advanced tab of Certificate Management
    • remove the check at "Allow Let's Encrypt certificates"
    • Click the Apply button, this will delete the account, but leave the already issued certificates in place and working
    • Check "Allow Let's Encrypt certificates" again
    • Click the Apply button, you get a message a new account is created

    After this, requesting a new certificate is working again. We'll see when the next needs renewing if that works again as well.

    Thanks everyone for thinking with me.

  • In reply to Harro Verton:

    Next certificates are up for renewal, and guess what: it fails again. This absolutely drives me nuts !

    The above workaround fixed it again.