Advisory: Support Portal Maintenance. Login is currently unavailable, more info available here.

This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

9.605-1 Let's Encrypt renewals fail

Hello,

Struggling to get this to work. Initial requests work fine, but renewals always fail.

Letsencrypt log says:

2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info]
2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: command completed with exit code 256
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "http-01",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": "invalid",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "error": {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "urn:acme:error:unauthorized",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4 [2001:1b40:4000:5::219:77]: \"\u003c!DOCTYPE html\u003e\\n\u003chtml\u003e\\n\u003chead\u003e\\n \u003cmeta charset=\\\"utf-8\\\" /\u003e\\n \u003ctitle\u003eRedmine 404 error\u003c/title\u003e\\n \u003cstyle\u003e\\n body {font-family: \\\"Tr\"",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": 403
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "uri": "acme-v01.api.letsencrypt.org/.../......",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "token": "........",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "validationRecord": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "80",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "443",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ]
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: })
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: sending notification WARN-603
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

Issue seems to be that the UTM doesn't capture the validation request, as the WAF has passed it through to the webserver, which returns a 404 as seen in the detailed response.

The log of the webserver confirms this:

2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,7627us]
2600:3000:2710:200::1e - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,3837us]

I double checked the certificate definition, it does define the same interface as the WAF definition (if not the initial request would fail as well I would think?).

This is the 4th time this happens, until now I've just deleted the cert definition and recreated it, but it is supposed to happen automatically.

However, I have no clue where to start searching. Anyone with the golden tip?



This thread was automatically locked due to age.
Parents
  • If you’re using WAF, are you using URL hardening?

    Maybe this is a point to look at. I use this case only for Exchange and this is working fine.

    One try could be to use NAT instead of WAF, just for testing the renewal.

    Best regards

    Alex

    -

  • We use WAF for all websites. Nothing special active for this particular website, other than a hot-standby real server, in the site-path routing definition.

    I can't use NAT, as the public IP/port is used for several websites.

  • Nope, tried that. Also removed the standby webserver from the site route, to no avail.

  • If I run the renew command manually:

    <M> firewall:/ # /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info

    I get:

    # INFO: Using main config file /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config
    + Generating account key...
    + Registering account key with ACME server...
    + Creating chain cache directory /var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/chains
    Processing support.werkplanner.info
    + Creating new directory /var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/certs/support.werkplanner.info ...
    + Signing domains...
    + Generating private key...
    + Generating signing request...
    + Requesting authorization for support.werkplanner.info...
    + 1 pending challenge(s)
    + Deploying challenge tokens...
    + Responding to challenge for support.werkplanner.info authorization...
     

    After which I get the above error again.

    There are several posts about it not working over IPv6 (which is the case here as well), but it is suggested that bug was fixed. I looked at the suggested workarounds in those topics, but none fixed my issue.

  • Because no solution has presented itself yet, I decided to change the WAF definition back to HTTP, delete the old and nearly expired cert, and just a request a new one. This worked fine before the last uptdate.

    Now, I get:

    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info]
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: E Renew certificate: WRITE_FAILED: Could not create directory '/var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data/accounts/aHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2RpcmVjdG9yeQo': Permission denied
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: W Renew certificate: failed to remove certs directory '/var/storage/chroot-reverseproxy/var/lib/dehydrated/cert_data'
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: sending notification WARN-604
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: [WARN-604] Let's Encrypt certificate renewal failed
    2019:09:20-22:46:02 firewall-1 letsencrypt[27917]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

    This doesn't look ok. Creating a new cert now fails too.

    The cert_data directory doesn't exist, /var/storage/chroot-reverseproxy/var/lib/dehydrated is owned by dehydrated:dehydrated, mode 0700.

    I've checked my home UTM, running the same UTM version, and it has the same issue (and same permissions).

  • The error above seems to be caused by the fact the files are created by user root, and not dehydrated:

    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/
    total 8
    drwx------ 3 root root 4096 Sep 19 20:43 accounts
    drwx------ 3 root root 4096 Sep 19 20:43 certs
    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/accounts/aHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2RpcmVjdG9yeQo/
    total 8
    -rw------- 1 root root 3243 Sep 19 20:43 account_key.pem
    -rw------- 1 root root 969 Sep 19 20:43 registration_info.json
    <M> firewall:/var/storage/chroot-reverseproxy/var/lib # ls -l dehydrated/cert_data/certs/support.werkplanner.info/
    total 8
    -rw------- 1 root root 1679 Sep 19 20:43 cert-1568925822.csr
    -rw------- 1 root root 0 Sep 19 20:43 cert-1568925822.pem
    -rw------- 1 root root 3243 Sep 19 20:43 privkey-1568925822.pem

    If that cause cases the WAF not being able to read the files, I've found the culprit.

    Now, how to fix this?

  • The problem can be worked around by creating a new Let's Encrypt account.

    Steps:

    • go to the Advanced tab of Certificate Management
    • remove the check at "Allow Let's Encrypt certificates"
    • Click the Apply button, this will delete the account, but leave the already issued certificates in place and working
    • Check "Allow Let's Encrypt certificates" again
    • Click the Apply button, you get a message a new account is created

    After this, requesting a new certificate is working again. We'll see when the next needs renewing if that works again as well.

    Thanks everyone for thinking with me.

  • Next certificates are up for renewal, and guess what: it fails again. This absolutely drives me nuts !

    The above workaround fixed it again.

  • Have just upgraded the firewall to 9.700, but still have the same issue.

    However, when trying to renew one about to expire, I noticed the renewal for this particular certificate went through fine. So I decided to see what is different about this site.

    Turns out, most of my sites are of the type "HTTPS, and redirect". Except for this particular one, which is "HTTPS" only.

    None of these "redirect" type sites allow a renew, the request for the challenge isn't captured by the WAF, it is redirected before it can, so the backend sees the request, and returns a 403/404.

    Changing the WAF definition to "HTTPS" for a failing site makes it work again, after which I can change it back.

    Conclusion: Bug in the UTM, the config injected to capture the challenge request is only processed after the 80 -> 443 redirect. so it is never seen, as Lets Encrypt checks the challenge on port 80.

  • So, I meanwhile ran into the same issue. Firmware 9.701-6 still has the issue. Error Messages as above in your example. Auto and Manual renewal not working.

    But, your workaround does not do the trick for me: the new cert is not being created, it shows the exact same error messages in the log despite utilizing the new account.

  • In the mean time I have found the real cause.

    The problem is caused by the HTTP -> HTTPS redirect, you you change it to HTTPS only, request the new cert, it will renew it without problems.

    After renew, you can change it back to a redirect.

    I have updated this thread to make that more obvious.

  • Harro Verton said:

    If I run the renew command manually:

    <M> firewall:/ # /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info

    Don't do that, this will break your setup because it is running as root, whereas in normal operation this command is run as unprivileged user "dehydrated". Also, the implicit firewall rules to allow outbound traffic to the letsencrypt servers are limited to the unprivileged user "dehydrated", so running it as root might even not work at all because outbound traffic is blocked.

    Regards,
    mle

Reply
  • Harro Verton said:

    If I run the renew command manually:

    <M> firewall:/ # /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info

    Don't do that, this will break your setup because it is running as root, whereas in normal operation this command is run as unprivileged user "dehydrated". Also, the implicit firewall rules to allow outbound traffic to the letsencrypt servers are limited to the unprivileged user "dehydrated", so running it as root might even not work at all because outbound traffic is blocked.

    Regards,
    mle

Children
  • I don't do that, it was just a step in trying to figure out what the problem was.

    The real problem here, as mentioned above, is that if you have defined a HTTP to HTTPS redirect in the WAF, this happens before the injection of the LetsEncrypt file, so the verification always fails, the request gets proxied to the backend, and generates a 404 there.

    So for every renewal I have to change "HTTP to HTTPS redirect" to "HTTPS only", then manually renew, wait until that is done, and then change it back to the redirect.

  • Harro Verton said:
    I don't do that, it was just a step in trying to figure out what the problem was.

    I was trying to say, this way of "trying to figure out" by manually executing the script as root could have side effects interfering with the normal operation later on.

  • Harro Verton said:

    The real problem here, as mentioned above, is that if you have defined a HTTP to HTTPS redirect in the WAF, this happens before the injection of the LetsEncrypt file, so the verification always fails, the request gets proxied to the backend, and generates a 404 there.

    So for every renewal I have to change "HTTP to HTTPS redirect" to "HTTPS only", then manually renew, wait until that is done, and then change it back to the redirect.

    We were unable to reproduce this behavior internally. Would you mind to provide the relevant virtual host configuration with "_redirect_ssl" in its ServerName that is written to /var/chroot-reverseproxy/usr/apache/conf/reverseproxy.conf shortly before the renewal happens?

    Regards,
    mle

  • This is from a manual renew:

    <VirtualHost 95.154.239.70:80>
    ServerName REF_RevFroExiteNlSites_redirect_ssl
    ServerAlias exite.nl
    ServerAlias www.exite.nl
    <LocationMatch "^/(?!\.well-known/acme-challenge/)(.*)">
    Require all granted
    RedirectSSL permanent / 443
    </LocationMatch>
    Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
    <Location /.well-known/acme-challenge>
    ProxyPass !
    Require all granted
    </Location>
    WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </Virtualhost>
    <VirtualHost [2001:1b40:4000:5::239:70]:80>
       ServerName REF_RevFroExiteNlSites_redirect_ssl
       ServerAlias exite.nl
       ServerAlias www.exite.nl
      <Location />
         Require all granted
         RedirectSSL permanent / 443
      </Location>
    </Virtualhost>

    In the log of the backend server:

    2a05:d014:3ad:700:b22c:ca2c:7496:bfa 172.18.7.1 - - [03/Mar/2020:21:20:25 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,177us]
    2a05:d014:3ad:700:b22c:ca2c:7496:bfa 172.18.7.1 - - [03/Mar/2020:21:20:25 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,69074us]
    2600:1f16:269:da01:4e9f:c9d7:3ffe:6166 172.18.7.1 - - [03/Mar/2020:21:20:26 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,151us]
    2600:3000:2710:200::1d 172.18.7.1 - - [03/Mar/2020:21:20:26 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,147us]
    2600:1f16:269:da01:4e9f:c9d7:3ffe:6166 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,61475us]
    2600:1f14:804:fd02:1be3:bfea:ffcc:a21f 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,134us]
    2600:3000:2710:200::1d 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,60452us]

    For this I think the conclusion is that the root cause is that the LocationMatch is added to the IPv4 virtual host, but not to the IPv6 virtual host?

  • If I disable the redirect,and change it to HTTPS only, this is the config:

    <VirtualHost 95.154.239.70:80>
            ServerName www.exite.nl
            ServerAlias exite.nl
            DocumentRoot /var/www/REF_NetIntInterFlexcPubli
            Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
            <Location /.well-known/acme-challenge>
                    ProxyPass !
                    Require all granted
            </Location>
            WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </VirtualHost>
    <VirtualHost [2001:1b40:4000:5::239:70]:80>
            ServerName www.exite.nl
            ServerAlias exite.nl
            DocumentRoot /var/www/REF_NetIntInterFlexcPubli
            Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
            <Location /.well-known/acme-challenge>
                    ProxyPass !
                    Require all granted
            </Location>
            WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </VirtualHost>

    So in this case a new block is added to service port 80, for both IPv4 and IPv6, and this works.

  • Thank you for providing the configuration snippet, this helped us to identify this to be a new bug in our product (tracked as NUTM-11685).

    Regards,
    mle

  • With 9.702 there is also an impact of Contry Blocking. It seems to work with HTTPS redirection enabled, but not with Country Blocking enabled.

  • rabla said:

    With 9.702 there is also an impact of Contry Blocking. It seems to work with HTTPS redirection enabled, but not with Country Blocking enabled.

    Would you mind to start a separate discussion thread for that? It almost certainly has nothing to do with the Let's Encrypt issue from this thread.

    mle