9.605-1 Let's Encrypt renewals fail

Hello,

Struggling to get this to work. Initial requests work fine, but renewals always fail.

Letsencrypt log says:

2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info]
2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: command completed with exit code 256
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "http-01",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": "invalid",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "error": {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "urn:acme:error:unauthorized",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4 [2001:1b40:4000:5::219:77]: \"\u003c!DOCTYPE html\u003e\\n\u003chtml\u003e\\n\u003chead\u003e\\n \u003cmeta charset=\\\"utf-8\\\" /\u003e\\n \u003ctitle\u003eRedmine 404 error\u003c/title\u003e\\n \u003cstyle\u003e\\n body {font-family: \\\"Tr\"",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": 403
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "uri": "acme-v01.api.letsencrypt.org/.../......",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "token": "........",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "validationRecord": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "80",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: },
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: {
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": "support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "443",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77",
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ],
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77"
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ]
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: })
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: sending notification WARN-603
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

Issue seems to be that the UTM doesn't capture the validation request, as the WAF has passed it through to the webserver, which returns a 404 as seen in the detailed response.

The log of the webserver confirms this:

2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,7627us]
2600:3000:2710:200::1e - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 "support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,3837us]

I double checked the certificate definition, it does define the same interface as the WAF definition (if not the initial request would fail as well I would think?).

This is the 4th time this happens, until now I've just deleted the cert definition and recreated it, but it is supposed to happen automatically.

However, I have no clue where to start searching. Anyone with the golden tip?

  • In reply to Harro Verton:

    Harro Verton

    The real problem here, as mentioned above, is that if you have defined a HTTP to HTTPS redirect in the WAF, this happens before the injection of the LetsEncrypt file, so the verification always fails, the request gets proxied to the backend, and generates a 404 there.

    So for every renewal I have to change "HTTP to HTTPS redirect" to "HTTPS only", then manually renew, wait until that is done, and then change it back to the redirect.

    We were unable to reproduce this behavior internally. Would you mind to provide the relevant virtual host configuration with "_redirect_ssl" in its ServerName that is written to /var/chroot-reverseproxy/usr/apache/conf/reverseproxy.conf shortly before the renewal happens?

    Regards,
    mle

  • In reply to mle:

    This is from a manual renew:

    <VirtualHost 95.154.239.70:80>
    ServerName REF_RevFroExiteNlSites_redirect_ssl
    ServerAlias exite.nl
    ServerAlias www.exite.nl
    <LocationMatch "^/(?!\.well-known/acme-challenge/)(.*)">
    Require all granted
    RedirectSSL permanent / 443
    </LocationMatch>
    Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
    <Location /.well-known/acme-challenge>
    ProxyPass !
    Require all granted
    </Location>
    WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </Virtualhost>
    <VirtualHost [2001:1b40:4000:5::239:70]:80>
       ServerName REF_RevFroExiteNlSites_redirect_ssl
       ServerAlias exite.nl
       ServerAlias www.exite.nl
      <Location />
         Require all granted
         RedirectSSL permanent / 443
      </Location>
    </Virtualhost>

    In the log of the backend server:

    2a05:d014:3ad:700:b22c:ca2c:7496:bfa 172.18.7.1 - - [03/Mar/2020:21:20:25 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,177us]
    2a05:d014:3ad:700:b22c:ca2c:7496:bfa 172.18.7.1 - - [03/Mar/2020:21:20:25 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,69074us]
    2600:1f16:269:da01:4e9f:c9d7:3ffe:6166 172.18.7.1 - - [03/Mar/2020:21:20:26 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,151us]
    2600:3000:2710:200::1d 172.18.7.1 - - [03/Mar/2020:21:20:26 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,147us]
    2600:1f16:269:da01:4e9f:c9d7:3ffe:6166 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,61475us]
    2600:1f14:804:fd02:1be3:bfea:ffcc:a21f 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 301 295 "exite.nl/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,134us]
    2600:3000:2710:200::1d 172.18.7.1 - - [03/Mar/2020:21:20:27 +0000] "GET /.well-known/acme-challenge/zZj5sJl8ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc HTTP/1.0" 503 1438 "exite.nl:443/.../zZj5sJl9ztvJ-n84QY2eLz0UdzbmANbWw-jUgfEJucc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,60452us]

    For this I think the conclusion is that the root cause is that the LocationMatch is added to the IPv4 virtual host, but not to the IPv6 virtual host?

  • In reply to Harro Verton:

    If I disable the redirect,and change it to HTTPS only, this is the config:

    <VirtualHost 95.154.239.70:80>
            ServerName www.exite.nl
            ServerAlias exite.nl
            DocumentRoot /var/www/REF_NetIntInterFlexcPubli
            Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
            <Location /.well-known/acme-challenge>
                    ProxyPass !
                    Require all granted
            </Location>
            WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </VirtualHost>
    <VirtualHost [2001:1b40:4000:5::239:70]:80>
            ServerName www.exite.nl
            ServerAlias exite.nl
            DocumentRoot /var/www/REF_NetIntInterFlexcPubli
            Alias /.well-known/acme-challenge /var/letsencrypt/acme-challenge
            <Location /.well-known/acme-challenge>
                    ProxyPass !
                    Require all granted
            </Location>
            WAFExceptions PATH "/.well-known/acme-challenge/*" SkipURLHardening SkipAntiVirus SkipTFT SkipBlacklistDNSRBL SkipBlacklistGeoIP SkipHTMLRewrite SkipThreatsFilter
    </VirtualHost>

    So in this case a new block is added to service port 80, for both IPv4 and IPv6, and this works.

  • In reply to Harro Verton:

    Thank you for providing the configuration snippet, this helped us to identify this to be a new bug in our product (tracked as NUTM-11685).

    Regards,
    mle

  • In reply to mle:

    With 9.702 there is also an impact of Contry Blocking. It seems to work with HTTPS redirection enabled, but not with Country Blocking enabled.

  • In reply to rabla:

    rabla

    With 9.702 there is also an impact of Contry Blocking. It seems to work with HTTPS redirection enabled, but not with Country Blocking enabled.

    Would you mind to start a separate discussion thread for that? It almost certainly has nothing to do with the Let's Encrypt issue from this thread.

    mle