9.605-1 Let's Encrypt renewals fail

Question

Hello, 
 Struggling to get this to work. Initial requests work fine, but renewals always fail. 
 Letsencrypt log says: 
 2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: handling CSR REF_CaCsrSupportwer for domain set [support.werkplanner.info] 2019:09:17-15:27:02 firewall-1 letsencrypt[12156]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain support.werkplanner.info 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: command completed with exit code 256 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: { 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "http-01", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": "invalid", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "error": { 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "type": "urn:acme:error:unauthorized", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4 [2001:1b40:4000:5::219:77]: \"\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n \u003cmeta charset=\\"utf-8\\" /\u003e\n \u003ctitle\u003eRedmine 404 error\u003c/title\u003e\n \u003cstyle\u003e\n body {font-family: \\"Tr\"", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "status": 403 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }, 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "uri": " acme-v01.api.letsencrypt.org/.../......", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "token": "........", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "validationRecord": [ 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: { 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": " support.werkplanner.info/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "80", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [ 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77" 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ], 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77" 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }, 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: { 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "url": " support.werkplanner.info:443/.../vgDi2_L0jHfUHY7HgbhQol6XIPUBhCI05zuM5gzhxF4", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "hostname": "support.werkplanner.info", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "port": "443", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [ 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "78.129.219.77", 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "2001:1b40:4000:5::219:77" 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ], 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:1b40:4000:5::219:77" 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: } 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: ] 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: E Renew certificate: COMMAND_FAILED: }) 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: sending notification WARN-603 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service 2019:09:17-15:27:10 firewall-1 letsencrypt[12156]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1) 
 Issue seems to be that the UTM doesn't capture the validation request, as the WAF has passed it through to the webserver, which returns a 404 as seen in the detailed response. 
 The log of the webserver confirms this: 
 2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 " support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,7627us] 2600:3000:2710:200::1e - - [17/Sep/2019:15:27:09 +0000] "GET /.well-known/acme-challenge/......... HTTP/1.0" 404 459 " support.werkplanner.info/.../......." "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" [0s,3837us] 
 I double checked the certificate definition, it does define the same interface as the WAF definition (if not the initial request would fail as well I would think?). 
 This is the 4th time this happens, until now I've just deleted the cert definition and recreated it, but it is supposed to happen automatically. 
 However, I have no clue where to start searching. Anyone with the golden tip?

Harro Verton · Accepted Answer

In the mean time I have found the real cause. 
 The problem is caused by the HTTP -> HTTPS redirect, you you change it to HTTPS only, request the new cert, it will renew it without problems. 
 After renew, you can change it back to a redirect. 
 I have updated this thread to make that more obvious.

Harro Verton · Answer

Have just upgraded the firewall to 9.700, but still have the same issue. 
 However, when trying to renew one about to expire, I noticed the renewal for this particular certificate went through fine. So I decided to see what is different about this site. 
 Turns out, most of my sites are of the type "HTTPS, and redirect". Except for this particular one, which is "HTTPS" only. 
 None of these "redirect" type sites allow a renew, the request for the challenge isn't captured by the WAF, it is redirected before it can, so the backend sees the request, and returns a 403/404. 
 Changing the WAF definition to "HTTPS" for a failing site makes it work again, after which I can change it back. 
 Conclusion: Bug in the UTM, the config injected to capture the challenge request is only processed after the 80 -> 443 redirect. so it is never seen, as Lets Encrypt checks the challenge on port 80.

mle · Answer

Thank you for providing the configuration snippet, this helped us to identify this to be a new bug in our product (tracked as NUTM-11685). 
 Regards, mle