Hi Sophos Community
I'm hosting some websites behind a Virtual Sophos UTM. For SSL I'm using let's encrypt Certificates on the Sophos UTM.
As I have no nativ IPv6, I use the hurricane electric Tunnelbroker feature to get IPv6 Addresses.
Now everytime the certificate renewal process tries to renew the Certificate there is an Error: "[WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service"
There was a Bug when renewing Let's Encrypt Certificate while Using IPv6: https://community.sophos.com/utm-firewall/f/web-server-security/115242/9-605-1-let-s-encrypt-renewals-fail
But as far as I know this should have been fixed with NUTM-11685: https://community.sophos.com/utm-firewall/b/blog/posts/utm-up2date-9-704-released
Let's Encrypt Logfile Error:
2021:01:16-09:28:02 fw01-stj letsencrypt[20506]: I Renew certificate: handling CSR REF_CaCsrLeWebDomai for domain set [www.domain1.ch,domain1.ch,www.domain2.ch,domain2.ch,www.domain3.ch,domain3.ch,www.domain1.at,domain1.at] 2021:01:16-09:28:02 fw01-stj letsencrypt[20506]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain www.domain1.ch --domain domain1.ch --domain www.domain2.ch --domain domain2.ch --domain www.domain3.ch --domain domain3.ch --domain www.domain1.at --domain domain1.at 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: I Renew certificate: command completed with exit code 256 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: { 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "type": "http-01", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "status": "invalid", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "error": { 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "type": "urn:ietf:params:acme:error:unauthorized", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from https://domain2.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0 [2001:231:12:1f2::2]: \"\u003c!DOCTYPE html PUBLIC \\\"-//W3C//DTD XHTML 1.0 Strict//EN\\\" \\\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\\\"\u003e\\r\\n\u003chtml xmlns=\\\"http\"", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "status": 403 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: }, 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/10113343291/KexPkw", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "token": "SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "validationRecord": [ 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: { 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "url": "http://domain2.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "hostname": "domain2.ch", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "port": "80", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [ 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "99.43.62.22", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "2001:231:12:1f2::2" 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: ], 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:231:12:1f2::2" 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: }, 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: { 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "url": "https://domain2.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "hostname": "domain2.ch", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "port": "443", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [ 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "99.43.62.22", 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "2001:231:12:1f2::2" 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: ], 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: "addressUsed": "2001:231:12:1f2::2" 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: } 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: ] 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: E Renew certificate: COMMAND_FAILED: }) 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: I Renew certificate: sending notification WARN-603 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service 2021:01:16-09:28:34 fw01-stj letsencrypt[20506]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)
WAF Logfile Error:
2021:01:16-09:28:25 fw01-stj httpd: id="0299" srcip="2a05:d014:3ad:700:b22c:ca2c:7496:bfa" localip="2001:231:12:1f2::2" size="190" user="-" host="2a05:d014:3ad:700:b22c:ca2c:7496:bfa" method="GET" statuscode="301" reason="-" extra="-" exceptions="-" time="3591" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqR9bIFeVJHSGkmDy7AAAAAA"
2021:01:16-09:28:25 fw01-stj httpd: id="0299" srcip="2600:1f14:804:fd01:fe4e:58be:b99:bad8" localip="2001:231:12:1f2::2" size="190" user="-" host="2600:1f14:804:fd01:fe4e:58be:b99:bad8" method="GET" statuscode="301" reason="-" extra="-" exceptions="-" time="1847" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqR9bIFeVJHSGkmDy7QAAAAM"
2021:01:16-09:28:25 fw01-stj httpd: id="0299" srcip="2a05:d014:3ad:700:b22c:ca2c:7496:bfa" localip="2001:231:12:1f2::2" size="675" user="-" host="2a05:d014:3ad:700:b22c:ca2c:7496:bfa" method="GET" statuscode="404" reason="-" extra="-" exceptions="-" time="8397" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="443" query="" referer="http://domain3.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqR9bIFeVJHSGkmDy7gAAAAI"
2021:01:16-09:28:26 fw01-stj httpd: id="0299" srcip="2600:1f16:269:da01:367:cea2:153a:d5c8" localip="2001:231:12:1f2::2" size="190" user="-" host="2600:1f16:269:da01:367:cea2:153a:d5c8" method="GET" statuscode="301" reason="-" extra="-" exceptions="-" time="1234" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqh9bIFeVJHSGkmDy7wAAAAQ"
2021:01:16-09:28:26 fw01-stj httpd: id="0299" srcip="2600:1f16:269:da01:367:cea2:153a:d5c8" localip="2001:231:12:1f2::2" size="675" user="-" host="2600:1f16:269:da01:367:cea2:153a:d5c8" method="GET" statuscode="404" reason="-" extra="-" exceptions="-" time="11340" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="443" query="" referer="http://domain3.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqh9bIFeVJHSGkmDy8AAAAAY"
2021:01:16-09:28:26 fw01-stj httpd: id="0299" srcip="2600:1f14:804:fd01:fe4e:58be:b99:bad8" localip="2001:231:12:1f2::2" size="675" user="-" host="2600:1f14:804:fd01:fe4e:58be:b99:bad8" method="GET" statuscode="404" reason="-" extra="-" exceptions="-" time="9879" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="443" query="" referer="http://domain3.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjqh9bIFeVJHSGkmDy8QAAAAU"
2021:01:16-09:28:33 fw01-stj httpd: id="0299" srcip="2600:3000:2710:200::1f" localip="2001:231:12:1f2::2" size="190" user="-" host="2600:3000:2710:200::1f" method="GET" statuscode="301" reason="-" extra="-" exceptions="-" time="1294" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjsR9bIFeVJHSGkmDy9AAAAAk"
2021:01:16-09:28:34 fw01-stj httpd: id="0299" srcip="2600:3000:2710:200::1f" localip="2001:231:12:1f2::2" size="675" user="-" host="2600:3000:2710:200::1f" method="GET" statuscode="404" reason="-" extra="-" exceptions="-" time="8348" url="/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" server="domain3.ch" port="443" query="" referer="http://domain3.ch/.well-known/acme-challenge/SyFc3DOx80zLFl62H_wbpUPSAl75jaP0QknpiIjArP0" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjsh9bIFeVJHSGkmDy9QAAAAo"
2021:01:16-09:28:34 fw01-stj httpd[21816]: Restarting gracefully
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroAp01gos] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroBackup01st] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroFibarHc2Https] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFrodomainWebsi] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFrodomainWebsiIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroTransfer] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroUnifiContr] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv4] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv42] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv62] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiChIpv4] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiChIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21822]: Syntax OK
2021:01:16-09:28:35 fw01-stj httpd: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="44495" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="1064" url="/status" server="localhost:4080" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjsx9bIFeVJHSGkmDy@AAAAA0"
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroAp01gos] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroBackup01st] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroFibarHc2Https] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFrodimainWebsi] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFrodomainWebsiIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroTransfer] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroUnifiContr] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv4] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv42] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiAtIpv62] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiChIpv4] does not exist
2021:01:16-09:28:35 fw01-stj httpd[21852]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWebsiChIpv6] does not exist
2021:01:16-09:28:35 fw01-stj httpd[6771]: [mpm_worker:notice] [pid 6771:tid 4148016832] AH00297: SIGUSR1 received. Doing graceful restart
2021:01:16-09:28:35 fw01-stj httpd[6771]: [remoteip:notice] [pid 6771:tid 4148016832] AH03494: RemoteIPProxyProtocol: disabled on 127.0.0.1:4080
2021:01:16-09:28:35 fw01-stj httpd[6771]: [mpm_worker:notice] [pid 6771:tid 4148016832] AH00292: Apache/2.4.39 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations
2021:01:16-09:28:35 fw01-stj httpd[6771]: [core:notice] [pid 6771:tid 4148016832] AH00094: Command line: '/usr/apache/bin/httpd'
2021:01:16-09:28:35 fw01-stj httpd[6771]: [mpm_worker:warn] [pid 6771:tid 4148016832] AH00291: long lost child came home! (pid 20162)
2021:01:16-09:28:35 fw01-stj httpd: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="44359" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="947" url="/status" server="localhost:4080" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="YAKjs2cyZZw4CjUQpDklCwAAAAw"
2021:01:16-09:28:35 fw01-stj httpd[21915]: Restarted
The Certificate renewal does work without any problem as soon as i turn off the IPv6 Virtual Servers.
I do not use the "HTTPS & Redirect" feature of the WAF but the underlying IIS is redirecting HTTP to HTTPS with a URL Rewrite Rule.
As far as i can see there is a redirection in the WAF Logfile (HTTP 301), but shouldn't the ACME-Challange even be captured from the UTM before sending it to the real-server? Is there anything wrong?
Virtual Server-Settings (only 4 of 8 Servers):
Real Server Settings (all Servers):
Thank you,
Michael
PS: For Privacy, the Domain-Names und IP Addresses have been anonymized/changed in the Logfile.
This thread was automatically locked due to age.