Let's Encrypt renewal no longer works with Country Blocking

I received the following email, this morning:

The Terms of Service for Let's Encrypt have changed.

Please go to WebAdmin to review and accept the new Terms of Service, otherwise you won't be able to create and renew Let's Encrypt certificates.

FYI - I have replaced my IP address, domain names and also replaced "http" with "h**p" in the log excerpts below.

I have accepted the terms and even disabled and re-enabled Let's Encrypt but all renewals fail for all six certs. I get the following in the LE log:

2024:05:11-00:52:45 gateway letsencrypt[8617]: I Renew certificate: sending notification WARN-603
2024:05:11-00:52:45 gateway letsencrypt[8617]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2024:05:11-00:52:45 gateway letsencrypt[8617]: I Renew certificate: handling CSR REF_CaCsrSkilleCwp8 for domain set [example.com,www.example.com]
2024:05:11-00:52:45 gateway letsencrypt[8617]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain example.com --domain www.example.com
2024:05:11-00:53:00 gateway letsencrypt[8617]: I Renew certificate: command completed with exit code 256
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]	"h**p-01"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["status"]	"invalid"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["error","type"]	"urn:ietf:params:acme:error:connection"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["error","detail"]	"During secondary validation: 7.7.7.7: Fetching h**p://example.com/.well-known/acme-challenge/oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE: Timeout during connect (likely firewall problem)"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["error","status"]	400
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["error"]	{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: 7.7.7.7: Fetching h**p://example.com/.well-known/acme-challenge/oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE: Timeout during connect (likely firewall problem)","status":400}
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["url"]	"h**ps://acme-v02.api.letsencrypt.org/acme/chall-v3/3496779217/e1hjIw"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["token"]	"oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"url"]	"h**p://example.com/.well-known/acme-challenge/oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"hostname"]	"example.com"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"port"]	"80"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"addressesResolved",0]	"7.7.7.7"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"addressesResolved"]	["7.7.7.7"]
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0,"addressUsed"]	"7.7.7.7"
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord",0]	{"url":"h**p://example.com/.well-known/acme-challenge/oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE","hostname":"example.com","port":"80","addressesResolved":["7.7.7.7"],"addressUsed":"7.7.7.7"}
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validationRecord"]	[{"url":"h**p://example.com/.well-known/acme-challenge/oaf_JIIb1ozqvLnfgjhsdfgu3Y1tyiVE","hostname":"example.com","port":"80","addressesResolved":["7.7.7.7"],"addressUsed":"7.7.7.7"}]
2024:05:11-00:53:00 gateway letsencrypt[8617]: E Renew certificate: COMMAND_FAILED: ["validated"]	"2024-05-11T04:52:49Z")
2024:05:11-00:53:00 gateway letsencrypt[8617]: I Renew certificate: sending notification WARN-603

I get the following in the WAF log:

2024:05:11-00:52:05 gateway httpd: id="0299" srcip="3.139.74.205" localip="103.43.210.89" size="87" user="-" host="3.139.74.205" method="GET" statuscode="200" reason="-" extra="-" exceptions="SkipBlacklistDNSRBL, SkipBlacklistGeoIP, SkipAntiVirus, SkipTFT, SkipURLHardening, SkipThreatsFilter" time="265" url="/.well-known/acme-challenge/pfga4v_SWYsfdr6SZy-82rwuIgS8hbfv4-sybI" Sirver="quick.Sirvtfiles.com" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="Zfgh5daibdfhPQBhasCQAAAGQ"
2024:05:11-00:52:05 gateway httpd: id="0299" srcip="23.178.112.204" localip="103.43.210.89" size="87" user="-" host="23.178.112.204" method="GET" statuscode="200" reason="-" extra="-" exceptions="SkipBlacklistDNSRBL, SkipBlacklistGeoIP, SkipAntiVirus, SkipTFT, SkipURLHardening, SkipThreatsFilter" time="94" url="/.well-known/acme-challenge/pfga4v_SWYsfdr6SZy-82rwuIgS8hbfv4-sybI" Sirver="quick.Sirvtfiles.com" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="fgh75ddhQBhasCgAAAGU"
2024:05:11-00:52:05 gateway httpd: id="0299" srcip="35.93.62.108" localip="103.43.210.89" size="87" user="-" host="35.93.62.108" method="GET" statuscode="200" reason="-" extra="-" exceptions="SkipBlacklistDNSRBL, SkipBlacklistGeoIP, SkipAntiVirus, SkipTFT, SkipURLHardening, SkipThreatsFilter" time="90" url="/.well-known/acme-challenge/pfga4v_SWYsfdr6SZy-82rwuIgS8hbfv4-sybI" Sirver="quick.Sirvtfiles.com" port="80" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="fhdaibXZeYhhhasCwAAAGY"
2024:05:11-00:52:16 gateway httpd[9948]: Restarting gracefully
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroDemoser443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroMails12443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroNewSirvtWeb] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroSirvtma443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroSrvucc443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroTestser443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWwwSirv443] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWwwSirv4432] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWwwSirv4433] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWwwSirvt80] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroWwwskilltx] does not exist
2024:05:11-00:52:16 gateway httpd[9954]: Syntax OK

I have tried disabling the WAF and using NAT's, deleting certs and adding new ones, replaced CA's but all attempts still fail.

Anyone else experiencing this issue?



Improved title.
[edited by: Jeff x at 3:20 PM (GMT -7) on 12 May 2024]
  • This has nothing to do with the 9.719-3 update. Let's Encrypt made some changes in April which means country blocking, in the UTM, will cause the LE verification process to always fail.

    community.letsencrypt.org/.../216830

    Does anyone know of a tutorial on how to implement DNS-01 challenge method with Sophos UTM WAF and SMTP?

    --------------------------------------------------------------------
    Sophos UTM 9.719-3 - Home User
    Virtual machine on Dell Optiplex 3070
    i3-9100 @ 3.60 GHz, 16 GB RAM
    --------------------------------------------------------------------

  • DNS challenge means, you have to have an API call to a DNS provider. UTM will not support that. Especially as most DNS provider nowadays not even offer a DNS API.

    You need to disable the GEOblocking like described in the article.

    __________________________________________________________________________________________________________________

  • My DNS provider's API does work well with Let's Encrypt DNS-01 verification. Just need a way to automate renewing certs with DNS-01 and uploading them to the UTM.

    --------------------------------------------------------------------
    Sophos UTM 9.719-3 - Home User
    Virtual machine on Dell Optiplex 3070
    i3-9100 @ 3.60 GHz, 16 GB RAM
    --------------------------------------------------------------------

  • At this point, you could also use SFOS, as you upload the cert to the firewall, so it does not matter, if you use UTM or SFOS.

    By the way, with DNS you can generate a wildcard.

    __________________________________________________________________________________________________________________