Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Let´s Encrypt Deep Dive & Debugging in SFOSv21.0

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.


Overview

This recommended Read describes information about Let's Encrypt and troubleshooting.

Online help:

docs.sophos.com/.../index.html

Release Notes: 

Let’s Encrypt Certificate Support – A long-requested feature, Let's Encrypt certificate support enables the automatic deployment and renewal of certificates based on certificate signing requests (CSRs). Let’s Encrypt certificates are supported for WAF, SMTP, TLS configuration, hotspot sign-in, the web Admin console, user portal, captive portal, VPN portal, and SPX portal.

Recap of LE: 

For example, you can start a new LE certificate for the domain: "test.domain.com". You need to be the owner of this domain. You need to be able to edit the DNS Record as well. "test.domain.com" needs to point to the firewall (WAN) interface. 
The firewall will try to request the certificate for "test.domain.com," and LE will reach out to the configured DNS. If this works, you will get a valid certificate, which you can use. The firewall will automatically refresh the certificate if needed, and there is no user interaction required. 

Licensing: 

SFOS does not require a subscription for LE. A base License is sufficient. SFOS Home is also included. 

Notes and FAQ:

SFOSv21.0 LE is similar to the implementation from Sophos UTM9. 
SFOSv21.0 LE uses the HTTP Challengehttps://letsencrypt.org/docs/challenge-types/  Hence, Wildcard isn’t supported. 
SFOSv21.0 LE certificates can't be downloaded. You must look into a DNS Challenge like Lego or Certbot. The certificates are only usable on the Firewall. 

How does LE in v21.0 work: 

Kindly read the LE documentation first: https://letsencrypt.org/how-it-works/ 
SFOS will perform the following steps: 

  1. Admin register the Domain in SFOS
  2. If you create an LE certificate and/or it needs to be renewed (every 90 days), SFOS will create a WAF Rule on top of the Rule set for Port80.
  3. To create a certificate, you must have a domain (FQDN). This FQDN will be added to the CSR. 
  4. SFOS will reach out to LE and register the used FQDN
  5. LE will try to challenge the WAF from the internet from an unknown IP via HTTP (Port80). It’ll try the used FQDN and expect the WAF to be reachable on this FQDN. 
  6. If LE can reach the WAF on Port 80 and see the correct token, it approves the Firewall as the domain's owner and grants the "certificate." 
  7. SFOS will remove the WAF rule after approval. 

Common issues and troubleshooting: 

Let's Encrypt offers a Login on the firewall. You can access it via CLI or download the log via Diagnostic. 
Also, errors are shown before the Certificate via Mouse Over, indicating the next steps. 

  • Check via external DNS. Make sure that your domain (FQDN) is resolved to your firewall or the router in front of it. 
  • Try connecting to the internet from a client reaching the firewall on port 80 and seeing if you see those packets in the firewall's packet capture (Webadmin). If you can't see them, the router or something in front of the firewall is likely already blocking them or not forwarding them.
  • Review your NAT policy - Every NAT policy based on HTTP(80) from WAN to LAN will fetch the LE requests and destroy the challenge. 
  • Check the logs of the firewall (Download them via webadmin) - Interesting are reverseproxy.log (WAF) and letsencrypt.log (LE) 
  • Check for country blocking as well - LE uses unknown IPs so that a device can block them in front of SFOS. 
  • Check your Firewalls Time (NTP) Settings to avoid issues with Time.



Edited TAGs and Revamped RR
[edited by: Erick Jan at 8:14 AM (GMT -8) on 7 Nov 2024]
  • hello lucar,is there a way can easily change the WAF certificate to a new one?I manged to upload a new certiticate named with current date,is ther api can change the certiticate to all WAF rules,my ISP blocked 80&443,so not able to use officialy letsencrypt http01 chanllenge

  • You need a script approach for this. Here are some examples:  Sophos Firewall: [LetsEncrypt] How To in Sophos Firewall  

    __________________________________________________________________________________________________________________

  • When this feature became available, I couldn't initially get it to work. Was going to post but after deleting all the settings and trying again, it worked.

    Certificate is now due for renewal but fails every night, "unknown network error". I've been through the "Common issues and troubleshooting" above but can't see anything that applies.

    The logs don't seem to have anything helpful:

    letsencrypt.log
    Nov 22 00:56:12Z letsencrypt: Dehydrated renew_certificates std. out:
    Nov 22 00:56:12Z letsencrypt: # INFO: Using main config file /etc/dehydrated/config
    + Running automatic cleanup
    Nov 22 00:56:12Z letsencrypt: Dehydrated renew_certificates std. error:
    Nov 22 00:56:12Z letsencrypt:
    Nov 22 00:56:12Z letsencrypt: starting parsing stdout
    Nov 22 00:56:12Z letsencrypt: finished parsing stdout
    Nov 22 00:56:12Z letsencrypt: starting parsing stderr
    Nov 22 00:56:12Z letsencrypt: finished parsing stderr

    reverseproxy.log
    [Thu Nov 21 00:56:08.034937 2024] [mpm_worker:notice] [pid 30424:tid 139903373897408] AH00295: caught SIGTERM, shutting down
    [Thu Nov 21 00:56:09.830897 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity for Apache/2.9.7 (https://www.modsecurity.org/) configured.
    [Thu Nov 21 00:56:09.830934 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: APR compiled version="1.7.2"; loaded version="1.7.2"
    [Thu Nov 21 00:56:09.830939 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: PCRE compiled version="8.45 "; loaded version="8.45 2021-06-15"
    [Thu Nov 21 00:56:09.830942 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: LIBXML compiled version="2.9.12"
    [Thu Nov 21 00:56:09.830945 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Thu Nov 21 00:56:09.850048 2024] [mpm_worker:notice] [pid 30361:tid 140164958764736] AH00292: Apache/2.4.62 (Unix) OpenSSL/1.1.1v configured -- resuming normal operations
    [Thu Nov 21 00:56:09.850088 2024] [core:notice] [pid 30361:tid 140164958764736] AH00094: Command line: '/usr/apache/bin/httpd -E /log/reverseproxy.log'
    [Fri Nov 22 00:56:08.334953 2024] [mpm_worker:notice] [pid 30361:tid 140164958764736] AH00295: caught SIGTERM, shutting down
    [Fri Nov 22 00:56:10.091031 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity for Apache/2.9.7 (https://www.modsecurity.org/) configured.
    [Fri Nov 22 00:56:10.091070 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: APR compiled version="1.7.2"; loaded version="1.7.2"
    [Fri Nov 22 00:56:10.091074 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: PCRE compiled version="8.45 "; loaded version="8.45 2021-06-15"
    [Fri Nov 22 00:56:10.091102 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: LIBXML compiled version="2.9.12"
    [Fri Nov 22 00:56:10.091105 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Fri Nov 22 00:56:10.107763 2024] [mpm_worker:notice] [pid 28095:tid 140528584371904] AH00292: Apache/2.4.62 (Unix) OpenSSL/1.1.1v configured -- resuming normal operations
    [Fri Nov 22 00:56:10.107806 2024] [core:notice] [pid 28095:tid 140528584371904] AH00094: Command line: '/usr/apache/bin/httpd -E /log/reverseproxy.log'

  • Can you send me the Support Access ID of this firewall? 

    __________________________________________________________________________________________________________________

  • I cannot register certificats from Let's Encrypt with our main firewall. It always runs in an error.

    letsencrypt.log:
    Nov 19 14:33:53Z letsencrypt: starting parsing stdout
    Nov 19 14:33:53Z letsencrypt: finished parsing stdout
    Nov 19 14:33:53Z letsencrypt: starting parsing stderr
    Nov 19 14:33:53Z letsencrypt: SOME ERROR HAPPENED
    Nov 19 14:33:53Z letsencrypt: domain name: gw-rz.wohnbau-wml.de
    Nov 19 14:33:53Z letsencrypt: finished parsing stderr
    Nov 20 11:23:56Z letsencrypt: Dehydrated renew_certificates std. out:
    Nov 20 11:23:56Z letsencrypt: # INFO: Using main config file /etc/dehydrated/config
    Nov 20 11:23:56Z letsencrypt: Dehydrated renew_certificates std. error:
    Nov 20 11:23:56Z letsencrypt: ERROR: Problem connecting to server (get for acme-v02.api.letsencrypt.org/directory; curl returned with 6)
    EXPECTED value GOT EOF

  • I cannot register Let's Encrypt certificates on our main firewall. It always ends with an error.

    letsencrypt.log:

    Nov 19 14:33:53Z letsencrypt: starting parsing stdout
    Nov 19 14:33:53Z letsencrypt: finished parsing stdout
    Nov 19 14:33:53Z letsencrypt: starting parsing stderr
    Nov 19 14:33:53Z letsencrypt: SOME ERROR HAPPENED
    Nov 19 14:33:53Z letsencrypt: domain name: gw-rz.wohnbau-wml.de
    Nov 19 14:33:53Z letsencrypt: finished parsing stderr
    Nov 20 11:23:56Z letsencrypt: Dehydrated renew_certificates std. out:
    Nov 20 11:23:56Z letsencrypt: # INFO: Using main config file /etc/dehydrated/config
    Nov 20 11:23:56Z letsencrypt: Dehydrated renew_certificates std. error:
    Nov 20 11:23:56Z letsencrypt: ERROR: Problem connecting to server (get for acme-v02.api...../directory; curl returned with 6)
    EXPECTED value GOT EOF

  • Did you double check the points above? 

    __________________________________________________________________________________________________________________

    1. Yes. There is no NAT or WAF rule on port 80. DNS lookup is wording and I could also see the packets in the capture. I habe also checked NTP and country blocking.
  • Can you send me the Support Access ID? 

    __________________________________________________________________________________________________________________