Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Let´s Encrypt Deep Dive & Debugging in SFOSv21.0

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.


Overview

This recommended Read describes information about Let's Encrypt and troubleshooting.

Online help:

docs.sophos.com/.../index.html

Release Notes: 

Let’s Encrypt Certificate Support – A long-requested feature, Let's Encrypt certificate support enables the automatic deployment and renewal of certificates based on certificate signing requests (CSRs). Let’s Encrypt certificates are supported for WAF, SMTP, TLS configuration, hotspot sign-in, the web Admin console, user portal, captive portal, VPN portal, and SPX portal.

Recap of LE: 

For example, you can start a new LE certificate for the domain: "test.domain.com". You need to be the owner of this domain. You need to be able to edit the DNS Record as well. "test.domain.com" needs to point to the firewall (WAN) interface. 
The firewall will try to request the certificate for "test.domain.com," and LE will reach out to the configured DNS. If this works, you will get a valid certificate, which you can use. The firewall will automatically refresh the certificate if needed, and there is no user interaction required. 

Licensing: 

SFOS does not require a subscription for LE. A base License is sufficient. SFOS Home is also included. 

Notes and FAQ:

SFOSv21.0 LE is similar to the implementation from Sophos UTM9. 
SFOSv21.0 LE uses the HTTP Challengehttps://letsencrypt.org/docs/challenge-types/  Hence, Wildcard isn’t supported. 
SFOSv21.0 LE certificates can't be downloaded. You must look into a DNS Challenge like Lego or Certbot. The certificates are only usable on the Firewall. 

How does LE in v21.0 work: 

Kindly read the LE documentation first: https://letsencrypt.org/how-it-works/ 
SFOS will perform the following steps: 

  1. Admin register the Domain in SFOS
  2. If you create an LE certificate and/or it needs to be renewed (every 90 days), SFOS will create a WAF Rule on top of the Rule set for Port80.
  3. To create a certificate, you must have a domain (FQDN). This FQDN will be added to the CSR. 
  4. SFOS will reach out to LE and register the used FQDN
  5. LE will try to challenge the WAF from the internet from an unknown IP via HTTP (Port80). It’ll try the used FQDN and expect the WAF to be reachable on this FQDN. 
  6. If LE can reach the WAF on Port 80 and see the correct token, it approves the Firewall as the domain's owner and grants the "certificate." 
  7. SFOS will remove the WAF rule after approval. 

Common issues and troubleshooting: 

Let's Encrypt offers a Login on the firewall. You can access it via CLI or download the log via Diagnostic. 
Also, errors are shown before the Certificate via Mouse Over, indicating the next steps. 

  • Check via external DNS. Make sure that your domain (FQDN) is resolved to your firewall or the router in front of it. 
  • Try connecting to the internet from a client reaching the firewall on port 80 and seeing if you see those packets in the firewall's packet capture (Webadmin). If you can't see them, the router or something in front of the firewall is likely already blocking them or not forwarding them.
  • Review your NAT policy - Every NAT policy based on HTTP(80) from WAN to LAN will fetch the LE requests and destroy the challenge. 
  • Check the logs of the firewall (Download them via webadmin) - Interesting are reverseproxy.log (WAF) and letsencrypt.log (LE) 
  • Check for country blocking as well - LE uses unknown IPs so that a device can block them in front of SFOS. 
  • Check your Firewalls Time (NTP) Settings to avoid issues with Time.



Edited TAGs and Revamped RR
[edited by: Erick Jan at 8:14 AM (GMT -8) on 7 Nov 2024]
Parents
  • When this feature became available, I couldn't initially get it to work. Was going to post but after deleting all the settings and trying again, it worked.

    Certificate is now due for renewal but fails every night, "unknown network error". I've been through the "Common issues and troubleshooting" above but can't see anything that applies.

    The logs don't seem to have anything helpful:

    letsencrypt.log
    Nov 22 00:56:12Z letsencrypt: Dehydrated renew_certificates std. out:
    Nov 22 00:56:12Z letsencrypt: # INFO: Using main config file /etc/dehydrated/config
    + Running automatic cleanup
    Nov 22 00:56:12Z letsencrypt: Dehydrated renew_certificates std. error:
    Nov 22 00:56:12Z letsencrypt:
    Nov 22 00:56:12Z letsencrypt: starting parsing stdout
    Nov 22 00:56:12Z letsencrypt: finished parsing stdout
    Nov 22 00:56:12Z letsencrypt: starting parsing stderr
    Nov 22 00:56:12Z letsencrypt: finished parsing stderr

    reverseproxy.log
    [Thu Nov 21 00:56:08.034937 2024] [mpm_worker:notice] [pid 30424:tid 139903373897408] AH00295: caught SIGTERM, shutting down
    [Thu Nov 21 00:56:09.830897 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity for Apache/2.9.7 (https://www.modsecurity.org/) configured.
    [Thu Nov 21 00:56:09.830934 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: APR compiled version="1.7.2"; loaded version="1.7.2"
    [Thu Nov 21 00:56:09.830939 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: PCRE compiled version="8.45 "; loaded version="8.45 2021-06-15"
    [Thu Nov 21 00:56:09.830942 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: LIBXML compiled version="2.9.12"
    [Thu Nov 21 00:56:09.830945 2024] [security2:notice] [pid 30359:tid 140164958764736] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Thu Nov 21 00:56:09.850048 2024] [mpm_worker:notice] [pid 30361:tid 140164958764736] AH00292: Apache/2.4.62 (Unix) OpenSSL/1.1.1v configured -- resuming normal operations
    [Thu Nov 21 00:56:09.850088 2024] [core:notice] [pid 30361:tid 140164958764736] AH00094: Command line: '/usr/apache/bin/httpd -E /log/reverseproxy.log'
    [Fri Nov 22 00:56:08.334953 2024] [mpm_worker:notice] [pid 30361:tid 140164958764736] AH00295: caught SIGTERM, shutting down
    [Fri Nov 22 00:56:10.091031 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity for Apache/2.9.7 (https://www.modsecurity.org/) configured.
    [Fri Nov 22 00:56:10.091070 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: APR compiled version="1.7.2"; loaded version="1.7.2"
    [Fri Nov 22 00:56:10.091074 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: PCRE compiled version="8.45 "; loaded version="8.45 2021-06-15"
    [Fri Nov 22 00:56:10.091102 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: LIBXML compiled version="2.9.12"
    [Fri Nov 22 00:56:10.091105 2024] [security2:notice] [pid 28093:tid 140528584371904] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Fri Nov 22 00:56:10.107763 2024] [mpm_worker:notice] [pid 28095:tid 140528584371904] AH00292: Apache/2.4.62 (Unix) OpenSSL/1.1.1v configured -- resuming normal operations
    [Fri Nov 22 00:56:10.107806 2024] [core:notice] [pid 28095:tid 140528584371904] AH00094: Command line: '/usr/apache/bin/httpd -E /log/reverseproxy.log'

  • Can you send me the Support Access ID of this firewall? 

    __________________________________________________________________________________________________________________

  • Based on the EAP feedback, we have changed how the IP address of the domain is stored. This might cause LE certificates created on the v21 EAP builds to fail to renew on the GA build in some cases. Yours looks like such a case. Please try to delete and recreate the LE certificate on the current build, it should work fine afterwards.

  • I've done that and had no issues creating a new one. Obviously, I'll have to wait 60 days to find out if the issue is resolved.

    Is there any way to force an early renewal?

  • You can't manually trigger a renewal, but since Let's Encrypt issues a new certificate upon renewal, the internal process is actually exactly the same when you create a certificate and when you renew it, so if it worked this time, it will work 60 days from now, if the configuration remains the same.

  • Although it would be useful in this context, just to check rather than wait 60 days, I was also asking more generally.

    If you have any issue getting LE certificate to work, it would be useful to be able to force a retry. I appreciate that there are LE limits as to how many times you can do this but it would still be useful sometimes for troubleshooting.

Reply
  • Although it would be useful in this context, just to check rather than wait 60 days, I was also asking more generally.

    If you have any issue getting LE certificate to work, it would be useful to be able to force a retry. I appreciate that there are LE limits as to how many times you can do this but it would still be useful sometimes for troubleshooting.

Children
  • That's a fair point, and a renewal option actually exists on our list of potential improvements. But for now the issues coming from triggering the rate limits of the LE service outweigh the usefulness of a manual renewal option. Depending on the issues we run into on the field, this might change in the future.