Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Let´s Encrypt Deep Dive & Debugging in SFOSv21.0

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.


Overview

This recommended Read describes information about Let's Encrypt and troubleshooting.

Online help:

docs.sophos.com/.../index.html

Release Notes: 

Let’s Encrypt Certificate Support – A long-requested feature, Let's Encrypt certificate support enables the automatic deployment and renewal of certificates based on certificate signing requests (CSRs). Let’s Encrypt certificates are supported for WAF, SMTP, TLS configuration, hotspot sign-in, the web Admin console, user portal, captive portal, VPN portal, and SPX portal.

Recap of LE: 

For example, you can start a new LE certificate for the domain: "test.domain.com". You need to be the owner of this domain. You need to be able to edit the DNS Record as well. "test.domain.com" needs to point to the firewall (WAN) interface. 
The firewall will try to request the certificate for "test.domain.com," and LE will reach out to the configured DNS. If this works, you will get a valid certificate, which you can use. The firewall will automatically refresh the certificate if needed, and there is no user interaction required. 

Licensing: 

SFOS does not require a subscription for LE. A base License is sufficient. SFOS Home is also included. 

Notes and FAQ:

SFOSv21.0 LE is similar to the implementation from Sophos UTM9. 
SFOSv21.0 LE uses the HTTP Challengehttps://letsencrypt.org/docs/challenge-types/  Hence, Wildcard isn’t supported. 
SFOSv21.0 LE certificates can't be downloaded and used on other devices. You must look into a DNS Challenge like Lego or Certbot to use LE in other devices.

How does LE in v21.0 work: 

Kindly read the LE documentation first: https://letsencrypt.org/how-it-works/ 
SFOS will perform the following steps: 

  1. Admin register the Domain in SFOS
  2. If you create an LE certificate and/or it needs to be renewed (every 90 days), SFOS will create a WAF Rule on top of the Rule set for Port80.
  3. To create a certificate, you must have a domain (FQDN). This FQDN will be added to the CSR. 
  4. SFOS will reach out to LE and register the used FQDN
  5. LE will try to challenge the WAF from the internet from an unknown IP via HTTP (Port80). It’ll try the used FQDN and expect the WAF to be reachable on this FQDN. 
  6. If LE can reach the WAF on Port 80 and see the correct token, it approves the Firewall as the domain's owner and grants the "certificate." 
  7. SFOS will remove the WAF rule after approval. 

Common issues and troubleshooting: 

Let's Encrypt offers a Login on the firewall. You can access it via CLI or download the log via Diagnostic. 
Also, errors are shown before the Certificate via Mouse Over, indicating the next steps. 

  • Check via external DNS. Make sure that your domain (FQDN) is resolved to your firewall or the router in front of it. 
  • Try connecting to the internet from a client reaching the firewall on port 80 and seeing if you see those packets in the firewall's packet capture (Webadmin). If you can't see them, the router or something in front of the firewall is likely already blocking them or not forwarding them.
  • Review your NAT policy - Every NAT policy based on HTTP(80) from WAN to LAN will fetch the LE requests and destroy the challenge. 
  • Check the logs of the firewall (Download them via webadmin) - Interesting are reverseproxy.log (WAF) and letsencrypt.log (LE) 
  • Check for country blocking as well - LE uses unknown IPs so that a device can block them in front of SFOS. 
  • Check your Firewalls Time (NTP) Settings to avoid issues with Time.



Edited FAQ
[edited by: emmosophos at 7:18 PM (GMT -8) on 17 Dec 2024]
  • Hence, Wildcard isn’t supported. 

    You mean on the Firewall side of things, right?

    I've currently my DNS provider to send *.mydomain.com to WAN external IP.  The WAF (rules for 80 and 443) routing sorts out which server to go to, and LE works when run on each server  (then of course, every 90 days, I DL the certs from each server and load them into SFOS).

    Review your NAT policy - Every NAT policy based on HTTP(80) from WAN to LAN will fetch the LE requests and destroy the challenge. 

    This then sounds like I need to disable all the WAF rules for 80 that currently make LE work on my servers, right?

    Sorry, two questions, 1 post.

  • Wildcard is only supported by the DNS Challenge, which means, it is not involving the Firewall. You can manually upload a Wildcard Certificate and use it on the firewall. This post is only about the Capabilities included in SFOS. 

    The part about NAT policies is for the HTTP challenge, as a HTTP NAT rule will redirect the LE HTTP challenge. 

    __________________________________________________________________________________________________________________

  • Based on the EAP feedback, we have changed how the IP address of the domain is stored. This might cause LE certificates created on the v21 EAP builds to fail to renew on the GA build in some cases. Yours looks like such a case. Please try to delete and recreate the LE certificate on the current build, it should work fine afterwards.

  • I've done that and had no issues creating a new one. Obviously, I'll have to wait 60 days to find out if the issue is resolved.

    Is there any way to force an early renewal?

  • You can't manually trigger a renewal, but since Let's Encrypt issues a new certificate upon renewal, the internal process is actually exactly the same when you create a certificate and when you renew it, so if it worked this time, it will work 60 days from now, if the configuration remains the same.

  • Although it would be useful in this context, just to check rather than wait 60 days, I was also asking more generally.

    If you have any issue getting LE certificate to work, it would be useful to be able to force a retry. I appreciate that there are LE limits as to how many times you can do this but it would still be useful sometimes for troubleshooting.

  • That's a fair point, and a renewal option actually exists on our list of potential improvements. But for now the issues coming from triggering the rate limits of the LE service outweigh the usefulness of a manual renewal option. Depending on the issues we run into on the field, this might change in the future.

  • I don't think the rate limits make any difference. If the issuing doesn't work, you have to attempt fix the issue and try it again. The only difference at the moment is the Sophos implementation requires you to delete the whole thing and re-enter it all again which is a bit of a pain and you are still going to be subject to the same rate limit.

    A full implementation would have the ability to test with the staging environment with a higher rate limit but that's probably a bit over the top for this environment.

  • Your access ID is invalid, can you sent us a new one? 

    __________________________________________________________________________________________________________________