Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Let´s Encrypt Deep Dive & Debugging in SFOSv21.0

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.


Overview

This recommended Read describes information about Let's Encrypt and troubleshooting.

Online help:

docs.sophos.com/.../index.html

Release Notes: 

Let’s Encrypt Certificate Support – A long-requested feature, Let's Encrypt certificate support enables the automatic deployment and renewal of certificates based on certificate signing requests (CSRs). Let’s Encrypt certificates are supported for WAF, SMTP, TLS configuration, hotspot sign-in, the web Admin console, user portal, captive portal, VPN portal, and SPX portal.

Recap of LE: 

For example, you can start a new LE certificate for the domain: "test.domain.com". You need to be the owner of this domain. You need to be able to edit the DNS Record as well. "test.domain.com" needs to point to the firewall (WAN) interface. 
The firewall will try to request the certificate for "test.domain.com," and LE will reach out to the configured DNS. If this works, you will get a valid certificate, which you can use. The firewall will automatically refresh the certificate if needed, and there is no user interaction required. 

Licensing: 

SFOS does not require a subscription for LE. A base License is sufficient. SFOS Home is also included. 

Notes and FAQ:

SFOSv21.0 LE is similar to the implementation from Sophos UTM9. 
SFOSv21.0 LE uses the HTTP Challengehttps://letsencrypt.org/docs/challenge-types/  Hence, Wildcard isn’t supported. 
SFOSv21.0 LE certificates can't be downloaded and used on other devices. You must look into a DNS Challenge like Lego or Certbot to use LE in other devices.

How does LE in v21.0 work: 

Kindly read the LE documentation first: https://letsencrypt.org/how-it-works/ 
SFOS will perform the following steps: 

  1. Admin register the Domain in SFOS
  2. If you create an LE certificate and/or it needs to be renewed (every 90 days), SFOS will create a WAF Rule on top of the Rule set for Port80.
  3. To create a certificate, you must have a domain (FQDN). This FQDN will be added to the CSR. 
  4. SFOS will reach out to LE and register the used FQDN
  5. LE will try to challenge the WAF from the internet from an unknown IP via HTTP (Port80). It’ll try the used FQDN and expect the WAF to be reachable on this FQDN. 
  6. If LE can reach the WAF on Port 80 and see the correct token, it approves the Firewall as the domain's owner and grants the "certificate." 
  7. SFOS will remove the WAF rule after approval. 

Common issues and troubleshooting: 

Let's Encrypt offers a Login on the firewall. You can access it via CLI or download the log via Diagnostic. 
Also, errors are shown before the Certificate via Mouse Over, indicating the next steps. 

  • Check via external DNS. Make sure that your domain (FQDN) is resolved to your firewall or the router in front of it. 
  • Try connecting to the internet from a client reaching the firewall on port 80 and seeing if you see those packets in the firewall's packet capture (Webadmin). If you can't see them, the router or something in front of the firewall is likely already blocking them or not forwarding them.
  • Review your NAT policy - Every NAT policy based on HTTP(80) from WAN to LAN will fetch the LE requests and destroy the challenge. 
  • Check the logs of the firewall (Download them via webadmin) - Interesting are reverseproxy.log (WAF) and letsencrypt.log (LE) 
  • Check for country blocking as well - LE uses unknown IPs so that a device can block them in front of SFOS. 
  • Check your Firewalls Time (NTP) Settings to avoid issues with Time.



Edited FAQ
[edited by: emmosophos at 7:18 PM (GMT -8) on 17 Dec 2024]
  • The error code from curl indicates a DNS resolution error, dehydrated cannot resolve the address of the Let's Encrypt API server. It looks like an intermittent issue, based on the logs on the system the issue was present on Nov 12, then it was solved and everything worked as expected until Nov 14. Then on Nov 15 the issue appeared again. 

    Since you have multiple custom upstream DNS servers configured, it might be an issue with just one of them, and when that is the one responding to the query, then the issue appears. Unless there are some custom DNS records on these servers that you need, I suggest using one of the more common ones, e.g. Cloudflare or Google, and test with that.

  • Is it possible to export the lets encrypt certificate etc with private key?

  • No, as described above, You should use a Lego or Certbot for this to be usable on other devices. 

    __________________________________________________________________________________________________________________

  • I just got first cert using Sophos Let's Encrypt.

    Disappointed.  Private key needed to make it work on another machine.

    I received a cert for host.ourdomain.com created by XGS/LE and cannot install it without a PK.  WTH?

    Why no private key? or create a wildcard which would work on internal hosts?

    Is the Private key shared between the XGS and all of the LE certs?

    It seems like the community has waited for this functionality for a long time, and partially implemented.

  • I would disagree about your part here: Wildcard as a feature would need a different approach (DNS Challenge). Which would in fact exclude a lot of customers, who do not have access to their DNS vendors via API. Therefore SFOS choose the way like UTM did back in the day (HTTP challenge). 

    SFOS is not the "centralized certificate machine" - Instead we choose to build the system for the own certificate store to work. 

    Customers and primarily home users can easy choose a lego / certbot for their wildcard certificate and reuse it on multiple appliances. 

    I know some UTM customers, generating the certificate on UTM, downloading it via API, uploading it to other appliances etc. At this point (IMHO) it is easier to build a lego/certbot on the API appliance and upload it directly via this server to all appliances than let the UTM do it. 

    Maybe we will make the Private Key available in the future, but again: Do it manually every 60 days is a lot of work all the time, which you could automate quite easily on a own server with a wildcard, if you want. 

    We implemented LE as the UTM did it - Most customers from our feedback loops require LE for Webadmin/Captive Portal/VPN Portal and WAF. That were the primarily main reasons for LE in the first place. 

    Just a few customers generated / use LE on UTM to actually have a certificate server for internal resources - As it has the manual step of downloading it and maintaining it every 60 days. 

    __________________________________________________________________________________________________________________

  • I agree w what you have stated.  I was hoping to avoid having to create a LE server to manage whether its directly exposed to NET or passes through XGS.  You have a point about having to DL / install every 60 days, that would be a pain to propagate to a bunch of machines.

    I didn't realize that a wildcard implied API access to DNS.  I thought an A record for * might work.  - My ignorance.

    Thanks Toni !

  • DNS Challenges need to have access to the DNS Server. So you do not have to expose the Server through SFOS. You can even do both (LE on SFOS for WAF etc, and LE Wildcard via DNS behind it). 

    See: letsencrypt.org/.../

    __________________________________________________________________________________________________________________