Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.

---

Overview

This Recommended Read provides information regarding HTTPS scanning.

This applies to the following Sophos products and versions
Sophos Firewall
Sophos UTM
Sophos web Appliance

 

What is HTTPS?

HTTPS is HTTP messages sent over a TLS/SSL encrypted connection. Before the HTTP messages can be sent, a TLS/SSL connection must be established. This involves a handshake that includes negotiating encryption details such as ciphers and the server sending a certificate to the client. Most discussion about HTTPS decryption concerns the TLS/SSL connection, especially the handshake.

If both the client and server agree on the encryption mechanism, and the client is assured that the server is who it says it’s, then the connection is established. After that, all traffic within that connection is encrypted, often called an encrypted tunnel. If there’s an eavesdropper, they can see the negotiation but can't decode the encrypted traffic.

The only way an eavesdropper can decrypt the traffic is if they interfere with the handshake and insert themselves into the conversation. However, there are cryptographic reasons why they can't do this without changing the handshake in a way that the client will notice.

In a normal situation, HTTPS assures the browser and the user that the web server they connected to is the one that they intended to use and that no one is intercepting the traffic.

What is a certificate and a Certificate Authority?

A certificate is a blob of data that includes the domain hostname, owner, and other things. The certificate is then cryptographically signed by a Certificate Authority, who assures that the information in the certificate is accurate. The signing also acts as a checksum, so if any information in the certificate is modified, then the signing is no longer valid. The client browser looks at the certificate and the Certificate Authority that signed it. If the certificate matches the domain it’s trying to connect to and is signed by a Certificate Authority that it trusts, it completes the TLS/SSL negotiation. After that, the rest of the connection is encrypted, and the client sends the HTTP request.

A Certificate Authority (CA) is a term that can mean two different things. It can mean a special type of certificate that has the ability to sign other certificates. A Certificate Authority can also refer to a type of company that controls a Certificate Authority (a type of certificate) that can sign certificates for other companies. Many CA companies have their CA certificates trusted by default in all browsers (often called a Trusted Root CA or a Public Root CA). This way, an organization can go to a public CA and have them create a certificate that is signed by a CA that is automatically trusted by all browsers.

In your browser, if you go to https://www.example.com, you can examine the certificate and see that the Certificate Authority has signed it. Your browser has pre-installed it so that it trusts all certificates signed by Digicert, so it trusts that www.example.com is who it says it’s. If the browser tries going to that website and receives a certificate signed by someone it does not trust, it’ll warn the user. If the site also uses something called HSTS (HTTP Strict Transport Security), the warning will be one that you can't bypass.

The firewall comes installed with a certificate that was generated from an untrusted Certificate Authority and does not even cover its hostname (because that is set after installation). This certificate is used whenever a client goes to a page that is hosted on the firewall itself (where the firewall's hostname or IP appears in the address bar). Most administrators will want to regenerate or install a certificate rather than use the one that comes straight from the installation. You can purchase a certificate from a Certificate Authority that your browser already trusts. This certificate is used for several things, regardless of whether you’re doing HTTPS decrypting and scanning.

The web proxy part of the firewall also has its own Certificate Authority that it uses when doing man-in-the-middle HTTPS decrypt and scan. When a browser then goes to https://www.example.com, if you look at the certificate, you’ll see that the Sophos CA signed it. Since the browser does not automatically trust the Sophos CA, it’ll show a warning.

It’s important to understand the difference between the firewall certificate and certificates created on the fly from the web proxy's Certificate Authority. The firewall certificate can be purchased so no user sees a warning. The certificates generated from the Certificate Authority will present warnings to users unless the firewall's Certificate Authority is installed (trusted) in the browser.

Certificate Authorities can be a "root CA" (top level, not signed by anyone else) or an "intermediate" / "leaf" CA. The latter can sign certificates but is itself signed by a root CA. Sometimes, the action of creating and signing a certificate is called issuing a certificate. Therefore, it’s said that a certificate was issued from a specific CA. Some browsers will also say "Verified By, " meaning a specific CA signed the certificate.

What is HTTPS Decrypt and Scan?

A man-in-the-middle is when an eavesdropper pretends to be the web server (to the client) and then pretends to be the client when it passes the information up to the real web server. When you turn on HTTPS decrypt and scan, the web proxy will start doing man-in-the-middle decryption of HTTPS traffic.

A TLS/SSL session is established between the web server and the web proxy, and a second TLS/SSL session is established between the web proxy and the client browser. The web proxy will use the information from the real certificate of the server to create a new certificate on the fly and then sign it using its own Certificate Authority. This new certificate is passed to the client browser in the TLS/SSL handshake. It’s important to note that the web proxy can't use the firewall's certificate because the hostname would not match the site the browser is trying to connect to. It can't re-use the actual certificate from the real web server, because it would be unable to decrypt the traffic. It must create a new certificate that it can use for decryption, and then it must have that certificate signed.

If the client browser accepts the certificate and completes the TLS/SSL handshake, it’ll send an HTTP request over the encrypted connection to what it thinks is the web server, but what, in reality, is the web proxy. The proxy impersonates the website, decrypts the data, scans it, and applies any policy. It then re-encrypts the data to send to the web server (now pretending it’s the browser). The same happens in the response.

If the client browser sees a certificate that looks correct but is signed by a Certificate Authority that it does not trust, it won’t complete the TLS/SSL handshake. The users will get a warning because of the security design of browsers when doing HTTPS. HTTPS is designed and implemented so that no one can eavesdrop on the conversation without the user knowing.

Browsers can have additional specific trusted website certificates that only prevent warnings for those websites. Browsers can have additional Certificate Authorities trusted on them, preventing warnings due to the certificate being signed by an untrusted CA. Therefore, a network administrator can install (trust) the web proxy's Certificate Authority on each computer/browser; when the web proxy creates a certificate and signs it with its CA, the browser trusts that CA and will establish the connection.

When HTTPS decrypt and scan aren’t turned on, and a client browser makes the first HTTPS request to a website, the web browser will start negotiating a TLS/SSL connection with SNI information that has the website's domain name. For example, the browser may request www.google.com/search, but the proxy will only see a TLS/SSL connection with SNI for www.google.com. Or it may be requesting innocentsite.com/.../malware.exe, and the proxy only sees innocentsite.com. After the TLS/SSL connection is fully established and there’s an encrypted tunnel, the proxy can't see the HTTP requests and responses within it. Therefore, it can only perform blocks based on the domain category in the SNI while the connection is still being established. It can't categorize or block based on the path or virus scan of any downloaded files. It does know the total bytes transmitted and the connection time.

What are some security concerns for HTTPS scanning?

HTTPS decryption means that the web proxy can now see inside the encrypted HTTPS traffic. Anyone with login access to the firewall could see that traffic as well. If you don’t have strong passwords on your firewall, allow SSH access, or leave your firewall insecure, then HTTPS scanning makes the clients less secure. However, if your firewall is insecure, a potential attacker has much richer targets than HTTPS traffic.

HTTPS decryption means that the web proxy will see and log the paths within an HTTPS session. This does not affect security, but it may affect privacy. For example, an administrator can see what their users search for in Google, or what URLs they’re going to at their bank.

Aside from the firewall's security, turning on HTTPS Scanning does not make you or the clients any less secure.

Does AV scanning on the web proxy interfere with AV scanning on the client?

If you have AV scanning on your endpoint, scanning the Sophos Firewall is fine. There’s no problem with scanning twice (aside from it taking longer), and in fact, you can specifically turn on two independent AV scanners within the Sophos Firewall and UTM web proxy. Some people like to have Single Scan with the Avira engine on the web proxy and then Sophos AV on the endpoint, with the concept that two different scan vendors are better. People with endpoint AV software from other vendors prefer to use the Sophos AV on the web proxy. Some jurisdictions and regulations require dual scanning.

Unless you have a tightly controlled network where you administer every device, there’s no guarantee that every device has a virus scanner. Therefore, virus scanning in the web proxy at the firewall is a good thing.

Some applications, websites, and devices have problems with virus scanning or situations where protections that are part of virus scanning interfere with traffic. An example would be web requests for partial files. These can't be correctly virus scanned and are therefore blocked (in UTM and Sophos Firewall proxy mode). If you have specific applications or sites that make web requests for partial files, you need an exception that turns off the virus scanner for that specific traffic.

An administrator can create web exceptions that prevent virus scanning for certain sources or destinations. They can also turn off virus scanning in HTTPS traffic by turning off HTTPS decryption for specific traffic using web exceptions (Sophos Firewall and UTM), web profiles (UTM), firewall rules (Sophos Firewall), or SSL/TLS inspection rules (Sophos Firewall). They can also cause certain web traffic not to go through the web proxy at all (in UTM, this is the transparent mode skip list; in Sophos Firewall, is it a higher-level firewall rule).

 

What are the impacts of turning on HTTPS Decrypt and Scan?

Here are some things to consider when trying to decide whether to use HTTPS Decrypt and Scan:

Some computers and devices will have endpoint software installed, which may also be able to provide protection, but HTTPS and virus scanning at the firewall is the only way to make sure that all traffic is scanned for all devices.

However, there’s a balance of things to consider:

The web proxy is used for two main purposes: Enforcement of organization policy (no visiting of Adult sites) and security (virus scanning of all traffic) and sometimes a mix (no downloading of any application). The purpose you are using the proxy and how strongly you need that enforcement will influence your decision. If the most important thing is policies based on category, you might be able to go without HTTPS scanning. If the most important thing is security, then you’re more likely to need it.

You can have some network segments with HTTPS scanning and others without. For example, you can have wired computers do HTTPS scanning, a wireless network with corporate access for corporate devices with HTTPS scanning, and a guest wireless network that does not have HTTPS scanning. You can also have different network segments have different web policies and access to different categories of websites. The Sophos Firewall in proxy mode configures this with multiple firewall rules. In Sophos Firewall DPI mode with SSL/TLS inspection rules, while in the UTM, this is configured with multiple Web Filtering Profiles.

If every computer is connected to Active Directory, then AD can be used to push certificates and Certificate Authorities to those computers, making deployment easy.

If every mobile device has corporate control software, you can push certificates and Certificate Authorities to those devices, making deployment easy.

If you have BYOD devices that aren’t corporately managed, then it’s hard to put a Certificate Authority on the device. That being said, they’re less likely to have antivirus software on them and, therefore, are more likely to download or already contain malware. Regardless of the HTTPS Decrypt and Scan, you should probably not have them on the same network segment as the rest of your corporate network or give them access to your corporate network. It’s better to put them on a separate guest network.

Any mobile device connecting to the cell network can bypass any web proxy controls you have in place by connecting via mobile data. However, because they’re not on the Wi-Fi connection and corporate LAN, they’re less likely to be a security risk to your network, only to themselves.

Every administrator must balance their need for insight into their user's actions, blocking traffic that does not conform to policy and protecting malware against the additional administration work of deploying a CA to every device.

Another consideration is that more websites worldwide use HTTPS, and there are only a growing number of HTTPS sites. Every year, the percentage of web traffic that is encrypted goes up. In the long term, not performing HTTPS decryption will mean less traffic will be scanned.

Finally, HTTPS decryption does require more CPU resources. The maximum web throughput will lower with HTTPS decryption. However, this highly depends on your hardware, the number of users, the features turned on, and many other factors.

Can I purchase a Certificate Authority that allows Decrypt and Scan without needing to deploy anything to clients?

No, you can't. HTTPS is designed so that you can't. Consider the implications if this wasn’t true. If you took your phone to a coffee shop and logged on to their free Wi-Fi to do banking, would you like the coffee shop to be able to see all your bank account information? Because your phone is communicating with the bank's website using HTTPS, and because the coffee shop can't spy on the traffic without you getting a warning, you know that communication is secure. A coffee shop (or your organization) can't use a Sophos web proxy to silently decrypt all HTTPS traffic without the user allowing it, either by having them install a CA or go through browser warnings.

You must use the CA that comes with the firewall or create your own CA certificate, which can be a self-signed root. Your Microsoft Active Directory also has its own Certificate Authority (see Active Directory Certificate Services), where you can create an intermediate/leaf CA that you can use within the firewall.

If it were possible to purchase a signing CA that a public root CA ultimately authenticates, it would break the whole trust model on which SSL encryption is based. The trust model depends on the organizations that control the trusted root certificates being selective and applying strict criteria about when they allow a certificate to be signed. For example, they should only sign a certificate for google.com if the person requesting the certificate can prove that they control that domain.

But if they issued a certificate to a third party that could be used to sign whatever certificate that third party likes, they would basically be delegating the ability to create and sign certificates for ANY domain that would be trusted automatically by ANY browser. It would allow the purchaser of that certificate to set up their website and pretend to be google.com, facebook.com, wellsfargo.com, or anyone they like, and browsers would accept it.

There was a genuine situation where a French government CA, which was widely trusted by most web browsers, issued a signing CA to another government department for use in an HTTP web filtering product. The dangerous thing about this was that this web filtering product was now effectively minting new certificates left, right, and center for all sorts of websites that would have been trusted by ANY browser run by anyone in the world. 

You can, however, buy a certificate for a specific hostname or all hostnames within a domain that a trusted CA signs. Purchasing a certificate and using it on the firewall will stop any warnings of users who are accessing WebAdmin, User Portal, Captive Portal, or other pages displayed from the firewall itself. But purchased certificates can't be used as a Certificate Authority for HTTPS scanning.

I am not doing HTTPS scanning and do not have the certificate or Certificate Authority deployed, why do users sometimes get warnings about certificates?

If you’re blocking a category such as Gambling and a user goes to https://www.pokerstars.com/, then you want that web request to be blocked. The web proxy sees that the client is trying to go somewhere they should not and wants to display a block page. To do so, they need to do man-in-the-middle decryption so that it can insert a block page that pretends to be www.pokerstars.com.

In Sophos Firewall and later, under Web > General Settings, you can configure that HTTPS connections that are blocked when scanning is off should instead just be dropped. In that case, if a user went to www.pokerstars.com, they would get a failed connection rather than a block page.

If you enforce authentication and an unauthenticated user is browsing, they must be redirected to AD SSO or Captive Portal to authenticate. If the site they’re trying to get to is HTTPS, then to do the redirection, it may need to do man-in-the-middle decryption using the Certificate Authority. The user is then sent to the Captive Portal, which is displayed in HTTPS using the firewall's certificate.

To be clear, you may get a certificate generated by the man-in-the-middle HTTPS scanning to perform redirection. You may also get a certificate from the firewall itself.

If you’re using Sandstorm and the firewall needs time to analyze the file, then it displays a message to the user and allows the user to download the file a few minutes later. Those pages use the certificate from the firewall.

What is the recommended way to reduce/remove warnings for pages served by the firewall?

All pages that are served by the firewall (where the browser's address bar has yourfirewall.yourcompany/ or the IP) use the certificate of the firewall. This includes WebAdmin, User Portal, Captive Portal (Sophos Firewall), and Sandstorm.

1. You can go to one of many Certificate Authority companies and purchase a certificate for your firewall. Your organization or for *.yourcompany. Many companies will already have a purchased certificate that they already use on their other servers. One way to purchase a certificate is to generate a CSR (Certificate Signing Request), which is a certificate that isn’t signed. You then send that CSR to a public CA. Public Certificate Authorities must ensure that the CSR information is correct and that you own the domain you are requesting before they sign it.
2. If you already have an internal Certificate Authority (often from Active Directory Certificate Service), you can use that to create and sign a certificate.
3. If you’re doing HTTPS Decryption and deploying the Certificate Authority to all clients, you can use that CA to issue a certificate.

By default, in Sophos Firewall, when the web proxy redirects a user to the firewall, it uses the IP of the firewall, which a certificate can't cover. To redirect to the hostname instead, see the following:

Sophos Firewall: Use hostname for page redirects

In Sophos Firewall 18.0 and later, the hostname used in redirection settings can be found at Administration > Admin user Settings > Admin console and end-user interaction.

It’s possible to configure the captive portal to be displayed in HTTP rather than HTTPS and, therefore, not require a certificate. However, this isn’t recommended because passwords would be sent across the network unencrypted.

What is the recommended way to reduce/remove warnings for pages served by the proxy?

Most customers will use the Certificate Authority that comes with the system by default.

Some companies may already have a Certificate Authority that they’re using that has generated certificates for their internal computers and deployed them on their clients. One example would be from Active Directory Certificate Services. Another example would be from a pre-existing web proxy. If you have a CA that you want to use, you can install it on the firewall and use it within the web proxy. Rather than using your root Certificate Authority, you may instead want to create an intermediate Certificate Authority and use that. You’ll still need to install the root Certificate Authority (without a private key) so that the web proxy can build the complete certificate chain.

From WebAdmin, you can download a copy of the Certificate Authority that you can deploy to all clients using Active Directory. Please note that Windows/internet Explorer and Chrome use the same Certificate Authority store. However, Firefox has its separate store.

If you have multiple firewalls, you may want to create a Certificate Authority separate from a firewall, and then install and use it on all your firewalls.

If you want to use a certificate on the firewall itself that is generated from the Certificate Authority, it’s easier to do this by starting with a Certificate Authority that is separate from the firewall and then installing it. 

Sophos Mobile Control and some other ways of managing corporate mobile devices allow you to push the CA to all managed devices.

 

Related information

• Sophos Firewall: Does the Web Filter block HTTPS based websites when HTTPS Scanning is disabled?
• Sophos Firewall: How to install a subordinate certificate authority (CA) for HTTPS inspection
• Sophos Firewall: How to configure HTTPS inspection using a privately signed CA
• Sophos Firewall: SSL CA certificate installation guide
• Sophos Firewall: How to bypass web categories from HTTPS Scanning
• Sophos Firewall: How to Bypass HTTPS Scanning for a Web Category
• Sophos Firewall: How to Enforce Safe Search

Removed outdated link (Naked Security), edited Grammar
[edited by: Raphael Alganes at 11:06 AM (GMT -7) on 15 Oct 2024]