Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.

Overview

This Community Guide provides information regarding HTTPS scanning.

The following sections are covered:

• What is HTTPS?
• What is a certificate and a Certificate Authority?
• What is HTTPS Decrypt and Scan?
• What are some security concerns for HTTPS scanning?
• Does AV scanning on the web proxy interfere with AV scanning on the client?
• What are the impacts of turning on HTTPS Decrypt and Scan?
• Can I purchase a Certificate Authority that allows Decrypt and Scan without needing to deploy anything to clients?
• I am not doing HTTPS scanning and do not have the certificate or Certificate Authority deployed, why do users sometimes get warnings about certificates?
• What is the recommended way to reduce/remove warnings for pages served by the firewall?
• What is the recommended way to reduce/remove warnings for pages served by the proxy?
• Related information

Applies to the following Sophos products and versions
Sophos Firewall
Sophos UTM
Sophos Web Appliance

What is HTTPS?

HTTPS is HTTP messages sent over a TLS/SSL encrypted connection. Before the HTTP messages can be sent, a TLS/SSL connection must be established. This involves a handshake that includes negotiating encryption details such as ciphers and the server sending a certificate to the client. Most discussion about HTTPS decryption is centered around the TLS/SSL connection, especially the handshake.

If both the client and server agree on the encryption mechanism, and the client is assured that the server is who it says it is, then the connection is established. After that, all traffic within that connection is encrypted, often called an encrypted tunnel. If there is an eavesdropper they would be able to see the negotiation but be unable to decode the encrypted traffic.

The only way that an eavesdropper can decrypt the traffic is if they interfered with the handshake and inserted themselves into the conversation. However, there are cryptographic reasons why they cannot do this without changing the handshake in a way that the client will notice.

In a normal situation, HTTPS provides an assurance to the browser and the user that the web server they connected to is the one that they intended to and that no one is intercepting the traffic.

What is a certificate and a Certificate Authority?

A certificate is a blob of data that includes the domain hostname, owner, and other things. The certificate is then cryptographically signed by a Certificate Authority, who assures that the information in the certificate is accurate. The signing also acts a checksum so if any information in the certificate is modified then the signing is no longer valid. The client browser looks at the certificate and the Certificate Authority that signed it. If the certificate matches the domain it is trying to connect to, and the certificate is signed by a Certificate Authority that it trusts, it completes the TLS/SSL negotiation. After that, the rest of the connection is encrypted and the client sends the HTTP request.

A Certificate Authority (CA) is a term that can mean two different things. It can mean a special type of certificate that has the ability to sign other certificates. A Certificate Authority can also refer to a type of company who controls a Certificate Authority (a type of certificate) which can sign certificates for other companies. There are many CA companies that have their CA certificates trusted by default in all browsers (often called a Trusted Root CA or a Public Root CA). This way a company can go to a public CA and have them create a certificate that is signed by a CA that is automatically trusted by all browsers.

In your browser, if you go to https://www.example.com, you can examine the certificate and see that it was signed by the Certificate Authority digicert. Your browser has pre-installed it that it trusts all certificates signed by digicert, so it trusts that www.example.com is who it says it is. If the browser tried going to that website and received a certificate that was signed from someone which it does not trust, it will warn the user. If the site also uses something called HSTS (HTTP Strict Transport Security), the warning will be one that you cannot bypass.

The firewall comes installed with a certificate that was generated from an untrusted Certificate Authority and does not even cover its own hostname (because that is set after installation). This certificate is used whenever a client goes to a page that is hosted on the firewall itself (where the firewall's hostname or IP appears in the address bar). Most administrators will want to regenerate or install a certificate rather than use the one that comes straight from the installation. You can purchase a certificate from a Certificate Authority that your browser already trusts. This certificate is used for several things, regardless of whether you are doing HTTPS decrypt and scan.

The web proxy part of the firewall also has its own Certificate Authority that it uses when doing man-in-the-middle HTTPS decrypt and scan. When a browser then goes to https://www.example.com, if you look at the certificate, you will see that it was signed by the Sophos CA. Since the browser does not automatically trust the Sophos CA, it will display a warning.

It is important to understand the difference between the certificate of the firewall and certificates created on-the-fly from the web proxy's Certificate Authority. The certificate of the firewall can be purchased so that no user ever sees a warning. The certificates that are generated from the Certificate Authority will present warnings to users unless the firewall's Certificate Authority is installed (trusted) in the browser.

Certificate Authorities can be a "root CA" (top level, not signed by anyone else) or an "intermediate" / "leaf" CA. The latter has the ability to sign certificates but is itself signed by a root CA. Sometimes the action of creating and signing a certificate is called issuing a certificate. Therefore it is said that a certificate was issued from a particular CA. Some browsers also will say "Verified By" to mean the certificate was signed by a specific CA.

What is HTTPS Decrypt and Scan?

A man-in-the-middle is when an eavesdropper pretends to be the web server (to the client) and then pretends to be the client when it passes the information up to the real web server. When you turn on HTTPS decrypt and scan, the web proxy will start doing man-in-the-middle decryption of HTTPS traffic.

A TLS/SSL session is established between the web server and the web proxy, and a second TLS/SSL session is established between the web proxy and the client browser. The web proxy will use the information from the real certificate of the server to create a new certificate on-the-fly, and then sign it using its own Certificate Authority. This new certificate is passed to the client browser in the TLS/SSL handshake. It is important to note that the web proxy cannot use the certificate of the firewall itself because the hostname would not match the site the browser is trying to connect to. It cannot re-use the actual certificate from the real web server, because it would be unable to decrypt the traffic. It must create a new certificate which it can use for decryption and then it must have that certificate signed.

If the client browser accepts the certificate and completes the TLS/SSL handshake, it will send an HTTP request over the encrypted connection to what it thinks is the web server, but what in reality is the web proxy. The proxy impersonates the website, decrypts the data, scans it and applies any policy. It then re-encrypts the data to send it to the web server (now pretending that it is the browser). The same happens in the response.

If the client browser sees a certificate that looks correct but is signed by a Certificate Authority that it does not trust, it will not complete the TLS/SSL handshake. Because of the security design of browsers when doing HTTPS, the users will get a warning. HTTPS is designed and implemented so that no one can eavesdrop on the conversation without the user knowing.

Browsers can have additional specific trusted website certificates that prevent warnings for those websites only. Browsers can have additional Certificate Authorities trusted on them, which prevent warnings that are due to the certificate being signed by an untrusted CA. Therefore a network administrator can install (trust) the web proxy's Certificate Authority on each computer/browser. Now when the web proxy creates a certificate and signs it with its CA, the browser trusts that CA and will establish the connection.

When HTTPS decrypt and scan is not enabled and a client browser makes the first HTTPS request to a website, the web browser will start negotiating a TLS/SSL connection with SNI information that has the domain name of the website. For example, the browser may be requesting www.google.com/search but the proxy will only see a TLS/SSL connection with SNI for www.google.com. Or it may be requesting innocentsite.com/.../malware.exe and the proxy only sees innocentsite.com. After the TLS/SSL connection is fully established and there is an encrypted tunnel, the proxy cannot see the HTTP requests and responses within it. Therefore it can only perform blocks based on the category of the domain in the SNI while the connection is still being established. It cannot do categorization or blocks based on the path and it cannot virus scan any files that get downloaded. It does know the total bytes transmitted and the connection time.

What are some security concerns for HTTPS scanning?

HTTPS decryption means that the web proxy can now see inside the encrypted HTTPS traffic. Anyone who has login access to the firewall could potentially see that traffic as well. If you do not have strong passwords on your firewall, if you allow ssh access, if you leave your firewall insecure, then HTTPS scanning makes the clients less secure. However, if your firewall is insecure, a potential attacker has much richer targets than HTTPS traffic.

HTTPS decryption means that the web proxy will see and log the paths within an HTTPS session. This does not affect security but it may affect privacy. For example, an administrator will be able to see what their users are searching for in Google, or what URLs they are going to at their bank.

Aside from the security of the firewall itself, turning on HTTPS Scanning does not make you or the clients any less secure.

Does AV scanning on the web proxy interfere with AV scanning on the client?

If you have AV scanning on your endpoint, it is perfectly fine to also scan on the XG. There is no problem with scanning twice (aside from it taking longer), and in fact, you can specifically turn on two independent AV scanners within the XG and UTM web proxy. Some people like to have Single Scan with the Avira engine on the web proxy and then Sophos AV on the endpoint, with the concept that two different scan vendors are better. Other people who have endpoint AV software from other vendors prefer to use the Sophos AV on the web proxy. Some jurisdictions and regulations require dual scanning.

Unless you have a tightly controlled network where you administer every device, there is no guarantee that every device has a virus scanner. Therefore virus scanning in the web proxy at the firewall is a good thing.

There are some applications, websites, and devices that have problems with virus scanning, or situations where protections that are part of virus scanning interfere with traffic. An example would be web requests for partial files. These cannot be correctly virus scanned, and are therefore blocked (in UTM and XG proxy mode). If you have specific applications or sites that do web requests for partial files, you need an exception that turns off the virus scanner for that specific traffic.

An administrator can put in web exceptions that prevent virus scanning for certain sources or destinations. They can also disable virus scanning in HTTPS traffic by turning off HTTPS decryption for specific traffic using web exceptions (XG and UTM), web profiles (UTM), firewall rules (XG), or SSL/TLS inspection rules (XG). They can also cause certain web traffic not to go through the web proxy at all (in UTM this is the transparent mode skiplist, in XG is it a higher level firewall rule).

What are the impacts of turning on HTTPS Decrypt and Scan?

Here are some things to consider when trying to decide whether to use HTTPS Decrypt and Scan:

Some computers and devices will have endpoint software installed which may also be able to provide protection, but HTTPS and virus scanning at the firewall is the only way to make sure that all traffic is scanned for all devices.

However, there is a balance of things to consider:

The web proxy is used for two main purposes: Enforcement of company policy (no visiting of Adult sites) and security (virus scanning of all traffic) and sometimes a mix (no downloading of any executable). The purpose you are using the proxy and how strong you need that enforcement will influence your decision. If the most important thing is policies based on category, you might be able to go without HTTPS scanning. If the most important thing is security then you are more likely to need it.

You can have some network segments that have HTTPS scanning and others that do not. For example, you can have wired computers do HTTPS scanning, a wireless network with corporate access for corporate devices that has HTTPS scanning, and a guest wireless network that does not have HTTPS scanning. You can also have different network segments have different web policies and access to different categories of websites. In the XG Firewall in proxy mode, this is configured with multiple firewall rules, in XG Firewall in DPI mode with SSL/TLS inspection rules, while in the UTM this is configured with multiple Web Filtering Profiles.

If every computer is connected to Active Directory, then AD can be used to push certificates and Certificate Authorities to those computers, making deployment easy.

If every mobile device has corporate control software on it, then you can push certificates and Certificate Authorities to those devices, making deployment easy.

If you have BYOD devices that are not corporately managed then it is hard to put a Certificate Authority on the device. That being said, they are less likely to have antivirus software on them and therefore are more likely to download or already contain malware. Regardless of the HTTPS Decrypt and Scan, you should probably not have them on the same network segment as the rest of your corporate network or give them access to your corporate network. It is better to put them on a separate guest network.

Any mobile device that can connect to the cell network can bypass any web proxy controls you have in place by connecting via mobile data. However, because they are not on the wifi connection and corporate LAN, they are less likely to be a security risk to your network, only to themselves.

Every administrator needs to balance their need for insight into their user's actions, blocking of traffic that does not conform to policy, and protection of malware against the additional administration work of deploying a CA to every device.

Another consideration is that more and more websites around the world are using HTTPS, and there are a growing number of sites that are HTTPS only. Every year the percentage of web traffic that is encrypted goes up. In the long term, not performing HTTPS decryption will mean less and less of the traffic will be scanned.

Finally, HTTPS decryption does require more CPU resources. The maximum web throughput will lower with HTTPS decryption. However, this is highly dependent on your hardware, the number of users, features that are enabled and many other factors.

Can I purchase a Certificate Authority that allows Decrypt and Scan without needing to deploy anything to clients?

No, you cannot. HTTPS is designed so that you cannot. Consider the implications if this was not true. If you took your phone to a coffee shop and logged on to their free wifi to do banking, would you like the coffee shop to be able to see all your bank account information? Because your phone is communicating with the bank's website using HTTPS, and because the coffee shop cannot spy on the traffic without you getting a warning, you know that communication is secure. A coffee shop (or your company) cannot use a Sophos web proxy to silently decrypt all HTTPS traffic without the user allowing it, either by having them install a CA or go through browser warnings.

You must use the CA that comes with the firewall or create your own CA certificate, which can be a self-signed root. Your Microsoft Active Directory also has its own Certificate Authority (see Active Directory Certificate Services) where you can create an intermediate/leaf CA that you can use within the firewall.

If it were possible to purchase a signing CA that is ultimately authenticated by a public root CA, it would break the whole trust model on which SSL encryption is based. The trust model depends on the organizations who control the trusted root certificates being selective and applying strict criteria about when they allow a certificate to be signed. For example, they should only sign a certificate for google.com if the person requesting the certificate can prove that they control that domain.

But if they issued a certificate to a third party that could be used to sign whatever certificate that third party likes, they would basically be delegating the ability to create and sign certificates for ANY domain that would be trusted automatically by ANY browser. It would allow the purchaser of that certificate to set up their own website and pretend to be google.com, or facebook.com, or wellsfargo.com or anyone they like, and browsers would just accept it.

There was a genuine situation where a French government CA, which was widely trusted by most web browsers, issued a signing CA to another government department for use in an HTTP web filtering product. The dangerous thing about this was that this web filtering product was now effectively minting new certificates left, right and center for all sorts of websites that would have been trusted by ANY browser run by anyone in the world. See this article about the incident: https://nakedsecurity.sophos.com/2013/12/09/serious-security-google-finds-fake-but-trusted-ssl-certificates-for-its-domains-made-in-france/.

You can however buy a certificate for a specific hostname or all hostnames within a domain that is signed by a trusted CA. Purchasing a certificate and using it on the firewall will stop any warnings of users who are accessing WebAdmin, User Portal, Captive Portal, or other pages displayed from the firewall itself. But purchased certificates cannot be used as a Certificate Authority for HTTPS scanning.

I am not doing HTTPS scanning and do not have the certificate or Certificate Authority deployed, why do users sometimes get warnings about certificates?

If you are blocking a category such as Gambling and a user goes to https://www.pokerstars.com/ then you want that web request to be blocked. The web proxy sees that the client is trying to go somewhere they should not and wants to display a block page. In order to do so, they need to do man-in-the-middle decryption so that it can insert a block page that pretends to be www.pokerstars.com.

In XG 17.5 and later, under Web > General Settings, you can configure that HTTPS connections that are blocked when scanning is off should instead just be dropped. In that case, if a user went to www.pokerstars.com, they would just get a failed connection rather than a block page.

If you are enforcing authentication and an unauthenticated user is browsing, they need to be redirected to either do AD SSO or Captive Portal to authenticate. If the site they are trying to get to is HTTPS, then in order to do the redirection it may need to do man-in-the-middle decryption using the Certificate Authority. If the user is then sent to the Captive Portal, that is displayed in HTTPS using the firewall's own certificate.

To be clear, you may get a certificate generated by the man-in-the-middle HTTPS scanning in order to perform redirection. You may also get a certificate from the firewall itself.

If you are using sandstorm and the firewall needs time to analyze the file, then it displays a message to the user and allows the user to download the file a few minutes later. Those pages use the certificate from the firewall.

What is the recommended way to reduce/remove warnings for pages served by the firewall?

All pages that are served by the firewall (where the browsers address bar has yourfirewall.yourcompany/ or the IP) use the certificate of the firewall. This includes WebAdmin, User Portal, Captive Portal (XG Firewall), and Sandstorm.

1. You can go to one of many Certificate Authority companies and purchase a certificate for yourfirewall.yourcompany or for *.yourcompany. Many companies will already have a purchased certificate that they already use on their other servers. One way to purchase a certificate is to generate a CSR (Certificate Signing Request) which is basically a certificate that is not signed. You then send that CSR to a public CA. Public Certificate Authorities are required to ensure that the information in the CSR is correct and that you own the domain you are requesting for before they sign it.
2. If you already have an internal Certificate Authority (often from Active Directory Certificate Service), you can use that to create and sign a certificate.
3. If you are doing HTTPS Decryption and deploying the Certificate Authority to all clients, you can use that CA to issue a certificate.

By default in XG, when the web proxy redirects a user to the firewall it uses the IP of the firewall which cannot be covered by a certificate. To redirect to the hostname instead, see the following:

Sophos XG Firewall: Use hostname for page redirects

In Sophos Firewall 18.0 and later, the hostname used in redirection settings can be found at Administration > Admin user Settings > Admin console and end-user interaction

It is possible to configure the captive portal to be displayed in HTTP rather than HTTPS and therefore not require a certificate. However, this is not recommended because passwords would be sent across the network unencrypted.

What is the recommended way to reduce/remove warnings for pages served by the proxy?

Most customers will use the Certificate Authority that comes with the system by default.

Some companies may already have a Certificate Authority that they are using that has generated certificates for their internal computers and deployed on their clients. One example would be from Active Directory Certificate Services. Another example would be from a pre-existing web proxy. If you have a CA that you want to use, you can install it on the firewall and use it within the web proxy. Rather than using your root Certificate Authority, you may instead want to create an intermediate Certificate Authority and use that. You will still need to install the root Certificate Authority (without private key) so that the web proxy can build the complete certificate chain.

From WebAdmin, you can download a copy of the Certificate Authority that you can deploy to all clients using Active Directory. Please note that Windows/Internet Explorer and Chrome use the same Certificate Authority store, however, FireFox has its own separate store.

If you have multiple firewalls, you may want to create a Certificate Authority separate from a firewall, and then install and use it on all your firewalls.

If you want to use a certificate on the firewall itself that is generated from the Certificate Authority, it is easier to do this starting with a Certificate Authority that is separate from the firewall and then installing it. 

Sophos Mobile Control and some other ways of managing corporate mobile devices allow you to push the CA to all managed devices.

Related information

• Sophos XG Firewall: Does the Web Filter block HTTPS based websites when HTTPS Scanning is disabled?
• Sophos XG Firewall: How to install a subordinate certificate authority (CA) for HTTPS inspection
• Sophos XG Firewall: How to configure HTTPS inspection using a privately signed CA
• Sophos XG Firewall: SSL CA certificate installation guide
• Sophos XG Firewall: How to bypass web categories from HTTPS Scanning
• Sophos Firewall: How to Bypass HTTPS Scanning for a Web Category
• Sophos XG Firewall: How to Enforce Safe Search