Sophos Firewall v18: XStream - the new DPI Engine for web proxy explained

Question

Disclaimer : This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment. 
 
 Xstream Architecture Sophos is pleased to introduce the new Xstream Architecture for Sophos Firewall, a new streaming packet processing architecture that provides extreme levels of protection and performance. The new architecture includes: 
 1) Xstream SSL Inspection: Organizations can enable SSL inspection on their networks without compromising network performance or user experience. It delivers high-performance, high connection capacity support for TLS 1.3 and all modern cipher suites providing extreme SSL inspection performance across all ports, protocols, and applications. It also comes equipped with enterprise-grade controls to optimize security, privacy, and performance. 
 2) Xstream DPI Engine: Enables comprehensive threat protection in a single high-performance streaming DPI engine with proxyless scanning of all traffic for AV, IPS, and web threats as well as providing Application Control and SSL Inspection. Pattern matching on decrypted traffic makes patterns more effective and provides increased protection from hash/pattern changing applications such as Psiphon proxy. 
 3) Xstream Network Flow FastPath: Provides the ultimate in performance by intelligently offloading traffic processing to transfer trusted traffic at wire speeds. FastPath offloading can be controlled through policy to accelerate important cloud application traffic, or intelligently by the DPI engine based on traffic characteristics. 
 
 So what does this mean? 
 One of the new features that is v18.0 is a new high performance way of handling web traffic, along with new high performance way of doing SSL/TLS decryption, and a lot of new options around enforcement of TLS/SSL rules. The web proxy from 17.5 is still present, and administrators have a choice which mode they want to use. 
 The following is an attempt to summarize the differences between the "proxy mode" and the new "DPI mode" (Deep Packet Inspection). Basically to explain 2) and the relevant parts of 1). But the overall feature is more than what I am covering. 
 It focuses on differences in web for the things you could do in 17.5, and do differently in 18.0.

Mode 
 Using the "web proxy" 
 Using the "DPI Engine"

Summary 
 Underlying process name 
 awarrenhttp 
 snort

How does it work 
 Terminates the connect from the client browser. Creates a new connection to the web server. Copies data from one connection to the other. Delays traffic while it processes access control. Delays traffic if using Batch AV scanning mode (default). Can modify headers, changes destination IP, etc. If blocking needed, displays block page. 
 Allows connection straight from client browser to the web server. Inspects the traffic from client to server. Does little to modify, delay, or interrupt traffic. Performs AV scanning in-line (like Real Time AV scanning mode). If blocking needed and response is already started to sent it kills the connection. If blocking needed and response is not sent it redirects browser to :8090 to display block page.

Benefits 
 More functionality. Better block pages. 
 Better Performance. More port and TLS inspection options.

Configuration 
 Creating a firewall rule for web traffic 
 Firewall Rule: Set the services to HTTP and HTTPS (port 80 and 443). Select a Web Policy Select Scan HTTP and decrypted HTTPS 
 No difference.

How to choose the web traffic proxy mode 
 Firewall Rule: Check "Use web proxy instead of DPI engine" 
 Firewall Rule: Uncheck "Use web proxy instead of DPI engine"

How to configure HTTPS decryption 
 Firewall Rule: Check "Decrypt HTTPS during web proxy filtering" 
 Go to Rules and policies > SSL/TLS inspection rules and create an inspection rule Set the services to HTTP and HTTPS (port 80 and 443). Set the Action to Decrypt. Select a Decryption Profile.

How to exclude a site from HTTPS decryption 
 Go to Web > Exceptions and create and exception. Set URL pattern match to be website FQDN. This is RegEx, see existing entries for proper syntax. Select skip HTTPS decryption. 
 Create a web exception (as in web proxy mode) -OR- Go to URL Groups and edit the Local TLS exclusion list. Add the domain name. This is plaintext not RegEx. Make sure the default SSL/TLS inspection rule "Exclusions by website" is enabled. -OR- Create an SSL/TLS inspection rule that is set to action "Do not decrypt". Create your own URL group or custom category and use it in the rule.

How to exclude a source from HTTPS decryption 
 Go to Web > Exceptions and create and exception. Set Source IP or IP range. Select skip HTTPS decryption. 
 Create a web exception (as in web proxy mode) -OR- Create an SSL/TLS inspection rule that is set to action "Do not decrypt". Base it on source IP or IP range or on user.

Port and TLS support 
 Transparent Mode Ports 
 Port 80/443 
 Any port

Standard/Explicit/Direct Mode 
 Supported (Default port 3128) 
 Not supported. Will direct to proxy.

Enforce TLS and certificate restrictions for port 80/443 traffic 
 Basic on/off flags that covers common settings: Web > General Settings > Block unrecognized SSL protocols Web > General Settings > Block invalid certificates 
 Detailed control: Profiles > Decryption Profiles Rules and policies > SSL/TLS inspection rules > SSL/TLS inspection settings

Enforce TLS and certificate restrictions for other ports 
 Not supported 
 Supported

Feature Support 
 HTTPS signing Certificate Authority 
 You can select a single CA that will be used for all traffic. 
 You can select different CAs for signing traffic where the web server is using an RSA or EC certificate. You can select different CAs based on source or destination.

AV scanning mode 
 Batch (holds entire file while it scans, displays block page if there is a virus) Real Time (delivers file as it arrives, holds last bit of file while it scans, kills connection if there is a virus) 
 Real Time (delivers file as it arrives, holds last bit of file while it scans, kills connection if there is a virus). If downloaded a second time soon after, redirect to a block page.

Pharming Protection 
 Supported 
 Not supported

Caching 
 Supported 
 Not supported

Enforce SafeSearch 
 Supported 
 Not supported

Enforce YouTube restrictions 
 Supported 
 Not supported

Restrict login domains for Google apps 
 Supported 
 Not supported

Michael Dunn · Accepted Answer

A brief step by step guide: 
 I would start by doing things incrementally, to build out knowledge and experience. By far the biggest and most common usage will be in HTTP/HTTPS so start there. Most customers will already be using the web proxy, this assumes that you have some knowledge. 
 Next is to determine if you are going to decrypt traffic or not. The cost and the reasons are the same as they are with the web proxy. Decrypting traffic means that you will need to deploy the appliance CA to all your devices. Regardless of whether you choose to do so in the long term, incrementally I would start by not decrypting. Make sure everything is stable without decryption before enabling. 
 Apply the workaround listed about so that TLS 1.3 traffic is downgraded to 1.2. 
 So create a a firewall rule with appropriate source and destination, service is HTTP and HTTPS. Choose a web policy, scan for malware, don't "use the web proxy". On the SSL/TLS inspection rules, just have the top two rules active. Don't create (or enable) others. 
 Run this for a few hours. You should see logs in the web filter log. You can test your policy works by going to HTTP and HTTPS sites that should be blocked due to category. If you do not have the CA installed, when you go to an HTTPS site that is blocked, you will first get a browser warning message. You should see in the firewall rules your new rule bandwidth goes up. 
 If everything is working, you can turn on HTTPS decryption. First deploy the CA to your devices. 
 Now you can create a SSL/TLS Inspection Rule. Use the same source and destination as your HTTPS firewall rule, HTTPS service only. Set action to decrypt. Set the profile to maximum compatibility. 
 Run like that for a while. See what works and what does not. If there is one particular device that is not working, check the CA on it. If there is one particular application that is not working, check the logs and see if there is anything interesting. Note what domains it is using. Some applications use their own certificate stores. 
 The 'just get it working' approach to HTTPS sites that don't work is to add them to the TLS exclusion list. Go to Web \ URL Groups and edit "Local TLS exclusion list" to add the domains. Then test again. Now when we (developers of this feature) find stuff that does not work, rather than adding to the exclusion list, we investigate to see why it is not working and if there is anything that the product could do differently.

Michael Dunn · Answer

There is no DPI mode box to uncheck. 
 If you have a web policy selected and/or malware scanning selected and you do not have "use web proxy instead of DPI" checked then it will use the DPI mode. 
 If you do not have a web policy and also do not have malware scanning then some options are disabled. Neither proxy nor DPI mode will be looking at the web traffic. If the Services for the firewall rule include port 80/443 then the traffic flows through the firewall as an open port. The HTTPS traffic might still be decrypted depending on the settings in the SSL/TLS inspection rules tab, but DPI will not be doing any enforcement.

Michael Dunn · Answer

[deleted] - v18.0 EAP specific