Sophos XG 18 MR3 DPI slow download

This is just the way it is.  Also you get the bonus of it breaking random sites with no idea why or what to do about it.

are there any new ideas on this topic?

I don't think that this is not happening to lot's of other people if this would be a bug?

These are interesting facts:

- Linux and Windows differ in behaviour
- Multiple parallel connections max out the connection
- CPU is not fully loaded (I'd expect that a single connection will always go on a single thread/core). In bigger installation this should not be an issue as there will be plenty of connections.


There are still other parameters that might have an impact:

- fastpath on/off
- avanced-firewall tcp settings (https://docs.sophos.com/nsg/sophos-firewall/18.0/Help/en-us/webhelp/cli/PDF/sfos_cliguide.pdf)
- SSL/TLS decryption

If the proxy is on there are probably two connections. One going from the client to the sophos and one from sophos to the website.

When DPI is used I'd assume that the connection is more "direct". As the behavior of linux and windows is different with respect to throughput I'd think that there might be some "live translation" between client <-> sophos and sophos <-> website which fits in case of a linux client and does not fit in case of newer Windows versions.  Analysing such a connection low level in wireshark might help to find the reason for this different behaviour (window size, window scaling, selective acknoledgement, mtu & mms, fragmentation, ...).

I have seen some bad throughput of Windows Servers through WAN lines with high latency (SMB - never tested with HTTP(S)) so this might be an issue of Sophos firewall OR Windows Server. Multiple connections also helped in this case to increase throughput off the underlying connection.

Besides some basic testing (and probably using the old proxy afterwards) I won't participate in this. Simply not my task to invest time here. As a lot of people seem to be able to reproduce this it is really surprising that this is lingering around in this forum for months (or even longer) and still seems not to be deeply analysed, troubleshooted and resoved by Sophos.

I did some more tesing:

direct download https://speed.hetnert.de ISO 1 GB

Win 7 Chrome Cable modem drect 70MB/s

Win 7 Chrome xg125 Web Proxy  32 MB/s

Win 7 Chrome xg125 dpi 2,5 MB/s (i read about snort being single core but this is really slow maybe the xg125 is at max cpu?)

Then i sat up an Ubuntu 20 VM and after fiddling with disabling ipv6 (no cert warning or appliance cert usage):
Firefox dpi 10 MB/s (I guess firefox uses several sessions for download?)

Firefox Webproxy 23 MB/s

If i have some more time i will test multiple sessions

Gunter Kaufmann thanks for testing this at your end.

Very strange that everyone seems to get different results while doing the tests.

If Firefox would use several sessions for a download this should also give good results on Windows 10 which it does not do at my end.

Currently I have no clue what I could test or change on top of the things I already did. Perhaps it is a bug or an issue I can't resolve, but as long as there is no feedback from Sophos I will stay at the legacy proxy configuration which works very well.

I recently updated my Home UTM to Sophos XG Home firewall, I had a few fun and games with the UEFI boot problem but when I had circumnavigated around that I was pleased with the look and feel of the new interface (despite there not being a SET STATIC option from the lease list in DHCP).  All appeared fine as I played around with the features and settings for a few days until I realised my download speeds via a windows client seemed severely limited from what I had before, I have a 200Mbps download link but via a windows client this was reported as 9Mbps, if I try the same speed test from a MAC computer on the same network it was revealing a 199Mbps download speed.  I tried switching of various features and disabling dpi and using traditional web proxy but could not get download speeds to match the MAC or what I have in reality, the actual system was not heavily loaded.  

I have now reverted back to my old UTM restore and download speeds are as they should be with 40GB downloads finishing in minutes again instead of the XG network predicted 10 hours.

This does appear to be an issue with the new XG which I am supposed to be rolling out in my company in a few weeks but am now hesitant if this issue carries over to the business environment.   

Could an engineer from Sophos confirm this is a known issue and when there is likely to be a fix or if anyone else has found a workaround could they let me know.

I took some time looking into this concept and found some interesting points. 

Of course IPS, App control and web filtering will cause some decrease of performance. 

I could found some limitation of single connection downloads (single stream). Using a speedtest, everything looks normal (speedtest.net). Using test downloads, which offers a download file, seems to point to this issue. It will slowly increase over time. 

So if i download for example a 1 GB test file (https://speed.hetzner.de/) which are encrypted, i can start with a slower speed but it increase over time to a acceptable speed. 

Also by starting multiple connection from this test page, i can quickly match the overall acceptable speed. 

It is important to understand about the DPI: The client actually communicate with the server. There is no redirect to any proxy what so ever (compared to proxy, which in direct/transparent, will redirect it to the proxy and the proxy build up a new connection).

Just to confirm: If you see different values on different OS, it could lead to a different handling of the flow state between the server/client. 

Another question, do you see any kind of increase over the time of a bigger download? 

And please: For the future discussions, please be specific in your numbers. There is MB/s and there is mbit/s. To mix this up is unhealthy to get a common sense of the actual throughput numbers. 

No long 40GB downloads did not appear to speed up, I left them for over two hours and they did not increase in speed HOWEVER when I reverted to UTM and tried again the downloads started and finished in minutes.  To my mind there is certainly an issue with the XG firewall, even with IPS and every other feature switched off the speed did not improve.

edited my original message to be more specific on numbers  :-)

We need to be careful. UTM does not have anything like DPI in XG. Its another technology, another approach. The proxy will work differently. Lets see what the other post. 

I understand DPI in XG is another technology but surely it should not bottleneck your internet bandwidth so severely, I understand it is a technology that is supposed to process packets faster not drop bandwidth from 200Mbps to 9Mbps.  I have three windows machines, a Macbook Pro and a number of wireless devices, CPU sat around11% so the firewall was not overly busy.  Even with features switched off or web proxy enabled the bottleneck was still present.  The reinstated home UTM is working as well as it has for the past 7 years with no discernible bottlenecks, I had upgraded to XG as a home experiment to learn about the new system and as well because I am looking to upgrade some business UTM's, I was just advertising the fact that there appears to be a problem with it and requesting some help from fellow forum members or a Sophos engineer.  Others have reported similar issues so I am not an isolated case.

General Question to all who tested:

Did you diable ip V6 or do you have it configured on the Sophos?

Since it did rerun some test and had an win10 VM newly created i thouht why not test it to.

After getting some relatively high speedtest results i checked the cert from the site and it was not from my sophos.

And that led me to ip v6, which is configured mostly in my infrastructure but not on the xg125, so my win10 client got an ip v6 lease from my dhcp and talked through/over the xg125 with my dsl modem. sort of bypass...i will check how thats possible or has someone a hint?

So here are some more Numbers, MB=Megabyte, kbit=kilobit

Testsites: www.wieistmeineip.de , speed.hetzner.de 10GB File for 2-3 Minutes

W7 + W10 current Chrome

Ubuntu current Fifrefox

DPI

W7  Speedtest  down 155.157 kbit  up fails   Download 10GB 3 MB/s (slows down, when the Ubuntu VM starts the same file)

Win10 Speedtest  down 160.960 kbit  up 6112 kbit   Download 10GB 2,5 MB/s 

Ubuntu Speedtest  down 248.445 kbit  up 1575 kbit   Download 10GB 10 MB/s (slows down, when the Ubuntu VM starts the same file)

Webproxy

W7  Speedtest  down 742.214 kbit  up 63.936 kbit   Download 10GB 30 MB/s 

Win10 Speedtest  down 460.061 kbit  up 64.005 kbit   Download 10GB 26 MB/s 

Ubuntu Speedtest  down 916.345 kbit  up 31.275 kbit   Download 10GB 30 MB/s 

Cpu Load was around 40% in the gui, top -d1 showed around 45% snort (dpi) org 40-50% awarrenhttp (proxy)

My test results, with Windows 10 Clients (VM) 

Provider vodafon business (1000mbits Download/ 50mbits Upload) 

Router Fritzbox 6591 (exposed host) --> Sophos XG 210 (Static IP) 

No ip V6 in Sophos and no ip V6 in Network LAN 

DPI is still very slow or the download crash after a time 

WebProx there is no problem 

CPU Load of the Sophos is max. at 25% with DPI and Multi-Downloads, at WebProxy the CPu goes to round about 50%. 

Why does Sophos only use one core for a download at DPI, that is actually an exclusion criterion for the DPI. We have to load a lot

of large files from different servers. With the web proxy you could choose the maximum size. Is this also used for DPI?

My test results, with Windows 10 Clients (VM) 

Provider vodafon business (1000mbits Download/ 50mbits Upload) 

Router Fritzbox 6591 (exposed host) --> Sophos XG 210 (Static IP) 

No ip V6 in Sophos and no ip V6 in Network LAN 

DPI is still very slow or the download crash after a time 

WebProx there is no problem 

CPU Load of the Sophos is max. at 25% with DPI and Multi-Downloads, at WebProxy the CPu goes to round about 50%. 

Why does Sophos only use one core for a download at DPI, that is actually an exclusion criterion for the DPI. We have to load a lot

of large files from different servers. With the web proxy you could choose the maximum size. Is this also used for DPI?

What happen, if you wait some more time after starting the download in DPI Engine? 

As already written, the download fails after a while. 

Dome said:

Why does Sophos only use one core for a download at DPI

The DPI Engine is based on Snort 2.9.16 which is a single core IDPS (Sophos partially solves this by spawning multiple process and sharing the load between them, but only works with multiple connections).

Snort 3.1 should resolve this problem since It's fully multi-threaded, but apparently It will take some time until we have It.

What Rev is your XG 210 ? I didn't expected It to be this slow since Sophos told AES-NI is being used on hardware appliances.

I can't say anything about the multi or single connections, because I don't know what is meant :-) However, I assume that a normal download is a single connection and thus represents the standard case. If the standard case of DPI isn't covered, DPI is just bad. We have the Rev.3

The XG 210 Rev.3 uses a Intel Celeron G3900, It's a really weak & old CPU, that's probably one of the reasons on why It's so slow.

Also, just for curiosity, can you SSH in your XG 210, go to advanced shell and give the output of "openssl speed -evp aes-128-cbc" ?

I want to see If It's actually using AES-Ni on the XG 210.

Thanks!

This is the Report of "openssl speed -evp aes-128-cbc" 

XG210_WP03_SFOS 18.0.4 MR-4# openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 20565502 aes-128-cbc's in 2.92s
Doing aes-128-cbc for 3s on 64 size blocks: 6121344 aes-128-cbc's in 2.96s
Doing aes-128-cbc for 3s on 256 size blocks: 1601184 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 404977 aes-128-cbc's in 2.96s
Doing aes-128-cbc for 3s on 8192 size blocks: 50164 aes-128-cbc's in 2.94s
OpenSSL 1.0.2u-fips  20 Dec 2019
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
compiler: ccache_cc -m32 -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/srv/jenkins/workspace/OmC/CI_64/staging_dir/target-x86_64_glibc/usr/include -I/srv/jenkins/workspace/OmC/CI_64/staging_dir/toolchain-x86_64_gcc-7.3.0_glibc/usr/include -I/srv/jenkins/workspace/OmC/CI_64/staging_dir/toolchain-x86_64_gcc-7.3.0_glibc/include -znow -zrelro -DOPENSSL_NO_HEARTBEATS -DTERMIOS -fpic -Wa,--noexecstack -O3 -fomit-frame-pointer -Wall -fomit-frame-pointer -Wall -I/srv/jenkins/workspace/OmC/CI_64/staging_dir/target-x86_64_glibc/usr/lib/fips-i386/include
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     112687.68k   132353.38k   136634.37k   140100.15k   139776.70k

but according to this list "since Sophos told AES-NI is being used on hardware appliances". AES-Ni is supported with the XG210_WP03. The Sophos is not even 1 year old

No, openssl isn't using AES-NI on your XG 210, those numbers are too low for the G3900.

I have no idea, it is in this list (see marking). But I don't care either, Sophos advertises the DPI. With which version should the DPI work without any problems? The XG210 Rev. 3 is the current version that is available for purchase.

I have an XG210 Rev2 and my numbers running that openssl test match your Rev3 almost exactly.  Not sure it means anything, just throwing it out there.

Hey Prism,

only the high end Sophos hardware supports AES-NI, XG660 and above according to the specifications.

Ian