Slow Throughput after installing v18 EAP

As posted in the initial Announcement: 

https://community.sophos.com/products/xg-firewall/sfos-eap/sfos-v18-early-access-program/b/blog/posts/sophos-xg-firewall-v18-fire-eap-firmware-is-here

• The firmware has yet to be tuned for performance. Expect to see faster speeds in future builds.

Do you use a hardware Bridge? 

Do you use IPS?

Do you use SSLx (even one rule with "Do not Decrypt")? 

Hello,

Is there any news on this question?

I’ve been using v18 EAP 1 since launch, and the performance difference between v17.5.8 and v18 is wierd. The v18 has supposed to be faster, but it’s slower.

I’m currently with Intel J1900 + 8GB ram with Intel 82576 NIC.

I’ve made a clean installation, and used IPS GeneralPolicy, ATP (Log and Drop), Default Policy for Web and no HTTPS Decrypt for the testing.

v17.5.8, I would be able to get 260mbit/s which is my WAN download limit, while using less than 45% of CPU usage. With HTTPS Decrypt on, i still has able to get 260mbit/s.

v18, i can barely get 120mbit/s, that’s without TLS/SSL Inspection or HTTPS Decrypt via Web Proxy. If i use HTTPS Decrypt via Web Proxy, i would get the same speeds on any HTML5 speedtest. With TLS/SSL Inspection the throughput would get even lower to 80mbit/s.

Here’s how it looks like with top on v18. Snort is always using 100% of the CPU.

Is there anything that i can do to archive better speeds. Or it’s an issue in my end?

Thanks,

Could you please verify the used drivers in V18? 

Check please via ethtool of each interface. 

SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool -i Port2
driver: igb
version: 5.3.5.20
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool -i Port1
driver: igb
version: 5.3.5.20
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Can you please show us the output of ethtool without -i ?

SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool Port1
Settings for Port1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool Port2
Settings for Port2:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

I too had issues with EAP1 on a hardware XG135 with the software build.  I did just update to EAP1-Refresh1 and there is marked improvement in reliability and speed for me.

With DPI on, no slow down.  Getting close to 200 down and 40 up.  Same as what I was getting on 17.5.8.  More importantly, the connection doesn't drop/reset every minute.

I haven't switched on SSL inspection yet.  Will do that after watching how this does for the morning.

Hi,

Do we have any news on this?

Is this an issue with only a small percentage of people, or an miss-configuration issue from everyone that's having performance issues?

Or it's a known issue on v18 EAP 1 ?

Thanks,

Could you try to change the adapters, just to be sure and reboot? 

Which hypervisor do you use? 

LuCar Toni said:

Could you try to change the adapters, just to be sure and reboot? 

Which hypervisor do you use? 

 

Hi,

I'm currently running v18 on bare-metal, with a Intel J1900 + 8GB RAM, with Intel 82576 2 Ports NIC.

On bare metal I has able to get >240mbit/s  while not even using more than 45% of the CPU on v17.5.8, with v18 EAP 1 Refresh 1 the maximum i get is around 120-130mbit/s, while snort is using 100% of all 4 Cores, something that never happened on v17.5.8.

The VM I've used to test the performance difference has on CentOS with KVM, using virt-manager to manage it. I've tested using VirtiO drivers and e1000e. Both gave me same throughput on the tests on v18.

Thanks,

Rana Sharma Could you take a look at this setup?

Hello All. I did some quick and dirty tests at home this evening to check that behaviour on my XG125. While LuCar Toni wrote it correctly, DPI is still under development and there's surely space upwards until GA to get some bottlenecks optimized to reach different speeds than today via DPI.

However I assume many people are testing their web speeds (http/https), which at least in case of HTTPS is not fully offloadable to FastPath due en-, and decryption in the DPI engine. However other connections as for example a copying a large file with a cient in client network from a network share in the server network via XG firewall should get offloaded and reach linespeeds.

I tested this today, and it seems, that offloadable connections gets marked correctly in conntrack as offloaded, but speed is still capped to ~peak throughput the DPI engine offers on that device (in my case ~60Mbit with the XG125). So I'm not sure, if in EAP1 or at least EAP1 Refresh 1 VFP offloading probably accidentially got broken, or if it maybe got disabled for whatever reason in that release(s).

However:

With firewall-acceleration (offloading) disabled it get in speedtest.net ~60 MBit/s downstream, with firewall-acceleration enabled too. That was expected, as the connection is only partly offloaded but still passing through DPI due HTTPS de-, and encryption.

With firewall-acceleration (offloading) disabled it get for my filecopy from a SMB share via firewall 9MByte/s, with firewall-acceleration enabled too. That was unexpected, as the connection is marked as offloaded in conntrack, and should reach linespeed (or at least ~80-90MByte/s, which my NAS should be capable to deliver.)

So it looks for me, that the VFP offloading process in that release probably isn't working as expected.

Some conntrack examples with firewall-acceleration enabled and disabled for the SMB connection to the network share

system firewall-acceleration enable

XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

proto=tcp      proto-no=6 timeout=10800 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=49794 orig-dport=445 packets=85888 bytes=3923517 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=49794 packets=147599 bytes=220884992 [ASSURED] mark=0x0 use=3 id=1733477632 masterid=0 devin=Port1.53 devout=Port3 nseid=404 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0xd004804000 flagvalues=3,21,25,27,43,78,87,90,100,102,103 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892018 microflowid[0]=78 microflowrev[0]=37 microflowid[1]=241 microflowrev[1]=54 hostrev[0]=4 hostrev[1]=4 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=0 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2546 current_state[1]=2546 vlan_id=0 inmark=0x0 brinindex=0 sessionid=1477 sessionidrev=9123 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=18446612138342663424 conn_fp_rev=0

system firewall-acceleration disable

XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

proto=tcp      proto-no=6 timeout=10786 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=50039 orig-dport=445 packets=145 bytes=24482 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=50039 packets=130 bytes=49211 [ASSURED] mark=0x0 use=1 id=1536525888 masterid=0 devin=Port1.53 devout=Port3 nseid=2 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0x1004800000 flagvalues=3,21,25,27,43,87,90,100 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892496 microflow[0]=INVALID microflow[1]=INVALID hostrev[0]=0 hostrev[1]=0 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=1 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2548 current_state[1]=2548 vlan_id=0 inmark=0x0 brinindex=0 sessionid=439 sessionidrev=43149 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=NOT_OFFLOADED

Hello All. I did some quick and dirty tests at home this evening to check that behaviour on my XG125. While LuCar Toni wrote it correctly, DPI is still under development and there's surely space upwards until GA to get some bottlenecks optimized to reach different speeds than today via DPI.

However I assume many people are testing their web speeds (http/https), which at least in case of HTTPS is not fully offloadable to FastPath due en-, and decryption in the DPI engine. However other connections as for example a copying a large file with a cient in client network from a network share in the server network via XG firewall should get offloaded and reach linespeeds.

I tested this today, and it seems, that offloadable connections gets marked correctly in conntrack as offloaded, but speed is still capped to ~peak throughput the DPI engine offers on that device (in my case ~60Mbit with the XG125). So I'm not sure, if in EAP1 or at least EAP1 Refresh 1 VFP offloading probably accidentially got broken, or if it maybe got disabled for whatever reason in that release(s).

However:

With firewall-acceleration (offloading) disabled it get in speedtest.net ~60 MBit/s downstream, with firewall-acceleration enabled too. That was expected, as the connection is only partly offloaded but still passing through DPI due HTTPS de-, and encryption.

With firewall-acceleration (offloading) disabled it get for my filecopy from a SMB share via firewall 9MByte/s, with firewall-acceleration enabled too. That was unexpected, as the connection is marked as offloaded in conntrack, and should reach linespeed (or at least ~80-90MByte/s, which my NAS should be capable to deliver.)

So it looks for me, that the VFP offloading process in that release probably isn't working as expected.

Some conntrack examples with firewall-acceleration enabled and disabled for the SMB connection to the network share

system firewall-acceleration enable

XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

proto=tcp      proto-no=6 timeout=10800 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=49794 orig-dport=445 packets=85888 bytes=3923517 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=49794 packets=147599 bytes=220884992 [ASSURED] mark=0x0 use=3 id=1733477632 masterid=0 devin=Port1.53 devout=Port3 nseid=404 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0xd004804000 flagvalues=3,21,25,27,43,78,87,90,100,102,103 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892018 microflowid[0]=78 microflowrev[0]=37 microflowid[1]=241 microflowrev[1]=54 hostrev[0]=4 hostrev[1]=4 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=0 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2546 current_state[1]=2546 vlan_id=0 inmark=0x0 brinindex=0 sessionid=1477 sessionidrev=9123 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=18446612138342663424 conn_fp_rev=0

system firewall-acceleration disable

XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

proto=tcp      proto-no=6 timeout=10786 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=50039 orig-dport=445 packets=145 bytes=24482 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=50039 packets=130 bytes=49211 [ASSURED] mark=0x0 use=1 id=1536525888 masterid=0 devin=Port1.53 devout=Port3 nseid=2 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0x1004800000 flagvalues=3,21,25,27,43,87,90,100 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892496 microflow[0]=INVALID microflow[1]=INVALID hostrev[0]=0 hostrev[1]=0 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=1 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2548 current_state[1]=2548 vlan_id=0 inmark=0x0 brinindex=0 sessionid=439 sessionidrev=43149 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=NOT_OFFLOADED

Hi, Thank you so much for this answer.

First, if I'm wrong then please correct me.

I also understand, as the name says, It's an Early Access Program, It can have bugs and performance issues.

But, The problem I've encountered it's a bit different, I'm not using any IPS/Web Proxy/SSL/TLS Decrypt and somehow snort is using 100% of all my 4 cores.

To start with it:

I've created an Rule which allows to pass the Traffic between LAN to LAN, and on this rule I've used.

Also there's no TLS/SSL Inspection rules being applied on it.

Testing with Iperf3, From 10.0.0.200 => 10.0.1.11 (VLAN 20), I've been getting the maximum throughput of 430Mbit/s with "iperf3 -c 10.0.1.11 -P 5" While Snort is using 100% of all my cores.

My question is, what is Snort doing? I've disabled all features on the Rule, I've checked to see if there's any other rule influencing on it, also there's no SSL/TSL Inspection being used on it, and still my throughput is limited by it.

The problem is, on v17.5.8 I've used to get line-rate throughput with this test.

Thanks,

What is the output of

system application_classification show

in your device console? If it's on, you might test setting it to off using same command but replace show by off...

Not sure about the implication of disabling global app classification in V18, as this command bypasses snort, if no explicit IPS or application control rule is configured in the matching firewall rule. This was a good workaround to get linespeeds up to 17.5.  However - I didn't test if this also bypasses the new fast path offloading capability of V18 (will test if I find some time), but as FastPath offloading in my eyes seems not to behave as intended at this specific release, this might help temporarily speed up things until Fast Path behaves as expected.

As you mentioned - it's early access..there is time until GA to bring things in shape, so I'm not too worried about in that early stage.

That is not fair sascha. You are asking him to disable application classification which breaks other things such as qos etc while he is clearly stating that he didn't have such problems with v17.5xx 

Probably turning off the IPS services will do the same thing as turning off classification and it will also break your dashboard that tells you the classification of different websites and other reports. Might as well run a simple iptables router at that point. Wire speed achieved.

Regards

Edit: I see you edited your post considerably. We do agree that it is early so things can only improve from here hopefully. But I think over reliance on snort on a UTM type device is never a good thing since what they are asking snort to do usually needs dedicated appliances due to heavy cpu/ram requirements.

Hi,

It showed as ON, after turning off i has able to archive full LAN gigabit on it, but...

The test I've made isn’t fully accurate since I don't have the knowledge nor the equipment to do it correctly.

But it gives an perspective on the performance difference.

Since I think there’s something wrong with v18 performance, I’ve decided to create two VM, one with v17.5.8, and another with v18 EAP 1 Refresh 1.

- Both VM’s had 4 Cores and 8GB RAM (6GB Usable)

- KVM, with virt-manager has used.

- Host OS: CentOS.

- Host: Ryzen 1700 / 32GB RAM.

- Fedora 31 as the LAN VM, for the testing. With 4 Cores 8GB RAM.

- VirtiO Driver has been used on all VM’s.

An outlook on how it has been run:

HOST - WAN - XG - Isolated/LAN – VM/FEDORA

---

Edit: Redone some tests in a better environment, also added pictures. Also used Iperf3.

---

v18 EAP 1 Refresh 1,

IPS – GeneralPolicy – SingleThread: 320 Mbit/s

IPS – GeneralPolicy – MultiThread: 1.28 Gbit/s

----------

v17.5.8,

IPS – GeneralPolicy – SingleThread: 926 Mbit/s

IPS – GeneralPolicy – MultiThread: 2.78 Gbit/s

I’m impressed on how much IPS single core speed has been penalized from v17.5.8 to v18, I’ve tested multiple times, but still the difference is too high.

And as I said before this is just an simple testing, to give an perspective on the performance difference.

Thanks,

Could you show us your Network configuration on this appliance? 

LuCar Toni 

Prism shared all the information to investigate. You question is not very helpful!

Very insightful, Thanks.

I too noticed slowness with v18.

I hoped this week-end I find time to revert back to v.17.5.8.

Because it has become a "trap" trying to keep both version, I will not try to upgrade an inactive v18 firmware anymore.

What frustrate me utterly, is that I have lost my v17.5.8 absolutely for nothing because v18 EAP refresh 1 is a correction for a single insignificant bug.

Again, that waste of time could have been avoided with professional communications from Sophos.

Paul Jr

That is some extensive testing and it takes a lot of time to do tests like these. You shouldn't have to show sophos hard proof, they should already have these numbers. While the rest of us are going by simple feel of our internet connection and then perform simple speed tests, you are taking testing to the next level. BRAVO and well done!

On a side note, I did revive my vm and turned off IPS completely and XG is fairly livable without IPS. Ofcourse none of the graphs work in application categories and all the other app detection is deactivated but for a simple web-filtering firewall/av, it is fairly snappy compared to old v17 I used about a year ago.

Regards