Slow Throughput after installing v18 EAP

Hi,

I upgraded from v17.5.8 to v18 EAP about a week ago and noticed a drop in performance and an increased RAM usage.

I do have a XG115 rev2 Appliance installed with the Software  Image and a Home Use License.

My Internet connection is 100/40.

With version 17.5.8 I was able to reach about 80 to 90 Mbit download (I already expected more from the hardware)

After the Upgrade I only reach about 50 to 60 Mbit download. There is no DPI or webfiltering activated and it doesn't matter if i activate IPS or not.

SSL/TLS Inspection is turned on but there are not any rules.

Are there any tweaking options for the software version of Sophos XG running on a HW Appliance?

Thank you!

Parents
  • As posted in the initial Announcement: 

    https://community.sophos.com/products/xg-firewall/sfos-eap/sfos-v18-early-access-program/b/blog/posts/sophos-xg-firewall-v18-fire-eap-firmware-is-here

    • The firmware has yet to be tuned for performance. Expect to see faster speeds in future builds.

     

    Do you use a hardware Bridge? 

    Do you use IPS?

    Do you use SSLx (even one rule with "Do not Decrypt")? 

    __________________________________________________________________________________________________________________

  • Hello,

    Is there any news on this question?

    I’ve been using v18 EAP 1 since launch, and the performance difference between v17.5.8 and v18 is wierd. The v18 has supposed to be faster, but it’s slower.

     

    I’m currently with Intel J1900 + 8GB ram with Intel 82576 NIC.

    I’ve made a clean installation, and used IPS GeneralPolicy, ATP (Log and Drop), Default Policy for Web and no HTTPS Decrypt for the testing.

    v17.5.8, I would be able to get 260mbit/s which is my WAN download limit, while using less than 45% of CPU usage. With HTTPS Decrypt on, i still has able to get 260mbit/s.

    v18, i can barely get 120mbit/s, that’s without TLS/SSL Inspection or HTTPS Decrypt via Web Proxy. If i use HTTPS Decrypt via Web Proxy, i would get the same speeds on any HTML5 speedtest. With TLS/SSL Inspection the throughput would get even lower to 80mbit/s.

    Here’s how it looks like with top on v18. Snort is always using 100% of the CPU.

     

    Is there anything that i can do to archive better speeds. Or it’s an issue in my end?

     

    Thanks,


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  • Interesting thread. Someone from Sophos dev team should investigate on that. It is still a beta and they still adjusting “the code” but your case seems to be interesting.

  • I saw this issue earlier. As far as i know, its already fixed in the next version.

     ? 

    __________________________________________________________________________________________________________________

  • Hi folks,

    my throughput is the same, but response times have blown out and ram usage is up which I assume is caused by all the debugging code left active.

    Ian

    XG115W - v20.0.2 MR-2 - Home

    XG on VM 8 - v21 GA

    If a post solves your question please use the 'Verify Answer' button.

  • I think ram usage is going to stay about the same due to additional DPI engine. They are not supporting appliances less that 4GB. I even tried installing a vm with less that 4GB and the installation fails. Even after installation, if you try to decrease the ram in a vm, the firewall goes to safe mode if the ram gets below 3600MB.

    I also found that the load average is considerably higher compared to v17.xx Compare you old load average graphs and there is definitely an uptick.

  • Hi Billyboy,

    RAM increase is fine, but going from low 40% to low 70% is a rather large increase on a 6gb system.

    Also as you point ou the load has increased from just over 2 to just under 4.

    So we wait and see what the next version brings?

    Ian

    XG115W - v20.0.2 MR-2 - Home

    XG on VM 8 - v21 GA

    If a post solves your question please use the 'Verify Answer' button.

  • Could you retest V18 EAP1 Refresh1? 

    __________________________________________________________________________________________________________________

  • Hi,

    I've retest it on V18 EAP1 Refresh1, throughput still the same as v18, which is way slower than v17.5.8. I've decided to go back to v17.5.8 just to do some testing again, because i through could be something wrong with my hardware, but the throughput difference still high.

    Here's an CPU Usage graph from v17.5.8, while using all my WAN download throughput limit, 240-260 mbit/s. With IPS on generalpolicy.

     

    Here's on v18 EAP 1 Refresh 1. You can see the CPU spike at 8:30, that's when i decide to try download centos 8, throughput limit at it has 130mbit/s

     

     

    I didn't take any pictures, but for fun i've decide to make a VM of v18 EAP1, with 4 cores/6GB ram(Host => Ryzen 1700, 32GB ram), while on v17.5.8 I could get gigabit speeds on it (Using VirtiO Drivers i could get aroung 1.2 - 1.4 Gbit/s), on v18 the max speed i would get has aroung 480-510 mbit/s on iperf3.

     

    Thanks,


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  • Could you please verify the used drivers in V18? 

    Check please via ethtool of each interface. 

    __________________________________________________________________________________________________________________

Reply Children
  • SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool -i Port2
    driver: igb
    version: 5.3.5.20
    firmware-version: 1.2.1
    expansion-rom-version:
    bus-info: 0000:01:00.1
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: no


    SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool -i Port1
    driver: igb
    version: 5.3.5.20
    firmware-version: 1.2.1
    expansion-rom-version:
    bus-info: 0000:01:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: no


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  • Can you please show us the output of ethtool without -i ?

    __________________________________________________________________________________________________________________

  • SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool Port1
    Settings for Port1:
            Supported ports: [ TP ]
            Supported link modes:   10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Full
            Supported pause frame use: Symmetric
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Full
            Advertised pause frame use: Symmetric
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: 1000Mb/s
            Duplex: Full
            Port: Twisted Pair
            PHYAD: 1
            Transceiver: internal
            Auto-negotiation: on
            MDI-X: off (auto)
            Supports Wake-on: pumbg
            Wake-on: g
            Current message level: 0x00000007 (7)
                                   drv probe link
            Link detected: yes

     

    SFVH_SO01_SFOS 18.0.0 EAP1-Refresh1# ethtool Port2
    Settings for Port2:
            Supported ports: [ TP ]
            Supported link modes:   10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Full
            Supported pause frame use: Symmetric
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Full
            Advertised pause frame use: Symmetric
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: 1000Mb/s
            Duplex: Full
            Port: Twisted Pair
            PHYAD: 1
            Transceiver: internal
            Auto-negotiation: on
            MDI-X: off (auto)
            Supports Wake-on: d
            Wake-on: d
            Current message level: 0x00000007 (7)
                                   drv probe link
            Link detected: yes


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  • I too had issues with EAP1 on a hardware XG135 with the software build.  I did just update to EAP1-Refresh1 and there is marked improvement in reliability and speed for me.

    With DPI on, no slow down.  Getting close to 200 down and 40 up.  Same as what I was getting on 17.5.8.  More importantly, the connection doesn't drop/reset every minute.

    I haven't switched on SSL inspection yet.  Will do that after watching how this does for the morning.

  • Hi,

     

    Do we have any news on this?

    Is this an issue with only a small percentage of people, or an miss-configuration issue from everyone that's having performance issues?

    Or it's a known issue on v18 EAP 1 ?

     

     

    Thanks,


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  • Could you try to change the adapters, just to be sure and reboot? 

    Which hypervisor do you use? 

    __________________________________________________________________________________________________________________

  • LuCar Toni said:

    Could you try to change the adapters, just to be sure and reboot? 

    Which hypervisor do you use? 

     

     

    Hi,

     

    I'm currently running v18 on bare-metal, with a Intel J1900 + 8GB RAM, with Intel 82576 2 Ports NIC.

    On bare metal I has able to get >240mbit/s  while not even using more than 45% of the CPU on v17.5.8, with v18 EAP 1 Refresh 1 the maximum i get is around 120-130mbit/s, while snort is using 100% of all 4 Cores, something that never happened on v17.5.8.

     

    The VM I've used to test the performance difference has on CentOS with KVM, using virt-manager to manage it. I've tested using VirtiO drivers and e1000e. Both gave me same throughput on the tests on v18.

     

    Thanks,


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home

  •  Could you take a look at this setup?

    __________________________________________________________________________________________________________________

  • Hello All. I did some quick and dirty tests at home this evening to check that behaviour on my XG125. While LuCar Toni wrote it correctly, DPI is still under development and there's surely space upwards until GA to get some bottlenecks optimized to reach different speeds than today via DPI.

    However I assume many people are testing their web speeds (http/https), which at least in case of HTTPS is not fully offloadable to FastPath due en-, and decryption in the DPI engine. However other connections as for example a copying a large file with a cient in client network from a network share in the server network via XG firewall should get offloaded and reach linespeeds.

    I tested this today, and it seems, that offloadable connections gets marked correctly in conntrack as offloaded, but speed is still capped to ~peak throughput the DPI engine offers on that device (in my case ~60Mbit with the XG125). So I'm not sure, if in EAP1 or at least EAP1 Refresh 1 VFP offloading probably accidentially got broken, or if it maybe got disabled for whatever reason in that release(s).

    However:

    With firewall-acceleration (offloading) disabled it get in speedtest.net ~60 MBit/s downstream, with firewall-acceleration enabled too. That was expected, as the connection is only partly offloaded but still passing through DPI due HTTPS de-, and encryption.

    With firewall-acceleration (offloading) disabled it get for my filecopy from a SMB share via firewall 9MByte/s, with firewall-acceleration enabled too. That was unexpected, as the connection is marked as offloaded in conntrack, and should reach linespeed (or at least ~80-90MByte/s, which my NAS should be capable to deliver.)

    So it looks for me, that the VFP offloading process in that release probably isn't working as expected.

     

    Some conntrack examples with firewall-acceleration enabled and disabled for the SMB connection to the network share

    system firewall-acceleration enable

    XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

    proto=tcp      proto-no=6 timeout=10800 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=49794 orig-dport=445 packets=85888 bytes=3923517 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=49794 packets=147599 bytes=220884992 [ASSURED] mark=0x0 use=3 id=1733477632 masterid=0 devin=Port1.53 devout=Port3 nseid=404 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0xd004804000 flagvalues=3,21,25,27,43,78,87,90,100,102,103 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892018 microflowid[0]=78 microflowrev[0]=37 microflowid[1]=241 microflowrev[1]=54 hostrev[0]=4 hostrev[1]=4 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=0 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2546 current_state[1]=2546 vlan_id=0 inmark=0x0 brinindex=0 sessionid=1477 sessionidrev=9123 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=18446612138342663424 conn_fp_rev=0

    system firewall-acceleration disable

    XG125_XN02_SFOS 18.0.0 EAP1-Refresh1# conntrack -L | grep -i orig-dport=445

    proto=tcp      proto-no=6 timeout=10786 state=ESTABLISHED orig-src=192.168.20.215 orig-dst=192.168.10.11 orig-sport=50039 orig-dport=445 packets=145 bytes=24482 reply-src=192.168.10.11 reply-dst=192.168.20.215 reply-sport=445 reply-dport=50039 packets=130 bytes=49211 [ASSURED] mark=0x0 use=1 id=1536525888 masterid=0 devin=Port1.53 devout=Port3 nseid=2 ips=0 sslvpnid=0 webfltid=0 appfltid=1 icapid=0 policytype=2 fwid=15 natid=0 fw_action=1 bwid=0 appid=10203 appcatid=0 hbappid=0 hbappcatid=0 dpioffload=0x3 inzone=1 outzone=1 devinindex=22 devoutindex=7 hb_src=8 hb_dst=0 flags0=0x8000a200008 flags1=0x1004800000 flagvalues=3,21,25,27,43,87,90,100 catid=0 user=78 luserid=38 usergp=65 hotspotuserid=0 hotspotid=0 dst_mac=00:1a:8c:48:75:d8 src_mac=30:24:32:ee:e5:58 startstamp=1572892496 microflow[0]=INVALID microflow[1]=INVALID hostrev[0]=0 hostrev[1]=0 ipspid=0 diffserv=0 loindex=7 tlsruleid=0 ips_nfqueue=1 sess_verdict=0 gwoff=0 cluster_node=0 current_state[0]=2548 current_state[1]=2548 vlan_id=0 inmark=0x0 brinindex=0 sessionid=439 sessionidrev=43149 session_update_rev=5 dnat_done=0 upclass=0:0 dnclass=0:0 pbrid_dir0=0 pbrid_dir1=0 nhop_id[0]=15 nhop_id[1]=6 nhop_rev[0]=0 nhop_rev[1]=0 conn_fp_id=NOT_OFFLOADED

  • Hi, Thank you so much for this answer.

     

    First, if I'm wrong then please correct me.

    I also understand, as the name says, It's an Early Access Program, It can have bugs and performance issues.

     

    But, The problem I've encountered it's a bit different, I'm not using any IPS/Web Proxy/SSL/TLS Decrypt and somehow snort is using 100% of all my 4 cores.

    To start with it:

    I've created an Rule which allows to pass the Traffic between LAN to LAN, and on this rule I've used .

    Also there's no TLS/SSL Inspection rules being applied on it.

     

    Testing with Iperf3, From 10.0.0.200 => 10.0.1.11 (VLAN 20), I've been getting the maximum throughput of 430Mbit/s with "iperf3 -c 10.0.1.11 -P 5" While Snort is using 100% of all my cores.

     

    My question is, what is Snort doing? I've disabled all features on the Rule, I've checked to see if there's any other rule influencing on it, also there's no SSL/TSL Inspection being used on it, and still my throughput is limited by it.

     

    The problem is, on v17.5.8 I've used to get line-rate throughput with this test.

     

    Thanks,


    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v21 GA @ Home

    Sophos ZTNA (KVM) @ Home