Slow Throughput after installing v18 EAP

As posted in the initial Announcement: 

https://community.sophos.com/products/xg-firewall/sfos-eap/sfos-v18-early-access-program/b/blog/posts/sophos-xg-firewall-v18-fire-eap-firmware-is-here

• The firmware has yet to be tuned for performance. Expect to see faster speeds in future builds.

Do you use a hardware Bridge? 

Do you use IPS?

Do you use SSLx (even one rule with "Do not Decrypt")? 

Hello,

Is there any news on this question?

I’ve been using v18 EAP 1 since launch, and the performance difference between v17.5.8 and v18 is wierd. The v18 has supposed to be faster, but it’s slower.

I’m currently with Intel J1900 + 8GB ram with Intel 82576 NIC.

I’ve made a clean installation, and used IPS GeneralPolicy, ATP (Log and Drop), Default Policy for Web and no HTTPS Decrypt for the testing.

v17.5.8, I would be able to get 260mbit/s which is my WAN download limit, while using less than 45% of CPU usage. With HTTPS Decrypt on, i still has able to get 260mbit/s.

v18, i can barely get 120mbit/s, that’s without TLS/SSL Inspection or HTTPS Decrypt via Web Proxy. If i use HTTPS Decrypt via Web Proxy, i would get the same speeds on any HTML5 speedtest. With TLS/SSL Inspection the throughput would get even lower to 80mbit/s.

Here’s how it looks like with top on v18. Snort is always using 100% of the CPU.

Is there anything that i can do to archive better speeds. Or it’s an issue in my end?

Thanks,

I would strongly disagree with your assessment. Throughput numbers should be the top priority for sophos. Bandwidth is getting cheaper and the requirements are increasing significantly. Its great that behind the scenes you guys know that the performance is not what it should be. What is concerning is that these performance levels would be expected during an alpha stage but not during the second public beta (one release away from GA?). Nobody is publicly acknowledging the problems other than you(thanks for that by the way) and a few comments about it will only get better from here. They did fix the load averages after EAP1 and that is great however the throughput numbers are still the same as EAP1. We are testing our systems mostly with very light loads and still bringing them down to their knees. With regular loads of 100s of employees, the system needs more than minor tuning at this point.

Part of the reason that there are so many threads on performance is due to the fact that everyone has a decent pipe to the internet and most of us utilize that bandwidth even if its just for watching cat videos on youtube. So to have a beta and then completely ignore the throughput numbers and then say that behind the scenes some people are aware and someone is going to tune is not very reassuring.

All the bells and whistles are worthless if I can't saturate my WAN pipe easily even with a modest recent processor.

Thanks again for your comments and this post is in no way directed at you personally.

Regards

Bill

He clearly stated that he expects the performance issues to get worked out before GA. Why do people always feel so entitled around about beta releases? They are beta for a reason. You're making baseless assumptions using a crystal ball and then portray them as a "concerning" reality. Go, watch some more cat videos on Youtube and relax.

[Moderated to remove profanity]

cryptochrome said:

 Go, watch some more cat videos on Youtube and relax.  

I am trying to relax but unfortunately my cpu isn't as chill as I am. Besides, its a public beta and the only time I get entitlement to tell sophos about how I feel. If you work for sophos then sorry, if not then why so defensive?

Regards

Billybob said:

I would strongly disagree with your assessment. Throughput numbers should be the top priority for sophos.

Throughput numbers are a top priority.

My point is that we have internal testbeds with 1GB and 10GB pipes hooked to a performance harness pushing through traffic on a dozen different hardware boxes of different sizes and producing our own detailed analysis with a per process breakdown that our team is acting on.

Getting a report from the forum saying that on their own custom hardware a CPU graph looks bad isn't something that is a useful input to our performance team. It can be useful for different people in the community to talk about, but it is not a report that our developers are going to be using. It can also managers prioritize performance over other existing issues and make sure that they have a minimum threshold before we release.

Don't stop reporting performance issues.  But realize that all the threads on performance go to the forum readers like myself and do not go to the developers. They are using their own test infrastructure to find the same thing but in much actionable detail.

I agree that we should have good throughput numbers in the beta before GA. The problem is like one of cooking a meal and the turkey still needs more time in the oven while the rest of the meal is ready. We decided to go to EAP because we had features ready that we wanted people to try even if the main course was still not ready.

Part of the reason that there are so many threads on performance is due to the fact that everyone has a decent pipe to the internet and most of us utilize that bandwidth even if its just for watching cat videos on youtube. So to have a beta and then completely ignore the throughput numbers and then say that behind the scenes some people are aware and someone is going to tune is not very reassuring.

There is a balance of never replying and appearing to ignore the issue, replying with enough details to let people know we are on it, or replying with huge amounts of data that give too much away about our development process.  Most employees are cautious about replying or don't feel they know the enough of details.  But they haven't locked my account yet so I'm still here to give away secrets.  :)  

I'm trying to be reassuring but more than that I want to set an expectation: When EAP3 comes up the performance numbers will not be good enough to go to GA.

There are significant performance code changes that are not merged into EAP3.  The builds that we are currently testing internally are significantly faster than the EAP3 that you will be testing.  The performance tests you are running on EAP2 is code that is over a month old.  Even when EAP3 comes out the performance tests you will be running are effectively going to be on old code.  There are reasons that the new performance code is not yet integrated, we are waiting for a few more changes and we want to run it internal dogfoods before it is goes out to you guys.  The schedule is not what we want it to be.   Either we delay all of EAP3 waiting for the performance improvements, or we release EAP3 now and do another release later.  I know this isn't anymore more than "trust us" but we are aware of the issue.

Thanks again for your comments and this post is in no way directed at you personally.

Because these constant complaints about a beta version are irrelevant and don't help anyone. I get tired of people who use Sophos for their home networks and complain about BS like Youtube cat videos or their shiny 400 mbit internet uplinks not being saturated. Use the stable version if you are concerned about throughput. You simply cannot expect a beta to have production quality. And thus, your complaining is just mute. Any annoying. 

Hi Michael,

I'm very grateful for your answer, this is the exact kind of answer I've been waiting from a Sophos employee, don't get me wrong, but if we had this exact answer when EAP 1 came out, there would be no thread discussing about v18 performance.

Michael Dunn said:

There are significant performance code changes that are not merged into EAP3.  The builds that we are currently testing internally are significantly faster than the EAP3 that you will be testing.  The performance tests you are running on EAP2 is code that is over a month old.  Even when EAP3 comes out the performance tests you will be running are effectively going to be on old code.  There are reasons that the new performance code is not yet integrated, we are waiting for a few more changes and we want to run it internal dogfoods before it is goes out to you guys.  The schedule is not what we want it to be.   Either we delay all of EAP3 waiting for the performance improvements, or we release EAP3 now and do another release later.  I know this isn't anymore more than "trust us" but we are aware of the issue.

Thank you again, this answer is perfect.

I'll be waiting for future releases.

Thanks again for your concise and helpful response. I always enjoy your engagement with the community and the glimpse you offer on the inner workings and the behind the scenes thinking at sophos headquarters.

Michael Dunn said:

I agree that we should have good throughput numbers in the beta before GA. 

Thanks for this. We shouldn't have to wait till GA, nor should sophos wait till the GA to guage how their products are behaving in the field. The reason I was mostly concerned was because there was some regression in the throughput even without depolying the DPI engine in some tests. At this point in the age of XG, some baseline performance is expected even during the beta phase so when the throughput dropped significantly without any clear explanations, it was concerning to some of us. 

Thanks again for taking the time as I always appreciate your detailed answers. Also glad that you didn't take my post as a dig at you or sophos which lots of people perceive them to be. We don't use sophos products at work any longer and I don't use sophos at home other than UTM for protecting my mail/outlook server in the lab. However it is always christmas time when new software is released and my criticism originates from my long appreciation for astaro and then for sophos and the higher standards that you and I both expect from them.

Regards

Bill

Please review your language it is totally inappropriate and not called for.

Ian

FloSupport: we need more inputs from Sophos Staff like Michael Dunn does. Community really enjoys this kind of "active participation" instead of SILENCE.

I know Michael Dunn since many years on the forum and all the time, I stop reading his responses as they are more informative than several official Sophos KB.

We would like to see at least one active dev for each XG's Unit on the community. Hope you understand our point, Flo.

Thanks Michael. We see the passion you have in your job.

How can we help debug the issue ? Running gigabit connection on comcast and all my speed test shows is  60mbs I know my comcast can be saturated but I would expect higher speeds. My cpu usage is 7%

How can we help debug the issue ? Running gigabit connection on comcast and all my speed test shows is  60mbs I know my comcast can be saturated but I would expect higher speeds. My cpu usage is 7%

Hi Jack,

please try again after disabling or removing IPS from your firewall rule.

Ian

Thank you rfcat_vk I had selected traffic shaping that was limiting my bandwidth.  On wifi I get 300mbs now instead of 60