DPI on 4 GB RAM system - Linkedin still does not open

I realize this doesn’t actually help you but just wanted to mention I enabled SSL/TLS decryption and have no issues accessing LinkedIn. I also have 4GB of RAM but running an Intel Core i5-5250U. My memory utilization is sitting around 85% with IPS and Application scanning off on all my firewall rules. It’s around 90% with IPS and Application scanning applies to my firewall rules.

I am beginning to think DPI will need a couple of versions to mature. I don't have any problems like  when accessing linkedin but the proxy seems a lot more mature compared to DPI and I am sticking with it. Also, DPI seems starved on a 4GB system. I am not running any IPS rules nor am I using ATP and my memory is around 85 percent with only a couple of firewall rules and only ONE user ME.

I don't think the system is tuned so far to intel that a very recent amd chip can't compete with older intel units. Hyperscan https://software.intel.com/en-us/articles/hyperscan-and-snort-integration may have a little advantage towards intel but I don't think that is the issue here. 

I have said many times during this beta that snort can only do so much with memory caps that have to be used in a XG type device. I think we are running into the limitations of snort here. The non multi-threaded design also doesn't help here and we may be in situation where we are too cutting edge even for the very fast cheap processors we have available.

Can one of you guys with access to higher end appliances run a test on DPI, IPS, ATP, and application categorization enabled and see if it brings an appliance to its knees. If I had to guess, I would think the performance would be very similar that we are seeing on our own setups in labs and home.  

I definitely agree that 4GB seems to be "bottoming out" on the XG and has done since v16.5, it's the main reason why I hate the XG8x and XG1xx series (minus the 135, he's ok).

The other issue which causes the high memory load is regardless if it is use or is licensed, pretty much all services will be active/live or will be loaded into memory or not running. This includes Web Proxy and IPS which are the two largest memory consumers.

Optimisations don't always mean "performance increase", they can be also be for stability and various other niggles and issues to be ironed out which may not apply to a competitor product. I have no doubt that an EPYC 7H12 will take a steaming <you know what I mean> all over a dual Xeon 8180 system and then make your breakfast for you in the morning while you hobble down the stairs. But that's not the point why a specific architecture is selected and optimised for.

I am extremely disappointed still that the multithread Snort is not in use and have instead opted for a multi-process approach which is something I mistook for quite a while. This means that proper threading loses out and performance is left on the table.

I sadly do not have access to the higher end equipment or even software to test this, the largest I have direct access to test on are Xeon E5 second gens. Would that help?

Emile

Oh no, I didn't mean throw the largest processor you have on it. I am sure the packet processing will be a lot faster on a newer i9 compared to an i3. What I am interested in are the sophos appliances that you guys are selling. Would they be able to handle and provide the throughput numbers published by sophos or the new numbers will be published shortly? Also people that bought a low end device that was good enough for v17.xx very recently... would it be just as good for v18 or would they be calling every day due to performance issues with DPI and additional ram demands.

And as a basic rule, I don't mind ALL my ram being utilized but XG is a firewall at heart. With cloud deployments, we should be looking at lighter footprint and not a cpu and memory hog.

Regards.

Billybob said:

I am beginning to think DPI will need a couple of versions to mature. I don't have any problems like lferrara when accessing linkedin but the proxy seems a lot more mature compared to DPI and I am sticking with it. Also, DPI seems starved on a 4GB system. I am not running any IPS rules nor am I using ATP and my memory is around 85 percent with only a couple of firewall rules and only ONE user ME.

With every EAP refresh I expect the CPU / memory to get better. I would not be surprised if this is a focus on the first few MRs as well. I make no promises, but our target is that the current 4GB hardware works fine. Do not use the current EAP performance as a guide for future system sizing.

I wanted to make a minor point and create a distinction.

There are two different things that are new and different in v18. Snort performing SSL decryption on all ports (internally referred to as SSLx) and web proxy enforcement within snort (internally referred to as web-in-snort, in the firewall rule it is called DPI mode).

I think a lot of the current CPU / memory / performance issues in the EAP are around snort inspecting every packet and snort decrypting SSL packets, in other words sslx.

In terms of processing web rules (DPI mode selection within the firewall) we have not had a lot of negative feedback in the EAP. Yes the DPI mode is less mature (really a infant compared to the awarrenhttp proxy) but so far we've had few issues. There are some architectural limitations (eg "that's just the way it works") but overall I think it is good. Let me know if that is not the case.

If you are having performance problems even if none of your rules specify "DPI mode" then the problem is in sslx.
If you are using "DPI Mode" and are having SSL/TLS decryption problems (everything works if decryption is off)  then the problem is probably in sslx.
If you are having enforcement problems when using "DPI mode" that exist in both HTTP and HTTPS then the problem is in web-in-snort.

At this point, I don't know which side is responsible for the linkedin problems, but I suspect sslx.  I know that no one of the public in the forums cares, you just want it fixed.  But I thought that I would give you a glimpse into the inner workings.  Now that I've shown you, I'll have to kill you.  :)

Billybob said:

I am beginning to think DPI will need a couple of versions to mature. I don't have any problems like lferrara when accessing linkedin but the proxy seems a lot more mature compared to DPI and I am sticking with it. Also, DPI seems starved on a 4GB system. I am not running any IPS rules nor am I using ATP and my memory is around 85 percent with only a couple of firewall rules and only ONE user ME.

With every EAP refresh I expect the CPU / memory to get better. I would not be surprised if this is a focus on the first few MRs as well. I make no promises, but our target is that the current 4GB hardware works fine. Do not use the current EAP performance as a guide for future system sizing.

I wanted to make a minor point and create a distinction.

There are two different things that are new and different in v18. Snort performing SSL decryption on all ports (internally referred to as SSLx) and web proxy enforcement within snort (internally referred to as web-in-snort, in the firewall rule it is called DPI mode).

I think a lot of the current CPU / memory / performance issues in the EAP are around snort inspecting every packet and snort decrypting SSL packets, in other words sslx.

In terms of processing web rules (DPI mode selection within the firewall) we have not had a lot of negative feedback in the EAP. Yes the DPI mode is less mature (really a infant compared to the awarrenhttp proxy) but so far we've had few issues. There are some architectural limitations (eg "that's just the way it works") but overall I think it is good. Let me know if that is not the case.

If you are having performance problems even if none of your rules specify "DPI mode" then the problem is in sslx.
If you are using "DPI Mode" and are having SSL/TLS decryption problems (everything works if decryption is off)  then the problem is probably in sslx.
If you are having enforcement problems when using "DPI mode" that exist in both HTTP and HTTPS then the problem is in web-in-snort.

At this point, I don't know which side is responsible for the linkedin problems, but I suspect sslx.  I know that no one of the public in the forums cares, you just want it fixed.  But I thought that I would give you a glimpse into the inner workings.  Now that I've shown you, I'll have to kill you.  :)