Sophos XG 125 Huge memory Usage

Hi,

that is not good, please review the disk usage, all directories, sounds like the disk might be full and swap failing.l

ian

Any suggestions?

BR,

ALessandro

we're at this IPS version.

but our XG is on Firmware 18.0 MR5

maybe some other 18.5.1 User can post a top on snort.

Same IPS signature for me. I don't really know what to do

LuCar Toni I think you may have an answer to that as you replying to many IPS threads here.

Wrote something in the German Community about this: 

The question is, is that a problem?

Basically behind this is a Linux-based operating system that can handle RAM differently than a Windows. See: https://www.linuxatemyram.com/

In my opinion it is a waste not to use the RAM resources that are available.

With version V18, Snort was more integrated into the architecture of the operating system. That means SFOS uses more Snort for all possible areas. And since the RAM is there and not in use, Snort takes what it can get.

As long as there is no overflow or MAX Out, I see no problem.

You should watch how the RAM behaves, if it stays constantly on this line, that's fine in my eyes.

My problem is that XG125 consumes all SWAP as well, becoming unusable after it reaches 100%

How long does it take to consume the memory? And can you identify the process, taking more and more RAM? 

Since I installed latest release 18.5.1, in about 3 days. I suppose is due to IPS but it could be any component.

Hello Alessandro,

Adding to what has been told in this thread, can you share your Case ID to see what has been done. 

I would also recommend you to run the following command form the advanced shell and provide the output to Support 

# atop -M 3600 -w /var/atop_output.raw

You will need to reboot the device, then run the command and wait for the Memory utilization to fill out again, then restart and provide the file to Support.

Additionally, you can run the following command that will tell you what service is occupying the SWAP memory

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

# free

# ls -lh /var/cores

And more importantly how many devices are behind the Firewall and what services are you using?

Regards,

Hi Emmanuel,

the opened support case ID is 04496415 . This week end happened again without user on place. I had to restart IPS service to flush all swap usage.

This morning before I restarted IPS, please note uptime

Below the output on the command 

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

awk: /proc/22165/status: No such file or directory
awk: /proc/22173/status: No such file or directory
zebra 428 kB
zebra 16348 kB
xgs-healthmond 11444 kB
writeback
worker 12192 kB
worker 12192 kB
worker 12192 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
watchdog/3

XG125_XN03_SFOS 18.5.1 MR-1-Build326# ls -lh /var/cores
-rw------- 1 root 0 7.0M Oct 7 15:45 core.awarrenhttp
-rw------- 1 root 0 45.3M May 4 16:10 core.garner

Behind firewall there are 50 physical devices and 100-200 Virtual machines. IPS is enable only for physical devices.

BR,

Alessandro

SWAP Usage after 2 hours of IPS restarted

SWAP Usage after 2 hours of IPS restarted

I disabled IPS on almost the rules. It remains active on few rules without traffic. I don't know which component is consuming so much swap memory.

Hello Alessandro,

Thank you for the Case ID. I can see the engineer has reach out to GES to see if  there is a way to increase the SWAP memory.

I can see you opened the Case on Oct 8 and the day before your appliance generated a core dump for the Awarrenhttp 

-rw------- 1 root 0 7.0M Oct 7 15:45 core.awarrenhttp

How many users are making use of the Web Filter/ DPI?

The XG 125 without using IPS, Web Filter, it is designed to handle no more than 50 users.

Most likely your device is undersized for this traffic, I would recommend you to check with your Sales Engineer to confirm.

18.5.1 is more resource intense, than 17.5 so from my point of view that would explain the increase in memory. 

Regards,

Good morning Emmanuel,

Behind the XG125 there are more or less 50 users, someday less someday more. The strange things is that now the 100% of swap is reached also in the weekend, without users connected. So, to answer at your question, Web Filter /DPI is active for more or less 50 users.

Thank you for your support,

Alessandro

Snort is not IPS. 

Snort is one of the backbone modules, used for most technologies in SFOS. Therefore, if there is an issue with Snort, it does not matter, if you unload or load pattern in Firewall rules. 

Could you check the console first? 

console> show ips-settings
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 0
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method hyperscan
sip_preproc enabled
sip_ignore_call_channel enabled
inspect all-content

-------------IPS Instances------------
IPS is running on all cores

console> show ips-settings
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 10
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method ac-q
sip_preproc enabled
sip_ignore_call_channel enabled
inspect untrusted-content

-------------IPS Instances------------
IPS CPU
1 0
2 1

I changed, yesterday, only maxsesbytes value as suggested by a Sophos Engineer.

Is this a old installation? 

Because i assumed, hyperscan should be enabled by most customers. But not if you use a old backup. 

Change search_method to hyperscan. 

Yes, It is an old installation.

After I change that setting snort process is using half of the RAM, at the moment. Could be the issue related to this setting?

Thank you,

Alessandro

New installations always use hyperscan to optimize the load of the pattern. You can read about it here: https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-hyperscan.html

But likely it will resolve your issue. You could restart the appliance now to flush the mem and it should be better. 

I will organize a reboot in a non work hours. I really appreciated your help! I'll let you know in these days the behaviour.

BR,

Alessandro

LuCar Toni nice job,that you figured this out! I think this is Alessandro's solution.

from SNORT Man

Queued match search methods - Matches arequeued until the fast pattern matcher is finished withthe payload, then evaluated. This was found to gen-erally increase performance through fewer cachemisses (evaluating each rule would generally blowaway the fast pattern matcher state in the cache).

∗ac and ac-q - Aho-Corasick Full (high memory, best performance).