Sophos XG 125 Huge memory Usage

Hi,

that is not good, please review the disk usage, all directories, sounds like the disk might be full and swap failing.l

ian

Any suggestions?

BR,

ALessandro

Yes, I restarted it and I will monitor in the next hours. I noticed 5 httpd processes consuming RAM. Is it normal?

I guess this is Webadmin, Userportal etc. - Web services on the XG.

snort processes are again using much memory. Is this screenshot after you restarted IPS?

Looks like much to much reserved RAM is used by them. They consume 1.2GB Our SNORT processes use 750MB after days of work.

Mem:  16315952k total, 15745324k used,   570628k free,   693288k buffers
Swap: 16305852k total,        0k used, 16305852k free,  9880132k cached

  PID  PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2314  20   0 5152m 751m  24m S 59.1  4.7  65:48.10 snort
 3198  20   0  162m  75m  12m S  4.0  0.5 374:15.26 garner
 2313  20   0 5153m 753m  24m S  3.7  4.7  49:38.89 snort
 2312  20   0 5152m 750m  24m S  2.3  4.7  61:28.67 snort
 2311  20   0 5156m 755m  24m R  2.0  4.7  42:16.61 snort
 4830  20   0 19484  13m 5432 S  2.0  0.1  40:34.56 sslvpn
 5875  20   0 25552 6544 2720 S  1.7  0.0  29:20.25 fqdnd

Note, this is a XG430 with 16GBRAM.

Yes, this image has been taken after IPS restart. Consider that I am working on a XG125 with 4GB Ram.

if you stop IPS service, do the snort processes disappear completely?

XG430_WP02_SFOS 18.0.5 MR-5-Build586# top | grep snort
 2314  20   0 5152m 751m  24m R 94.1  4.7  68:46.16 snort
 2312  20   0 5152m 750m  24m S  5.9  4.7  61:39.24 snort
 2311  20   0 5156m 755m  24m S  3.9  4.7  42:27.61 snort
 2313  20   0 5153m 753m  24m R  3.9  4.7  49:51.43 snort

Yes

we're at this IPS version.

but our XG is on Firmware 18.0 MR5

maybe some other 18.5.1 User can post a top on snort.

Same IPS signature for me. I don't really know what to do

LuCar Toni I think you may have an answer to that as you replying to many IPS threads here.

Wrote something in the German Community about this: 

The question is, is that a problem?

Basically behind this is a Linux-based operating system that can handle RAM differently than a Windows. See: https://www.linuxatemyram.com/

In my opinion it is a waste not to use the RAM resources that are available.

With version V18, Snort was more integrated into the architecture of the operating system. That means SFOS uses more Snort for all possible areas. And since the RAM is there and not in use, Snort takes what it can get.

As long as there is no overflow or MAX Out, I see no problem.

You should watch how the RAM behaves, if it stays constantly on this line, that's fine in my eyes.

My problem is that XG125 consumes all SWAP as well, becoming unusable after it reaches 100%

My problem is that XG125 consumes all SWAP as well, becoming unusable after it reaches 100%

How long does it take to consume the memory? And can you identify the process, taking more and more RAM? 

Since I installed latest release 18.5.1, in about 3 days. I suppose is due to IPS but it could be any component.

Hello Alessandro,

Adding to what has been told in this thread, can you share your Case ID to see what has been done. 

I would also recommend you to run the following command form the advanced shell and provide the output to Support 

# atop -M 3600 -w /var/atop_output.raw

You will need to reboot the device, then run the command and wait for the Memory utilization to fill out again, then restart and provide the file to Support.

Additionally, you can run the following command that will tell you what service is occupying the SWAP memory

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

# free

# ls -lh /var/cores

And more importantly how many devices are behind the Firewall and what services are you using?

Regards,

Hi Emmanuel,

the opened support case ID is 04496415 . This week end happened again without user on place. I had to restart IPS service to flush all swap usage.

This morning before I restarted IPS, please note uptime

Below the output on the command 

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

awk: /proc/22165/status: No such file or directory
awk: /proc/22173/status: No such file or directory
zebra 428 kB
zebra 16348 kB
xgs-healthmond 11444 kB
writeback
worker 12192 kB
worker 12192 kB
worker 12192 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
watchdog/3

XG125_XN03_SFOS 18.5.1 MR-1-Build326# ls -lh /var/cores
-rw------- 1 root 0 7.0M Oct 7 15:45 core.awarrenhttp
-rw------- 1 root 0 45.3M May 4 16:10 core.garner

Behind firewall there are 50 physical devices and 100-200 Virtual machines. IPS is enable only for physical devices.

BR,

Alessandro

SWAP Usage after 2 hours of IPS restarted

I disabled IPS on almost the rules. It remains active on few rules without traffic. I don't know which component is consuming so much swap memory.

Hello Alessandro,

Thank you for the Case ID. I can see the engineer has reach out to GES to see if  there is a way to increase the SWAP memory.

I can see you opened the Case on Oct 8 and the day before your appliance generated a core dump for the Awarrenhttp 

-rw------- 1 root 0 7.0M Oct 7 15:45 core.awarrenhttp

How many users are making use of the Web Filter/ DPI?

The XG 125 without using IPS, Web Filter, it is designed to handle no more than 50 users.

Most likely your device is undersized for this traffic, I would recommend you to check with your Sales Engineer to confirm.

18.5.1 is more resource intense, than 17.5 so from my point of view that would explain the increase in memory. 

Regards,

Good morning Emmanuel,

Behind the XG125 there are more or less 50 users, someday less someday more. The strange things is that now the 100% of swap is reached also in the weekend, without users connected. So, to answer at your question, Web Filter /DPI is active for more or less 50 users.

Thank you for your support,

Alessandro

Snort is not IPS. 

Snort is one of the backbone modules, used for most technologies in SFOS. Therefore, if there is an issue with Snort, it does not matter, if you unload or load pattern in Firewall rules. 

Could you check the console first? 

console> show ips-settings
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 0
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method hyperscan
sip_preproc enabled
sip_ignore_call_channel enabled
inspect all-content

-------------IPS Instances------------
IPS is running on all cores

console> show ips-settings
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 10
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method ac-q
sip_preproc enabled
sip_ignore_call_channel enabled
inspect untrusted-content

-------------IPS Instances------------
IPS CPU
1 0
2 1

I changed, yesterday, only maxsesbytes value as suggested by a Sophos Engineer.