Sophos XG 125 Huge memory Usage

Hi,

that is not good, please review the disk usage, all directories, sounds like the disk might be full and swap failing.l

ian

Any suggestions?

BR,

ALessandro

I already raised a support case. Also the standby node starting to use swap memory. They suggested me to reimage primary appliance but I don't want to perform this operation without further investigation.

Alessandro

The Swap flushes only after a IPS restart

alessandro scuderi said:

I already raised a support case. Also the standby node starting to use swap memory. They suggested me to reimage primary appliance

That's Sophos support quality. So sad. Yesterday I filed a case and wrote: I did "A" and had a HA failure. The first and until now only answer to my high priority case was 6h later: "why did you do "B"? B is not supported." I guess they did'nt even look at the logs I provided.

To your problem: did you check if you have high traffic going through IPS, and maybe have enabled too many checks?

IPS is tweakable, so for linux machines there is no need to scan for windows attack vectors.

Or maybe your server backups are running through IPS enabled firewall rules - would suggest to disable this.

Maybe you can check how the memory usage on your XG grows and find, it peaks at specific times a day?

Hi LHerzon,

memory consumptions started since I upgraded from rel 17 to 18 on may. You can see clearly on the following image. I can try to disable IPS on all rules to see if memory is released.

please restart IPS after you removed it from your rules and monitor it for one day.

Yes, I restarted it and I will monitor in the next hours. I noticed 5 httpd processes consuming RAM. Is it normal?

I guess this is Webadmin, Userportal etc. - Web services on the XG.

snort processes are again using much memory. Is this screenshot after you restarted IPS?

Looks like much to much reserved RAM is used by them. They consume 1.2GB Our SNORT processes use 750MB after days of work.

Mem:  16315952k total, 15745324k used,   570628k free,   693288k buffers
Swap: 16305852k total,        0k used, 16305852k free,  9880132k cached

  PID  PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2314  20   0 5152m 751m  24m S 59.1  4.7  65:48.10 snort
 3198  20   0  162m  75m  12m S  4.0  0.5 374:15.26 garner
 2313  20   0 5153m 753m  24m S  3.7  4.7  49:38.89 snort
 2312  20   0 5152m 750m  24m S  2.3  4.7  61:28.67 snort
 2311  20   0 5156m 755m  24m R  2.0  4.7  42:16.61 snort
 4830  20   0 19484  13m 5432 S  2.0  0.1  40:34.56 sslvpn
 5875  20   0 25552 6544 2720 S  1.7  0.0  29:20.25 fqdnd

Note, this is a XG430 with 16GBRAM.

Yes, this image has been taken after IPS restart. Consider that I am working on a XG125 with 4GB Ram.

if you stop IPS service, do the snort processes disappear completely?

XG430_WP02_SFOS 18.0.5 MR-5-Build586# top | grep snort
 2314  20   0 5152m 751m  24m R 94.1  4.7  68:46.16 snort
 2312  20   0 5152m 750m  24m S  5.9  4.7  61:39.24 snort
 2311  20   0 5156m 755m  24m S  3.9  4.7  42:27.61 snort
 2313  20   0 5153m 753m  24m R  3.9  4.7  49:51.43 snort

if you stop IPS service, do the snort processes disappear completely?

XG430_WP02_SFOS 18.0.5 MR-5-Build586# top | grep snort
 2314  20   0 5152m 751m  24m R 94.1  4.7  68:46.16 snort
 2312  20   0 5152m 750m  24m S  5.9  4.7  61:39.24 snort
 2311  20   0 5156m 755m  24m S  3.9  4.7  42:27.61 snort
 2313  20   0 5153m 753m  24m R  3.9  4.7  49:51.43 snort

Yes

we're at this IPS version.

but our XG is on Firmware 18.0 MR5

maybe some other 18.5.1 User can post a top on snort.

Same IPS signature for me. I don't really know what to do

LuCar Toni I think you may have an answer to that as you replying to many IPS threads here.

Wrote something in the German Community about this: 

The question is, is that a problem?

Basically behind this is a Linux-based operating system that can handle RAM differently than a Windows. See: https://www.linuxatemyram.com/

In my opinion it is a waste not to use the RAM resources that are available.

With version V18, Snort was more integrated into the architecture of the operating system. That means SFOS uses more Snort for all possible areas. And since the RAM is there and not in use, Snort takes what it can get.

As long as there is no overflow or MAX Out, I see no problem.

You should watch how the RAM behaves, if it stays constantly on this line, that's fine in my eyes.

My problem is that XG125 consumes all SWAP as well, becoming unusable after it reaches 100%

How long does it take to consume the memory? And can you identify the process, taking more and more RAM? 

Since I installed latest release 18.5.1, in about 3 days. I suppose is due to IPS but it could be any component.

Hello Alessandro,

Adding to what has been told in this thread, can you share your Case ID to see what has been done. 

I would also recommend you to run the following command form the advanced shell and provide the output to Support 

# atop -M 3600 -w /var/atop_output.raw

You will need to reboot the device, then run the command and wait for the Memory utilization to fill out again, then restart and provide the file to Support.

Additionally, you can run the following command that will tell you what service is occupying the SWAP memory

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

# free

# ls -lh /var/cores

And more importantly how many devices are behind the Firewall and what services are you using?

Regards,

Hi Emmanuel,

the opened support case ID is 04496415 . This week end happened again without user on place. I had to restart IPS service to flush all swap usage.

This morning before I restarted IPS, please note uptime

Below the output on the command 

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| sort -k 2 -n -r | head -20

awk: /proc/22165/status: No such file or directory
awk: /proc/22173/status: No such file or directory
zebra 428 kB
zebra 16348 kB
xgs-healthmond 11444 kB
writeback
worker 12192 kB
worker 12192 kB
worker 12192 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11076 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
worker 11004 kB
watchdog/3

XG125_XN03_SFOS 18.5.1 MR-1-Build326# ls -lh /var/cores
-rw------- 1 root 0 7.0M Oct 7 15:45 core.awarrenhttp
-rw------- 1 root 0 45.3M May 4 16:10 core.garner

Behind firewall there are 50 physical devices and 100-200 Virtual machines. IPS is enable only for physical devices.

BR,

Alessandro