SFOS 16.01.0 known IPS issue - Work arounds?

Hi Darrian,

To get a broader view on this, take SSH to XG and go to option 4. Device console and execute the command, show ips-settings. Post the output.

Which XG hardware model do you use and what is the number of concurrent active connection on XG when this issue is live? If there are some legitimate traffic being dropped through IPS, check in the Log Viewer>IPS page and allow the signature in the IPS policy.

Thanks

Hi, 

As requested: 

-------------IPS Settings-------------                                                              
        stream on                                                                                   
        lowmem on                                                                                   
        maxsesbytes 0                                                                               
        maxpkts 80                                                                                  
        mmap on                                                                                     
        enable_appsignatures on                                                                     
        http_response_scan_limit  65535                                                             
                                                                                                    
                                                                                                    
-------------IPS Instances------------                                                              
IPS CPU                                                                                             
 1  0                                                               

The issue is being experienced mainly on a PC (3GB RAM, dual core CPU @ 2.0GHz) running Sophos XG 16.1.01. The issue starts as soon as the IPS Service starts, and only 1 connection. 

                                

Hi Darrian,

Set the lowmem attribute for IPS off. Execute,

set ips lowmem-settings off

This will reduce the CPU consumption.

Thanks for that. I have given it a try and started the IPS Engine service. 

It seems to have lowered CPU usage while the service is trying to start, but it seems to be stuck trying to start and never actually starts now: 

sachingurung 

make sure to check that IPS engine is not eating all the resources on XG (even when the appliance is sized well for the environment).

XG installed on SG120 for example, the IPS engine eats all the CPU resources without any rule configured (but only the ips service).

Also a KB where you explain all the different settings of ips-setting is needed and helpful for users.

Thanks

Hi,

Does a system reboot help in that situation?

@Luk The IPS settings are usually not meant to be touched privately, any change must come from support officially. Which is why we do not have a KBA for the those settings for IPS.

Thanks

Thanks Sachin, however you need to improve ips performance. As I said try to install XG on sg 110/120 without any configuration and you will see how IPS eats the CPU and the 2 GB of ram on those systems and then reply back.

Thanks

Hi Darrian,

If possible report it to support and provide me the case#. I will still look around if I get a catch to this issue.

@luk - Thanks for the feedback, I agree and will raise this in the next internal team meet with the developers.

Thanks

Thank you for the suggestion. As far as I know, this being a home license, I am not able to report the issue to support. Is that correct? 

Thank you,

Darrian

Hello,

I'm in the same boat as Darrian.  I see the same warning messages as in his screenshot.  It seems to be cyclical though.  Sometimes it will show all is normal and IPS was started fine, then within a few minutes I'll see this warning pop up again.  Also, I'm not having the CPU or memory utilization he is though, mine has not risen above 30%.

I'm running XG 16.01.2 on a dedicated PC.  It's a new install that has very little config on it, as I've been trying to get this working before I complicated things.  I've tried reboots, stopping and starting the IPS service, and now I've re-installed the entire firewall software.  I've also tried what was suggested in this post with no results: https://community.sophos.com/products/xg-firewall/f/intrusion-prevention/10897/ips-engine-dead

Has anyone figured out how to get this working?

Thanks,

Jared

Hello,

I'm in the same boat as Darrian.  I see the same warning messages as in his screenshot.  It seems to be cyclical though.  Sometimes it will show all is normal and IPS was started fine, then within a few minutes I'll see this warning pop up again.  Also, I'm not having the CPU or memory utilization he is though, mine has not risen above 30%.

I'm running XG 16.01.2 on a dedicated PC.  It's a new install that has very little config on it, as I've been trying to get this working before I complicated things.  I've tried reboots, stopping and starting the IPS service, and now I've re-installed the entire firewall software.  I've also tried what was suggested in this post with no results: https://community.sophos.com/products/xg-firewall/f/intrusion-prevention/10897/ips-engine-dead

Has anyone figured out how to get this working?

Thanks,

Jared

Hi All,

who is experiencing this issue, could open a ticket with the Support and report the ticket number to sachingurung ?

The IPS problem should be fixed ASAP. I am sure that this is a bug and it occurs only on some systems.

Thanks

Thanks Iferrara,  as soon as the Contact Support link on Sophos's website works again, I'll submit a ticket.  I take it then that no one has figured this out?

I did discover one additional piece of information.  My CPU and Memory utilization would spike between 15-30% continually.  Once I just turned off the IPS service, it settled down to a constant value with no spiking at all.  Can't remember the exact percentage, I think it's somewhere between 5-10%.

Thanks,

Jared

Ok, made it in, and I got the below.  So unless there is someone else out there with an enterprise account, willing to ask on my behalf?

"Home User

Here's another question, is there a way to get into the actual Linux running Sophos?  Not just the limited Cisco'esk commands?  That way I can get see where the IPS service is failing?

Thanks,

Jared

Using ssh, option 5 and then 3.

Yeah as I stated earlier in the thread, it doesn't help that these home user licenses are guided to use the forum, and in the forum we are told to open a ticket when we cannot. 

I'm still having this issue on the latest release of the SFOS fw. Its the pits really, multiple revisions and still a problem. I still have the same CPU spiking and ram usage as before, with the service starting and failing on a loop. Pity because other than that I love the firewall. My Cyberoam CR300ing XP at the office doesn't exhibit this behavior at all. 

Here's some of the IPS.log and CSC.log in case anyone can look at it and tell me what's wrong.  Understand that I had IPS service turned off for a few days.  I just turned it on, and grabbed the logs in the IPS.log file from today to make it smaller.

Any sleuths out there that can tell me whats wrong?

Thanks,

Jared

8372.SophosLogs.zip

Ok, I see that the Snort process says it finishes starting successfully, however if you look at the files I posted, you see that the it is constantly being reloaded.  I'm hoping someone on this forum is able to help.

[Dec 07 01:18:22 :30756]:readfd cdata.cpipe[1] for pid 30854 set
INFO[30753]:Dec 07 01:18:22:s_worker.c:943:log_master_starttime:Snort start time(Success): 60 sec 220993 usec
[Dec 07 01:18:22 :30854]:Total preallocated memory : 2036.1094KB
[Dec 07 01:18:22 :30854]:Max memory alloc by webcat: 3732.1094KB
[Dec 07 01:18:22 :30756]:load module: '/sbin/modprobe nf_conntrack_pktq queuenum=100 qnum=0 maxsesbytes=0 mode=0' done
[Dec 07 01:18:22 :30756]:IPS now running
[Dec 07 01:18:22 :30756]:Time for change master(traffic bypass): 0 sec 38 msec
[Dec 07 01:18:22 :30756]:child 30854 dead
[Dec 07 01:18:22 :30756]:cdata[0].lstatus for pid 30854 set
[Dec 07 01:18:22 :30756]:IPS: child pid 30854 exited with signal 4
[Dec 07 01:18:22 :30859]:pktq DAQ configured to inline.
INFO[30859]:Dec 07 01:18:22:daq_pktq.c:459:InitPKTQ:msize 40000
INFO[30859]:Dec 07 01:18:22:daq_pktq.c:464:InitPKTQ:pkt_queue with nl_fd 7 device /dev/pktq0
[Dec 07 01:18:22 :30859]:Reload thread starting...
[Dec 07 01:18:22 :30859]:Reload thread started, thread 0x7f206caee700 (30860)

When I check the CSC log, I find these errors:

MESSAGE Dec 07 01:18:27 [ips:3131]: do_waitpid: pid 30753 exited with status 0
ERROR Dec 07 01:18:27 [ips:3131]: close(7) failed: Bad file descriptor
MESSAGE Dec 07 01:18:28 [ips:3131]: Child exited with status 1
ERROR Dec 07 01:18:28 [ips:3131]: do_stop: after_stop failed. not aborting!
ERROR Dec 07 01:18:30 [u2d_pt_installer:3111]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:18:30 [u2d_pt_installer:3111]: action with nofail failed
ERROR Dec 07 01:18:30 [u2d_dr_installer:3102]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:18:30 [u2d_dr_installer:3102]: action with nofail failed
ERROR Dec 07 01:19:30 [u2d_pt_installer:3111]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:19:30 [u2d_pt_installer:3111]: action with nofail failed
ERROR Dec 07 01:19:30 [u2d_dr_installer:3102]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:19:30 [u2d_dr_installer:3102]: action with nofail failed
MESSAGE Dec 07 01:19:49 [ips:3131]: do_stop(): status = EXITING
MESSAGE Dec 07 01:19:49 [ips:3131]: do_waitpid: pid 30986 exited with status 0
ERROR Dec 07 01:19:49 [ips:3131]: close(7) failed: Bad file descriptor
MESSAGE Dec 07 01:19:50 [ips:3131]: Child exited with status 1
ERROR Dec 07 01:19:50 [ips:3131]: do_stop: after_stop failed. not aborting!
ERROR Dec 07 01:20:30 [u2d_dr_installer:3104]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:20:30 [u2d_dr_installer:3104]: action with nofail failed
ERROR Dec 07 01:20:30 [u2d_pt_installer:3102]: nvram_get(is_eula): failed with -12
WARNING Dec 07 01:20:30 [u2d_pt_installer:3102]: action with nofail failed
MESSAGE Dec 07 01:21:11 [ips:3131]: do_stop(): status = EXITING

Thanks,

Jared

Hi guys,

how much do you have on your boxes? I am a home user with 8gb on my XG and not see these type of issues, though I do have some IPS features disabled because they block/throttle download traffic.

Hey rfcat,

I have 8gb of RAM, quad-core (I think), and SSD.  When I get home I can give you the exact hardware specs.  Do the logs make you think this is a RAM issue?

Sorry to hear about your IPS.  Did you have to disable the service, or just not enable it on your firewall rules?  I bought an SSD specifically so that it could do IPS quickly.

I noticed that when I turned off the IPS service, the hardware utilization dropped tremendously and the continual spiking stopped.  CPU load averages around .3 or .4.  Presumably the spiking is due to the IPS service constantly restarting itself.  

Wish someone from Sophos would help us out.  I work in IT and I've had Sophos try to sell me on this product for enterprise environments.  I want to like it because it is available for home use, and that I can train myself on it.  More companies should do that.

Thanks,

Jared