This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

SFOS 16.01.0 known IPS issue - Work arounds?

Hey all,

Anyone have any other work around for the known IPS issue (NC-8238 [IPS] IPS Service drops legitimate traffic in very high load average conditions)? The IPS service seems to constantly fail to start and causes this issue from what I can see (CPU usage and memory usage spike all over the place). As my work around, I set the IPS service to Stop, performance and traffic return to normal. Obviously this isn't a great solution... Anyone have anything better?

I'd like to know when this will be resolved too, seems to me to be a rather big problem. I may actually just roll back to 15 if this is going to be a thing for a while.

Thanks !!

This thread was automatically locked due to age.

Parents

0 sachingurung over 7 years ago

Hi Darrian,

To get a broader view on this, take SSH to XG and go to option 4. Device console and execute the command, show ips-settings. Post the output.

Which XG hardware model do you use and what is the number of concurrent active connection on XG when this issue is live? If there are some legitimate traffic being dropped through IPS, check in the Log Viewer>IPS page and allow the signature in the IPS policy.

Thanks

Sachin Gurung
Team Lead | Sophos Technical Support
Knowledge Base | @SophosSupport | Video tutorials
Remember to like a post. If a post (on a question thread) solves your question use the 'This helped me' link.
Cancel
Vote Up 0 Vote Down

Cancel

0 Darrian Churchill over 7 years ago in reply to sachingurung

Hi,

As requested:

-------------IPS Settings-------------                                                              
        stream on                                                                                   
        lowmem on                                                                                   
        maxsesbytes 0                                                                               
        maxpkts 80                                                                                  
        mmap on                                                                                     
        enable_appsignatures on                                                                     
        http_response_scan_limit  65535                                                             
                                                                                                    
                                                                                                    
-------------IPS Instances------------                                                              
IPS CPU                                                                                             
 1  0                                                               

The issue is being experienced mainly on a PC (3GB RAM, dual core CPU @ 2.0GHz) running Sophos XG 16.1.01. The issue starts as soon as the IPS Service starts, and only 1 connection.

0 sachingurung over 7 years ago in reply to Darrian Churchill

Hi Darrian,

Set the lowmem attribute for IPS off. Execute,

set ips lowmem-settings off

This will reduce the CPU consumption.

Sachin Gurung
Team Lead | Sophos Technical Support
Knowledge Base | @SophosSupport | Video tutorials
Remember to like a post. If a post (on a question thread) solves your question use the 'This helped me' link.
Cancel
Vote Up 0 Vote Down

Cancel

Reply

0 sachingurung over 7 years ago in reply to Darrian Churchill

Hi Darrian,

Set the lowmem attribute for IPS off. Execute,

set ips lowmem-settings off

This will reduce the CPU consumption.

Sachin Gurung
Team Lead | Sophos Technical Support
Knowledge Base | @SophosSupport | Video tutorials
Remember to like a post. If a post (on a question thread) solves your question use the 'This helped me' link.
Cancel
Vote Up 0 Vote Down

Cancel

Children

0 Darrian Churchill over 7 years ago in reply to sachingurung

Thanks for that. I have given it a try and started the IPS Engine service.

It seems to have lowered CPU usage while the service is trying to start, but it seems to be stuck trying to start and never actually starts now:
Cancel
Vote Up 0 Vote Down

Cancel
0 lferrara over 7 years ago in reply to sachingurung

sachingurung

make sure to check that IPS engine is not eating all the resources on XG (even when the appliance is sized well for the environment).

XG installed on SG120 for example, the IPS engine eats all the CPU resources without any rule configured (but only the ips service).

Also a KB where you explain all the different settings of ips-setting is needed and helpful for users.

Thanks
Cancel
Vote Up 0 Vote Down

Cancel
0 sachingurung over 7 years ago in reply to Darrian Churchill

Hi,

Does a system reboot help in that situation?

@Luk The IPS settings are usually not meant to be touched privately, any change must come from support officially. Which is why we do not have a KBA for the those settings for IPS.

Thanks

Sachin Gurung
Team Lead | Sophos Technical Support
Knowledge Base | @SophosSupport | Video tutorials
Remember to like a post. If a post (on a question thread) solves your question use the 'This helped me' link.
Cancel
Vote Up 0 Vote Down

Cancel
0 Darrian Churchill over 7 years ago in reply to sachingurung

Unfortunately not. Upon reboot the service does the same for a while, eventually starts, stops again and repeats the process.
Cancel
Vote Up 0 Vote Down

Cancel
0 lferrara over 7 years ago in reply to sachingurung

Thanks Sachin, however you need to improve ips performance. As I said try to install XG on sg 110/120 without any configuration and you will see how IPS eats the CPU and the 2 GB of ram on those systems and then reply back.

Thanks
Cancel
Vote Up 0 Vote Down

Cancel
0 sachingurung over 7 years ago in reply to Darrian Churchill

Hi Darrian,

If possible report it to support and provide me the case#. I will still look around if I get a catch to this issue.

@luk - Thanks for the feedback, I agree and will raise this in the next internal team meet with the developers.

Thanks

Sachin Gurung
Team Lead | Sophos Technical Support
Knowledge Base | @SophosSupport | Video tutorials
Remember to like a post. If a post (on a question thread) solves your question use the 'This helped me' link.
Cancel
Vote Up 0 Vote Down

Cancel
0 Darrian Churchill over 7 years ago in reply to sachingurung

Thank you for the suggestion. As far as I know, this being a home license, I am not able to report the issue to support. Is that correct?

Thank you,

Darrian
Cancel
Vote Up 0 Vote Down

Cancel
0 Jared Mathis over 7 years ago in reply to Darrian Churchill

Hello,

I'm in the same boat as Darrian. I see the same warning messages as in his screenshot. It seems to be cyclical though. Sometimes it will show all is normal and IPS was started fine, then within a few minutes I'll see this warning pop up again. Also, I'm not having the CPU or memory utilization he is though, mine has not risen above 30%.

I'm running XG 16.01.2 on a dedicated PC. It's a new install that has very little config on it, as I've been trying to get this working before I complicated things. I've tried reboots, stopping and starting the IPS service, and now I've re-installed the entire firewall software. I've also tried what was suggested in this post with no results: https://community.sophos.com/products/xg-firewall/f/intrusion-prevention/10897/ips-engine-dead

Has anyone figured out how to get this working?

Thanks,

Jared
Cancel
Vote Up 0 Vote Down

Cancel
0 lferrara over 7 years ago in reply to Jared Mathis

Hi All,

who is experiencing this issue, could open a ticket with the Support and report the ticket number to sachingurung ?

The IPS problem should be fixed ASAP. I am sure that this is a bug and it occurs only on some systems.

Thanks
Cancel
Vote Up 0 Vote Down

Cancel
0 Jared Mathis over 7 years ago in reply to lferrara

Thanks Iferrara, as soon as the Contact Support link on Sophos's website works again, I'll submit a ticket. I take it then that no one has figured this out?

I did discover one additional piece of information. My CPU and Memory utilization would spike between 15-30% continually. Once I just turned off the IPS service, it settled down to a constant value with no spiking at all. Can't remember the exact percentage, I think it's somewhere between 5-10%.

Thanks,

Jared
Cancel
Vote Up 0 Vote Down

Cancel