This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

IPS service makes CPU up to 100% this morning

Dear All,

Since the support portal is still disable today, I only can share the issue over community.

This morning two of our customers found their XG's CPU was up to 100%.

And the network become unstable.





Stop the IPS service will resolve the issue.

But when restart IPS service, CPU is up to 100% again!

The firmware version is 17.5.14 MR-14.

My office's XG is this version too, but the issue didn't happen in my office.

Any suggestion?

Shunze



This thread was automatically locked due to age.
Parents
  • Hi Shunze,

    we had this issue too on aXG210 18.01  MR-1-Build396

    Can you login to the advanced shell and tell us what processes consume the CPU?

    What we found out:

    1) Even if you set Detect and prevent:None on the rule(s) that might be responsible for the issue there is still some IPS activity
    2) You can try to disable IPS completely under System Services - Services - IPS (only recommendet for testing)
    3) You can bypass the IPS for indivdual rules on the CLI: https://support.sophos.com/support/s/article/KB-000038900?language=en_US

    The issue we had was that in a testing zone / testing networks (handled/configured somewhat like a DMZ) some devices were choosing the destination address randomly and sending UDP Traffic down to the default gateway which was the firewall. A continous stream of aprox. 300 MBit/s UDP Traffic was fired to an internal interface of the firewall which led to 100% CPU load on both processors. As the firewall also was the routing device the response times were very high for a LAN environment.

    Finally we removed the devices sending random addresses and decoupled the the testing network via a routing net. We did not fully test switching the IPS off completely (which is not a solution at all) and the rule exceptions as these things came up later.

    The traffic in the network of your customers make the difference not the firewall and most probably not the rules.

    So I'd also suggest you look in the network for malformed traffic (especially broadcasts) and devices that are doing weird things (wireshark, mirror port, drop rule).


    Best regards,
    Bernd

Reply
  • Hi Shunze,

    we had this issue too on aXG210 18.01  MR-1-Build396

    Can you login to the advanced shell and tell us what processes consume the CPU?

    What we found out:

    1) Even if you set Detect and prevent:None on the rule(s) that might be responsible for the issue there is still some IPS activity
    2) You can try to disable IPS completely under System Services - Services - IPS (only recommendet for testing)
    3) You can bypass the IPS for indivdual rules on the CLI: https://support.sophos.com/support/s/article/KB-000038900?language=en_US

    The issue we had was that in a testing zone / testing networks (handled/configured somewhat like a DMZ) some devices were choosing the destination address randomly and sending UDP Traffic down to the default gateway which was the firewall. A continous stream of aprox. 300 MBit/s UDP Traffic was fired to an internal interface of the firewall which led to 100% CPU load on both processors. As the firewall also was the routing device the response times were very high for a LAN environment.

    Finally we removed the devices sending random addresses and decoupled the the testing network via a routing net. We did not fully test switching the IPS off completely (which is not a solution at all) and the rule exceptions as these things came up later.

    The traffic in the network of your customers make the difference not the firewall and most probably not the rules.

    So I'd also suggest you look in the network for malformed traffic (especially broadcasts) and devices that are doing weird things (wireshark, mirror port, drop rule).


    Best regards,
    Bernd

Children
No Data