This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

osquery crushing CPU on Ubuntu 22.04 Server

In relation to this, which is a closed thread with no real solution:
Extreme High CPU Usage with sophps protection with Linux

I'm a bit confused here. We have this issue reoccurring on a Linux server at this point, entirely randomly. osqueryd eats up 100% CPU for what seems to be randomly, for an indefinite period of time, and affects the functionality of our server running the latest SPL client. I've tried numerous things to no avail. This appears to be part of EDR from systemctl output.

Are we supposed to create a cron job to just routinely delete all of the database files at some routine interval? This doesn't seem like a real solution. It would make more sense to solidify a way to locate what EDR / osqueryd is doing (or trying to do) and exempt it. This seems like the right way to do it. Is there a way to observe (via logs, etc) exactly what it's trying to do and make appropriate adjustments (for example, to the default policy for some piece of the client) to prevent this from happening? We can't have Sophos eating up valuable CPU that the system's running services require to function properly.

I have put some time into toying with osqueryd flags, but they seem to reset to defaults after a service restart via systemd.

Any help here would be appreciated. It would be much easier to figure out what it's doing and why, and exempt it from happening. Otherwise, I'm automatically nuking the entire osquery database twice a day.



This thread was automatically locked due to age.
Parents
  • Having the same issue here on one of our servers since ~ last week.

    VERSION="22.04.4 LTS (Jammy Jellyfish)"

    Linux <redacted> 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux


    Tried to clear the database of osquery and reinstall the edr agent as well, neither helped.


  • Try checking the "edr_osquery.log" in the following directory.
    /opt/sophos-spl/plugins/edr/log/

    If you are not able to find any log lines of note, you can also try increasing the verbosity of logging by checking the following .conf file. 
    - /opt/sophos-spl/base/etc/logger.cof

    By adding/un-commenting the following, you will increase the verbosity of logging for EDR.

    [edr_osquery]
    VERBOSITY = DEBUG
    

    Once enabled, please check for entries matching "Maximum sustainable CPU utilization" within the log file "edr_osquery.log".

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
  • So, here is what happened in our case. One of the services running on the host didnt play nice with the latest package upgrade.
    Ended up spamming messages in /var/log/syslog. Osquery seems to process all these messages, and from that came the constantly high CPU usage of osquery. Would be cool if it was either more efficient handling those messages or maybe ratelimiting itself. 

    Thanks for the helpful advice though.

  • Interestingly, that's what (at least some) of the values given by Support do -- rate limit osqueryd in some fashion. I'm now 3 weeks or so into having made the changes in my screenshot (all of them, not just one or a couple) and have been good on our monitoring server. My guess is that its interpretation of stuff like a socket file or quickly moving logfiles ends up being detrimentalto its functionality, and the values given for osquery.flags tone that down a bit.

  • Mind Sharing the flags you ended up using, for future reference, if anyone else runs into the same issues?

  • Sorry - I thought my post actually made it into the thread -- it appears that it didn't. Here it is in entirety. I used the values in the screenshot/image:

  • 2 screenshots for the entire response:

Reply Children
No Data