Thread Info
  • State Not Answered
  • Replies 4 replies
  • Subscribers 15 subscribers
  • Views 1337 views
  • Users 0 members are here
Options
Suggested

Ubuntu Linux Server high CPU for hours osqueryd.4 at 100%

LHerzog

We have a 4 core VM with Linux SPL Agent 2024.2.1.2

4 regular processes run and each run intended at 100% - usually. Currently Sophos osquery.4 is consuming 1 core at 100% so the other 4 processes run at about 75% only, slowing down calculations of the server.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                         
 610732 root      30  10 1717512 500888  36472 R 100.0   3.1   0:25.77 osqueryd.4                                      
1642808 root      20   0 2773708   1.8g  31924 R  77.6  11.8 144:22.59 process1                                          
1642685 root      20   0 2790760   1.1g   2620 R  74.6   7.3 226:23.88 process2                                          
1642944 root      20   0 2790400   1.8g  32064 R  72.9  11.8 169:42.81 process3                                          
1642540 root      20   0 2773976 605560   2620 R  71.6   3.7 246:18.94 process4                                          
3912470 _apt      20   0 1267244  31348  12884 S   0.7   0.2   2:44.03 xxxx                                       
   1197 root      20   0 3664680  68696  30748 S   0.3   0.4  27:14.77 xxxx

Reading this posting:  osquery crushing CPU on Ubuntu 22.04 Server 

I checked the edr_osquery.log

I wonder what happens here?

Why do I find Windows Registry Paths in Logs of  SPL?

# tail /opt/sophos-spl/plugins/edr/log/edr_osquery.log
20286393 [2024-10-17T12:12:45.743]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:12:45.739905 340305 watcher.cpp:424] osqueryd worker (597248) stopping: Memory limits exceeded: 453951488 bytes (limit is 300MB)
20350733 [2024-10-17T12:13:50.083]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:13:50.079986 340305 watcher.cpp:424] osqueryd worker (597966) stopping: Memory limits exceeded: 457064448 bytes (limit is 300MB)
20415061 [2024-10-17T12:14:54.411]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:14:54.411895 340305 watcher.cpp:424] osqueryd worker (598752) stopping: Memory limits exceeded: 456790016 bytes (limit is 300MB)
20479401 [2024-10-17T12:15:58.751]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:15:58.751880 340305 watcher.cpp:424] osqueryd worker (599188) stopping: Memory limits exceeded: 455233536 bytes (limit is 300MB)
20543745 [2024-10-17T12:17:03.095]    WARN [0901132864] edr_osquery <>     CAST(SPLIT(file_versionW1017 14:17:03.095901 340305 watcher.cpp:424] osqueryd worker (599676) stopping: Memory limits exceeded: 454733824 bytes (limit is 300MB)
20608069 [2024-10-17T12:18:07.419]    WARN [0901132864] edr_osquery <>     CAST(SPLIT(file_versionW1017 14:18:07.415895 340305 watcher.cpp:424] osqueryd worker (600274) stopping: Memory limits exceeded: 453185536 bytes (limit is 300MB)
20672417 [2024-10-17T12:19:11.768]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:19:11.767884 340305 watcher.cpp:424] osqueryd worker (601377) stopping: Memory limits exceeded: 458436608 bytes (limit is 300MB)
20736753 [2024-10-17T12:20:16.104]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:20:16.103886 340305 watcher.cpp:424] osqueryd worker (601684) stopping: Memory limits exceeded: 454168576 bytes (limit is 300MB)
20801057 [2024-10-17T12:21:20.407]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:21:20.403895 340305 watcher.cpp:424] osqueryd worker (602783) stopping: Memory limits exceeded: 458321920 bytes (limit is 300MB)
20865369 [2024-10-17T12:22:24.720]    WARN [0901132864] edr_osquery <>     path = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\EnW1017 14:22:24.719880 340305 watcher.cpp:424] osqueryd worker (603205) stopping: Memory limits exceeded: 453947392 bytes (limit is 300MB)

This file is unchanged currently:

cat /opt/sophos-spl/plugins/edr/etc/plugin.conf
running_mode=1
network_tables=1
scheduled_queries_next=0
disable_auditd=1

Having case 01959114 open



Edit tags
[edited by: GlennSen at 10:06 AM (GMT -8) on 4 Nov 2024]
  • 0

    support suggested to delete osquery db,

    rm -rf /opt/sophos-spl/plugins/edr/var/osquery.db/*

    then create this file:

    /opt/sophos-spl/plugins/edr/etc/osquery.conf.d# cat memory_limit.conf
    {
    "options": {
    "watchdog_memory_limit": 1000,
    "watchdog_utilization_limit": 80,
    "watchdog_latency_limit": 48,
    "watchdog_delay": 240
    }
    }

    and add the last line here:

    cat /opt/sophos-spl/plugins/edr/etc/plugin.conf
    running_mode=1
    network_tables=1
    scheduled_queries_next=0
    disable_auditd=1
    events_max=50000


    wonder why not putting them here?


    cat /opt/sophos-spl/plugins/edr/etc/osquery.flags

    this shows the current default values:


    ...
    --events_max=50000
    --watchdog_memory_limit=300
    --watchdog_utilization_limit=30
    --watchdog_latency_limit=0
    --watchdog_delay=60

    ...

    unsure if the client is now really picking the flags set in memory_limit.conf

    @Sophos how can I determine if spl is now running with the desired flags?

  • 0

    +1: we have the same issue here.
    Activated SPL on five ubuntu lts 24.04 servers today -> result two of the servers have a high cpu usage (first server 350% on process sophos_thread_d, second server 290% on proccess sophos_thread_d). 

    looks like these "protection" is absolutly useless for production environments...


  • 0 in reply to nils50122

    Hi nils50122,

    Threat Detector is a different process from osqueryd.

    High Threat Detector CPU usage means you are doing lots of file scanning. Usually that means that on-access (Real-time) scanning is enabled.

    If you contact Support, there are procedures for investigating Threat Detector high CPU. 

Unfiltered HTML