Thread Info
  • State Verified Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 13 replies
  • Answers 1 answer
  • Subscribers 14 subscribers
  • Views 6161 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Windows Server Taskmanager performance issues with Sophos Endpoint Exploit Mitigation

LHerzog

I notice a frustrating Performance issues on Windows Servers with the Taskmanager for some time now. Not sure if known or already fixed.

Taskmanager opens normally but when you switch to an other register tab like the Details register, it hangs like this

or shows only white space like this for up to 20-30 seconds, depending on CPU speed:

Once loaded, you can switch between the different registers and all is fine. If you close the Taskmanager and open it again, the issue is happening again.

After disabling Sophos Components one by one, it turned out to be  Exploit Mitigation causing the issue. Can be recreated at any time.

OS: Server 2019, Server 2022, and surely others.

Affected Endpoints, that's the latest Version we have:

All policy settings are default & recommended



This thread was automatically locked due to age.
Parents
  • 0

    Just to confirm, if you re-enable everything again in policy. Re-launch taskmgr to repro the issue and confirm it is still present. Then:

    Close taskmgr.

    Disable tamper for the computer

    Rename  c:\windows\system32\hmpalert.dll to something else, e.g. hmpalert.dll.test

    relaunch Taskmgr

    does this also fix the issue? This would confirm your theory that exploit mitigation feature is responsible. You can exclude the process but it would be interesting to understand. 

  • 0 in reply to Sophos User930

    good idea to disable the hmpalert module

    indeed, this fixes the issue. Tabs in Taskmanager open immediately.

    renaming it to the original filename, brings the issue back.

  • 0 in reply to LHerzog

    Would be good if someone could confirm this behaviour before opening an other case.

  • 0 in reply to LHerzog

    Good to confirm, the next step would be to identify which mitigation is the cause.  Hopefully it is one, but it could be a combination I suppose.

    I don't see this, but then, it could be related to the number of processes and the config of TaskManager.

    Maybe run from an admin PS command prompt:
    (Get-Process | Measure-Object).count

    ..to get a count of processes, does this server have a huge number of processes? 300-400 would be a typical number for an endpoint I would think. What do you have:

    To rule out TaskManager config, I would:

    1. Close Task Manager,

    2. Open Regedit and navigate to:
    HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\TaskManager

    Rename Preferences to PreferencesCustom. This will ensure a default Task Manager config. It would be good to rule out if any customisation could be involved, update interval, more columns, etc... You can always close Task Manager and rename back the original to restore any customisation.

    I assume this is still slow?

    Then I would open the Sophos log file under: C:\ProgramData\HitmanPro.Alert\Logs\

    For processes hmpalert.dll is injected into, it logs the processes and the "features" applied to the processes.  E.g.

    2023-08-04T16:19:14.833Z [Protected] PID 67104, Features 007D2E3000000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

    In this case: 007D2E3000000004, which was the default policy for me.  I could be slightly different for your but it doesn't matter too much.

    If in Central, for a custom Threat Protection policy linked to the test client, I uncheck "protect processes" as a starting point:


    Wait for the policy to arrive. Which can be seen under the HMPA reg key:



    A number of reg values will be removed for the disabled features as shown above. The left view is the new set, the right is the old.  Then, the next time the taskmgr.exe process is launched, it will have a different set of features applied, e.g.:

    2023-08-04T16:25:19.211Z [Protected] PID 66160, Features 0078200000000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

    In this case: 0078200000000004

    I would disable the features in Central that apply to the Exploit Mitigation features, each time, wait for the policy and, then relaunch TaskMgr.exe checking the features has changed.  At some point, the speed will return, you can then start adding them back.  Hopefully you can get to the point where you can conclude "Prevent APC violation" for example is the problem mitigation.

    ---
    As I mentioned, if you really have to as a workaround, under exclusions, you can add:


    At the endpoint, under:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\hmpalert\Config

    PolicyInjectionExclusions will contain the path, e.g.

    \Device\HarddiskVolume4\Windows\System32\Taskmgr.exe

    As a result, the hmpalert driver will not inject hmpalert.dll into the taskmgr.exe process next time it starts.  I am curious which mitigation is the issue though.

    Thanks

  • 0 in reply to Sophos User930

    Hi, and thank you very much for your detailed and excellent guide!

    There are only 120-130 tasks open.

    Taskmanager with default settings has the same delay when starting.

    In the log I can find some Alert about CredGuard:

    That is what is logged in C:\ProgramData\HitmanPro.Alert\Logs\sophoshmpaservice.log when opening Taskmanager :

    2023-08-07T14:42:07.717Z [ 2064: 2524] I [Protected] PID 9164, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-07T14:42:07.879Z [ 2064: 2484] I [Protected] PID 9208, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:42:07.944Z [ 2064: 2488] I [Protected] PID 4376, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:42:07.989Z [ 2064: 2480] I [Protected] PID 8232, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-07T14:42:09.501Z [ 2064: 2512] E WTSQueryUserToken failed with error code 1008, console session 2
2023-08-07T14:42:09.502Z [ 2064: 2512] W IsActiveSession failed, session 2
2023-08-07T14:42:09.527Z [ 2064: 2496] I [Protected] PID 8296, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\conhost.exe
2023-08-07T14:42:25.159Z [ 2064: 2512] E [Telemetry] FAILED Error 0, Status 502, C:\Program Files\Sophos\AutoUpdate\\Telemetry\SubmitTelem.exe
2023-08-07T14:42:25.159Z [ 2064: 2512] I [Alert] CredGuard, familyId=78004943-6fda-445a-b7cb-bb9ddbee350f, PID 8232, C:\Windows\System32\Taskmgr.exe

    So it looks like an issue with "Prevent credential theft" component.

    I'm logged in with RDP to the server with an AD user account. Probably that is an untested scenario for the CredGuard component (Error for "WTSQueryUserToken").

    And we can see the 20sec delay.

    Disabled it in Policy and Taskmanager runs fine now.

    new logs:

    2023-08-07T14:58:19.488Z [ 6136: 8492] I [Protected] PID 1960, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-07T14:58:19.641Z [ 6136: 8376] I [Protected] PID 8652, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:58:19.703Z [ 6136:  828] I [Protected] PID 8552, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:58:19.747Z [ 6136: 8324] I [Protected] PID 8620, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-07T14:58:27.639Z [ 6136: 8832] I [Protected] PID 1992, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
     

    I could now exclude the process easily. But Sophos should not slow down the most basic system processes. Will continue at this point at a later time.

    again: thank you very much  for your help here!

  • 0 in reply to LHerzog

    Glad you have made progress. I think you might need to raise a case with Support.  The information in here should be enough to at least try a reproduction of the scenario and theorise on if the checking the active session could be the cause. Thanks.

  • 0 in reply to Sophos User930

    I checked my theory of RDP session issue. It seems not to be valid. It happens also when loggen on the server console.

    Of course, there is no WTS error then.

    This log is from a different Win 2022 Server then the machine above, with faster CPU.

    The delay in the log is "only" 15 sec, the real delay of the taskmanager is 20 sec.

    and the debug log from the first server.

    2023-08-09T15:53:52.009Z [ 6136: 6772] A Log level changed from Info to Debug. Source: SOFTWARE\Sophos\Logging\HMPA
2023-08-09T15:53:57.590Z [ 6136: 8492] I [Protected] PID 6776, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-09T15:53:57.749Z [ 6136: 8832] I [Protected] PID 1408, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.813Z [ 6136: 8832] I [Protected] PID 2352, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.862Z [ 6136: 8832] I [Protected] PID 4700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:53:59.413Z [ 6136:  680] E WTSQueryUserToken failed with error code 1008, console session 2
2023-08-09T15:53:59.415Z [ 6136:  680] W IsActiveSession failed, session 2
2023-08-09T15:53:59.416Z [ 6136:  680] D IsActiveSession succeeded, session 7
2023-08-09T15:53:59.416Z [ 6136:  680] D WTSQueryUserToken succeeded, session 7
2023-08-09T15:53:59.441Z [ 6136: 5808] I [Protected] PID 3700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\conhost.exe
2023-08-09T15:53:59.476Z [ 6136: 8324] I [Protected] PID 7916, Features 08FD2E3040000104 Silent 0880000000000000, C:\Program Files\Sophos\AutoUpdate\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.920Z [ 6136:  680] E [Telemetry] FAILED Error 0, Status 502, C:\Program Files\Sophos\AutoUpdate\\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.921Z [ 6136:  680] I [Alert] CredGuard, familyId=77f10d91-d1c4-432f-8c82-11fca713312d, PID 4700, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:54:18.104Z [ 6136: 6772] A Log level changed from Debug to Info. Source: default
2023-08-09T15:54:22.225Z [ 6136:  680] I [Protected] PID 2296, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:54:27.079Z [ 6136: 8832] I [Protected] PID 884, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\rundll32.exe
2023-08-09T15:54:32.419Z [ 6136: 2692] I [Protected] PID 5600, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\notepad.exe

    Support case: 06890340

Reply
  • 0 in reply to Sophos User930

    I checked my theory of RDP session issue. It seems not to be valid. It happens also when loggen on the server console.

    Of course, there is no WTS error then.

    This log is from a different Win 2022 Server then the machine above, with faster CPU.

    The delay in the log is "only" 15 sec, the real delay of the taskmanager is 20 sec.

    and the debug log from the first server.

    2023-08-09T15:53:52.009Z [ 6136: 6772] A Log level changed from Info to Debug. Source: SOFTWARE\Sophos\Logging\HMPA
2023-08-09T15:53:57.590Z [ 6136: 8492] I [Protected] PID 6776, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-09T15:53:57.749Z [ 6136: 8832] I [Protected] PID 1408, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.813Z [ 6136: 8832] I [Protected] PID 2352, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.862Z [ 6136: 8832] I [Protected] PID 4700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:53:59.413Z [ 6136:  680] E WTSQueryUserToken failed with error code 1008, console session 2
2023-08-09T15:53:59.415Z [ 6136:  680] W IsActiveSession failed, session 2
2023-08-09T15:53:59.416Z [ 6136:  680] D IsActiveSession succeeded, session 7
2023-08-09T15:53:59.416Z [ 6136:  680] D WTSQueryUserToken succeeded, session 7
2023-08-09T15:53:59.441Z [ 6136: 5808] I [Protected] PID 3700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\conhost.exe
2023-08-09T15:53:59.476Z [ 6136: 8324] I [Protected] PID 7916, Features 08FD2E3040000104 Silent 0880000000000000, C:\Program Files\Sophos\AutoUpdate\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.920Z [ 6136:  680] E [Telemetry] FAILED Error 0, Status 502, C:\Program Files\Sophos\AutoUpdate\\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.921Z [ 6136:  680] I [Alert] CredGuard, familyId=77f10d91-d1c4-432f-8c82-11fca713312d, PID 4700, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:54:18.104Z [ 6136: 6772] A Log level changed from Debug to Info. Source: default
2023-08-09T15:54:22.225Z [ 6136:  680] I [Protected] PID 2296, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:54:27.079Z [ 6136: 8832] I [Protected] PID 884, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\rundll32.exe
2023-08-09T15:54:32.419Z [ 6136: 2692] I [Protected] PID 5600, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\notepad.exe

    Support case: 06890340

Children
  • 0 in reply to LHerzog

    I see the alert for Credguard in the log, the Event 911 in the Application Event log will detail more, but I assume it is TaskMgr.exe enumerating LSASS.exe that is triggering it?  It's info only alert.

    If you get the Thumbrint from the 911 alert, e.g.

    Thumbprint
    b0326af4f72d6a40822f8a13058756716ef921dd5d2e8eba45735d4a3ab60bb0

    In Central, under the exclusions for the device, add that as a "Detected Exploit (Windows/Mac)" using the "EXCLUDE EXPLOIT BY DETECTION ID" method.  The thumbprint will end up in WhiteThumprints value under: HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert

    Next time you launch TaskMmgr.exe I assume you no longer get the alert in the log or in the Application Event log?

    Is the time lost creating the alert maybe? Is it quicker with the feature enabled and the alert disabled? Thanks.

  • 0 in reply to Sophos User930

    correct. what a huge(!) event log. Looks almost like a process minidump.

    Information    10.08.2023 20:56:04    HitmanPro.Alert    911    Mitigation

    Mitigation   CredGuard
    Policy       CredGuard
    Timestamp    2023-08-10T18:56:04

    Platform     10.0.20348/x64 v2325 06_4f*
    PID          9648
    Enabled      007D2E3040000004
    Application  C:\Windows\System32\Taskmgr.exe
    Created      2023-01-04T16:25:49
    Modified     2023-01-04T16:25:49
    Description  Task Manager 10

    Reading LSASS (704) process memory (read TRUNCATED): 000000E2161EB000 L2000 (could read L928)

    ... truncated.

    Thumbprint
    b0326af4f72d6a40822f8a13058756716ef921dd5d2e8eba45735d4a3ab60bb0

    much more in the logs actually.

    ---

    to answer your questions:

    Is the time lost creating the alert maybe? obviously

    Is it quicker with the feature enabled and the alert disabled? yes, still a short delay but not as long as before. 3-4 seconds now compared with ~20 seconds before.

    log:

    2023-08-10T19:13:42.603Z [ 1900: 2108] I [Protected] PID 10072, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\consent.exe
    2023-08-10T19:13:42.750Z [ 1900: 2104] I [Protected] PID 8996, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

    --

    Great knowledge and great support from you. Thank you!

    The support case is now at GES.

  • 0 in reply to LHerzog

    it looks like

    2023-08-10T19:13:42.750Z [ 1900: 2104] I [Protected] PID 8996, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

    it is now excluded from Exploit Mitigation check like posted somewhere above. Not only the log disabled.

  • +1 in reply to LHerzog

    The problem has been identified by GES and is planned to be fixed in InterceptX 2023.2 later this year.

  • 0 in reply to LHerzog

    Thanks for the update. Good to know.

  • 0 in reply to Sophos User930

    A working fix is in a MR3 pre-Fix package KB-000038477 Sophos Central Intercept X Maintenance Release for Issue ID: WINEP-47494.

    It contains the following versions:

    Server Intercept-X: FTS 2023.1.0.104-MR3

    Hitman Pro Alert: 3.9.1.2431

    according to support, it will be integrated in the following Intercept-X release 2023.2 later this year.

Unfiltered HTML