Windows Server Taskmanager performance issues with Sophos Endpoint Exploit Mitigation

Just to confirm, if you re-enable everything again in policy. Re-launch taskmgr to repro the issue and confirm it is still present. Then:

Close taskmgr.

Disable tamper for the computer

Rename  c:\windows\system32\hmpalert.dll to something else, e.g. hmpalert.dll.test

relaunch Taskmgr

does this also fix the issue? This would confirm your theory that exploit mitigation feature is responsible. You can exclude the process but it would be interesting to understand. 

good idea to disable the hmpalert module

indeed, this fixes the issue. Tabs in Taskmanager open immediately.

renaming it to the original filename, brings the issue back.

good idea to disable the hmpalert module

indeed, this fixes the issue. Tabs in Taskmanager open immediately.

renaming it to the original filename, brings the issue back.

Would be good if someone could confirm this behaviour before opening an other case.

Good to confirm, the next step would be to identify which mitigation is the cause.  Hopefully it is one, but it could be a combination I suppose.

I don't see this, but then, it could be related to the number of processes and the config of TaskManager.

Maybe run from an admin PS command prompt:
(Get-Process | Measure-Object).count

..to get a count of processes, does this server have a huge number of processes? 300-400 would be a typical number for an endpoint I would think. What do you have:

To rule out TaskManager config, I would:

1. Close Task Manager,

2. Open Regedit and navigate to:
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\TaskManager

Rename Preferences to PreferencesCustom. This will ensure a default Task Manager config. It would be good to rule out if any customisation could be involved, update interval, more columns, etc... You can always close Task Manager and rename back the original to restore any customisation.

I assume this is still slow?

Then I would open the Sophos log file under: C:\ProgramData\HitmanPro.Alert\Logs\

For processes hmpalert.dll is injected into, it logs the processes and the "features" applied to the processes.  E.g.

2023-08-04T16:19:14.833Z [Protected] PID 67104, Features 007D2E3000000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

In this case: 007D2E3000000004, which was the default policy for me.  I could be slightly different for your but it doesn't matter too much.

If in Central, for a custom Threat Protection policy linked to the test client, I uncheck "protect processes" as a starting point:


Wait for the policy to arrive. Which can be seen under the HMPA reg key:

A number of reg values will be removed for the disabled features as shown above. The left view is the new set, the right is the old.  Then, the next time the taskmgr.exe process is launched, it will have a different set of features applied, e.g.:

2023-08-04T16:25:19.211Z [Protected] PID 66160, Features 0078200000000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

In this case: 0078200000000004

I would disable the features in Central that apply to the Exploit Mitigation features, each time, wait for the policy and, then relaunch TaskMgr.exe checking the features has changed.  At some point, the speed will return, you can then start adding them back.  Hopefully you can get to the point where you can conclude "Prevent APC violation" for example is the problem mitigation.

---
As I mentioned, if you really have to as a workaround, under exclusions, you can add:


At the endpoint, under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\hmpalert\Config

PolicyInjectionExclusions will contain the path, e.g.

\Device\HarddiskVolume4\Windows\System32\Taskmgr.exe

As a result, the hmpalert driver will not inject hmpalert.dll into the taskmgr.exe process next time it starts.  I am curious which mitigation is the issue though.

Thanks

Hi, and thank you very much for your detailed and excellent guide!

There are only 120-130 tasks open.

Taskmanager with default settings has the same delay when starting.

In the log I can find some Alert about CredGuard:

That is what is logged in C:\ProgramData\HitmanPro.Alert\Logs\sophoshmpaservice.log when opening Taskmanager :

2023-08-07T14:42:07.717Z [ 2064: 2524] I [Protected] PID 9164, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-07T14:42:07.879Z [ 2064: 2484] I [Protected] PID 9208, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:42:07.944Z [ 2064: 2488] I [Protected] PID 4376, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:42:07.989Z [ 2064: 2480] I [Protected] PID 8232, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-07T14:42:09.501Z [ 2064: 2512] E WTSQueryUserToken failed with error code 1008, console session 2
2023-08-07T14:42:09.502Z [ 2064: 2512] W IsActiveSession failed, session 2
2023-08-07T14:42:09.527Z [ 2064: 2496] I [Protected] PID 8296, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\conhost.exe
2023-08-07T14:42:25.159Z [ 2064: 2512] E [Telemetry] FAILED Error 0, Status 502, C:\Program Files\Sophos\AutoUpdate\\Telemetry\SubmitTelem.exe
2023-08-07T14:42:25.159Z [ 2064: 2512] I [Alert] CredGuard, familyId=78004943-6fda-445a-b7cb-bb9ddbee350f, PID 8232, C:\Windows\System32\Taskmgr.exe

So it looks like an issue with "Prevent credential theft" component.

I'm logged in with RDP to the server with an AD user account. Probably that is an untested scenario for the CredGuard component (Error for "WTSQueryUserToken").

And we can see the 20sec delay.

Disabled it in Policy and Taskmanager runs fine now.

new logs:

2023-08-07T14:58:19.488Z [ 6136: 8492] I [Protected] PID 1960, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-07T14:58:19.641Z [ 6136: 8376] I [Protected] PID 8652, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:58:19.703Z [ 6136:  828] I [Protected] PID 8552, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-07T14:58:19.747Z [ 6136: 8324] I [Protected] PID 8620, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-07T14:58:27.639Z [ 6136: 8832] I [Protected] PID 1992, Features 08FD2C3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe

I could now exclude the process easily. But Sophos should not slow down the most basic system processes. Will continue at this point at a later time.

again: thank you very much Sophos User930 for your help here!

Glad you have made progress. I think you might need to raise a case with Support.  The information in here should be enough to at least try a reproduction of the scenario and theorise on if the checking the active session could be the cause. Thanks.

I checked my theory of RDP session issue. It seems not to be valid. It happens also when loggen on the server console.

Of course, there is no WTS error then.

This log is from a different Win 2022 Server then the machine above, with faster CPU.

The delay in the log is "only" 15 sec, the real delay of the taskmanager is 20 sec.

and the debug log from the first server.

2023-08-09T15:53:52.009Z [ 6136: 6772] A Log level changed from Info to Debug. Source: SOFTWARE\Sophos\Logging\HMPA
2023-08-09T15:53:57.590Z [ 6136: 8492] I [Protected] PID 6776, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\consent.exe
2023-08-09T15:53:57.749Z [ 6136: 8832] I [Protected] PID 1408, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.813Z [ 6136: 8832] I [Protected] PID 2352, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:53:57.862Z [ 6136: 8832] I [Protected] PID 4700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:53:59.413Z [ 6136:  680] E WTSQueryUserToken failed with error code 1008, console session 2
2023-08-09T15:53:59.415Z [ 6136:  680] W IsActiveSession failed, session 2
2023-08-09T15:53:59.416Z [ 6136:  680] D IsActiveSession succeeded, session 7
2023-08-09T15:53:59.416Z [ 6136:  680] D WTSQueryUserToken succeeded, session 7
2023-08-09T15:53:59.441Z [ 6136: 5808] I [Protected] PID 3700, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\conhost.exe
2023-08-09T15:53:59.476Z [ 6136: 8324] I [Protected] PID 7916, Features 08FD2E3040000104 Silent 0880000000000000, C:\Program Files\Sophos\AutoUpdate\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.920Z [ 6136:  680] E [Telemetry] FAILED Error 0, Status 502, C:\Program Files\Sophos\AutoUpdate\\Telemetry\SubmitTelem.exe
2023-08-09T15:54:14.921Z [ 6136:  680] I [Alert] CredGuard, familyId=77f10d91-d1c4-432f-8c82-11fca713312d, PID 4700, C:\Windows\System32\Taskmgr.exe
2023-08-09T15:54:18.104Z [ 6136: 6772] A Log level changed from Debug to Info. Source: default
2023-08-09T15:54:22.225Z [ 6136:  680] I [Protected] PID 2296, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\dllhost.exe
2023-08-09T15:54:27.079Z [ 6136: 8832] I [Protected] PID 884, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\rundll32.exe
2023-08-09T15:54:32.419Z [ 6136: 2692] I [Protected] PID 5600, Features 08FD2E3040000104 Silent 0880000000000000, C:\Windows\System32\notepad.exe

Support case: 06890340

I see the alert for Credguard in the log, the Event 911 in the Application Event log will detail more, but I assume it is TaskMgr.exe enumerating LSASS.exe that is triggering it?  It's info only alert.

If you get the Thumbrint from the 911 alert, e.g.

Thumbprint
b0326af4f72d6a40822f8a13058756716ef921dd5d2e8eba45735d4a3ab60bb0

In Central, under the exclusions for the device, add that as a "Detected Exploit (Windows/Mac)" using the "EXCLUDE EXPLOIT BY DETECTION ID" method.  The thumbprint will end up in WhiteThumprints value under: HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert

Next time you launch TaskMmgr.exe I assume you no longer get the alert in the log or in the Application Event log?

Is the time lost creating the alert maybe? Is it quicker with the feature enabled and the alert disabled? Thanks.

correct. what a huge(!) event log. Looks almost like a process minidump.

Information    10.08.2023 20:56:04    HitmanPro.Alert    911    Mitigation

Mitigation   CredGuard
Policy       CredGuard
Timestamp    2023-08-10T18:56:04

Platform     10.0.20348/x64 v2325 06_4f*
PID          9648
Enabled      007D2E3040000004
Application  C:\Windows\System32\Taskmgr.exe
Created      2023-01-04T16:25:49
Modified     2023-01-04T16:25:49
Description  Task Manager 10

Reading LSASS (704) process memory (read TRUNCATED): 000000E2161EB000 L2000 (could read L928)

... truncated.

Thumbprint
b0326af4f72d6a40822f8a13058756716ef921dd5d2e8eba45735d4a3ab60bb0

much more in the logs actually.

---

to answer your questions:

Is the time lost creating the alert maybe? obviously

Is it quicker with the feature enabled and the alert disabled? yes, still a short delay but not as long as before. 3-4 seconds now compared with ~20 seconds before.

log:

2023-08-10T19:13:42.603Z [ 1900: 2108] I [Protected] PID 10072, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\consent.exe
2023-08-10T19:13:42.750Z [ 1900: 2104] I [Protected] PID 8996, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

--

Great knowledge and great support from you. Thank you!

The support case is now at GES.

it looks like

2023-08-10T19:13:42.750Z [ 1900: 2104] I [Protected] PID 8996, Features 007D2E3040000004 Silent 0000000000000000, C:\Windows\System32\Taskmgr.exe

it is now excluded from Exploit Mitigation check like posted somewhere above. Not only the log disabled.

The problem has been identified by GES and is planned to be fixed in InterceptX 2023.2 later this year.

Thanks for the update. Good to know.