Sophos Endpoint Intercept X 2.0 impacting Performance - slow?

Question

On a new software build of windows 10 on a T450 Lenovo, we found that at the end we installed Sophos Endpoint Intercept X 2.0 and it significantly slowed down the computer. All aspects of the computer became slow. On first bootup, connecting the Wifi - slow. On login, the CPU would pin at 100% for long periods of time with high memory usage. All applications would be slow to open, printing would be very slow. This is a new laptop i5, 8 GB RAM, 256 SSD. 
 We would remove the Intercept X and the computer would return to normal operation. Fast bootup, fast login, apps, etc... 
 Now for this customer, then use Trend Micro as their primary AV. We have Sophos Intercept X added on for the extra protection. We did not have issues previously until the Intercept X Version went up to 2.0. Has anyone else noticed a large performance hit with Intercept X 2.0?

FloSupport · Accepted Answer

Hi All, 
 We always advise to please create a new thread for your specific issue so that other community users can properly track your issue. 
 In regards to this thread and useful tips for troubleshooting your Intercept X issues, please note the following: 
 
 Useful tips from jak for performance troubleshooting
 
 Troubleshooting potential service startup performance issues 1 
 Troubleshooting potential service startup performance issues 2 
 Troubleshooting by certain mitigation check 
 Troubleshooting by drive exclusion

Sophos Intercept X and Exploit Prevention: Known issues 
 Sophos Central Endpoint: Basic troubleshooting 
 
 If you require immediate assistance or assistance troubleshooting, we would advise to please raise a support case so that further investigation can be performed for your situation. 
 Regards,

jak · Answer

I suppose a useful and very quick test would be to make a drive exclusion for simply: C: 
 So under the "Threat Protection" policy for the computer you're testing with set it as follows: 
 Note: it says (DRIVE). 
 If you remove scanning load do times improve? At least this way you know it's not just the weight of the services, the SAVService, loading virus data, etc... You can check the exclusion has made it in a few places at the endpoint but the value OnAccessExcludeFilePaths under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\Scanning\Config\ is probably the easiest. 
 If it does help then in theory an exclusion or two might suffice to restore speed. If nothing else it would be useful info to help further understand the issue. 
 If it doesn't help, disable the scanning of remote files. If it still doesn't help disable Tamper Protection for the computer in policy. These 3 tests help a lot to understand the nature of the issue. 
 My next test if excluding C: helps, would be try a directory exclusion for say: C:\windows\ 
 This will cover busy directories as system32, syswow64, WinSxS, Microsoft.NET, etc... and would also be useful. 
 At that point it might be worth capturing just a boot Process Monitor log with a destructive filter (to improve performance and reduce the log size) for file operations on in C:\windows\. Once collected, add the Duration column and maybe filter to just readfile, writefile operations. From there it might narrow it down further. This would be a fast way to troubleshoot the issue. 
 Regards, 
 Jak

jak · Answer

OK, that narrows it down. The next step for me would be to understand if a certain mitigation could be the cause.

In Central the mitigations are controlled in the threat protection policy which consists of the following relevant sections:

and ....

The next test might be to disable all the options above and restart having re-introduced the driver and native hmpalert.dll.

Does that help? Hopefully it does.. I would then consider try re-enabling the options one at a time. E.g.

"Enable CPU branch tracing"...

"Protect processes"

If you can find one or two options that helps, then at least you'd have a reasonable level of protection, starting services and support would have more to go on.

The other option here to rule in/out mitigations is to change settings locally in the registry at the client. This gives very fine grained control of what is enabled.

If you look in the log file of HMPA - "C:\ProgramData\HitmanPro.Alert\Logs\Sophos.log", there is a log line for each process launched and the mitigations (features) applied. E.g.

2018-07-04T21:40:19.329Z [Protected] PID 15384, Features 00072A3000000106, C:\Windows\System32\rundll32.exe
2018-07-04T21:40:22.055Z [Protected] PID 17464, Features 00072A300000010E, C:\Windows\System32\svchost.exe

The values 00072A3000000106 and 00072A300000010E represent the mitigations applied to the process.

Under:
HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\
Are the mitigation profiles applies to the certain class of applications. Media, Java, Browsers, etc...

You can see how under:
HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\java.exe
for example, the profile applied to the java process is the "Java" Profile. So you can see which mitigations get applied.

I assume that the svchost.exe processes you're having trouble with fall under the "Other" category: HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\Other\

All the DWORD values under this key show which mitigations apply to this category.

One option would therefore be to set the values to 0 and see what helps. When you toggle the registry values to control features, you do need to restart the Hitman Pro Alert service for the setting to take. That said, if you're going to be restarting the computer however I guess it doesn't matter. Hopefull you can see in the log file mentioned about the feature value changing.

The only thing I might do when performing this testing is to disable the "Sophos MCS Agent" service so the client doesn't apply any policies from Central during the testing.

I'm hoping you can then narrow it down to maybe one specific mitigations by registry value name.

Hopefully that makes sense.

Regards,
Jak

ChrisWhiting · Answer

Finally posted an official workaround, nearly a month after the suggestion answer post: https://community.sophos.com/kb/en-us/132998 
 
 Lets hope it doesnt take a month to finally fix it.

Shire of Mundaring · Answer

Hi All 
 We've had a call with Sophos about this for quite some time with a lot of back and forth gathering logs. We have now received this update and workaround from them: 
 
 This is a sequence of events leading to services timing out and not starting 1. A service is launched, e.g. SysMain, with command line "c:\windows\system32\svchost.exe -k localsystemnetworkrestricted -p -s SysMain" 2. Svchost.exe has "Code Integrity Guard" mitigation enabled in Audit mode. This feature comes from Windows Defender settings; there is a specific override for svchost.exe. 2.This feature checks that the binaries loaded/injected into the process are signed by Microsoft, but does not block them, since it is in audit mode only. 3. Hmpalert.dll (our driver) is injected into svchost.exe process by hmpalert.sys driver. 4. Because of "Code Integrity Guard", Windows checks the signature of hmpalert.dll, using functions in code integrity dll (ci.dll). 5. Windows checks both the embedded signature, and the signature in catalog (.cat) files &ndash; this causes ALL .cat files (thousands of files) to be opened. 6. Catalog file check happens in the context of one of the services (whichever was started first); all other services are waiting on the first one to finish the signature check. 7. The check can take a long time (due to opening thousands of files). In customer's procmons, the check was taking more than 1 minute. 8. If the check takes longer than 30 seconds, the service will time out (and all other services waiting for it). We are now looking into how we can improve this behaviour from our side - but are you able to please test the below workaround and see if this helps: 1. Open "Windows Defender Security Centre" 2. Go to "App & Browser control" 3. Scroll down and click on "Exploit protection settings" 4. Click on "Program Settings". 5. Wait for a list with a bunch of executable names to pop up (could take more than 10 seconds) 6. Scroll down until you find "svchost.exe" 7. Press "Edit" 8. Check if "Code Integrity Guard" is enabled in "Audit only" mode. If it is enabled, turn it off (uncheck "Override system settings"). 9. It might help to also disable "Arbitrary code guard" (if it is in "Audit only" mode), but it shouldn't be strictly necessary. 10. Click on "apply" to save the changes. 11. Reboot the machine. Note that we are not making the system less secure, since these mitigations were in audit only mode. If this workaround helps, it believe it should possible to deploy these settings across the organization (see https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-exploit-guard/import-export-exploit-protection-emet-xml ). 
 It seemed to work for us on a couple of machines I've tested on and am looking to roll this workaround out to the rest of the org. Hope this helps you guys. Cheers, Rob.