This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Endpoint Intercept X 2.0 impacting Performance - slow?

On a new software build of windows 10 on a T450 Lenovo, we found that at the end we installed Sophos Endpoint Intercept X 2.0 and it significantly slowed down the computer.  All aspects of the computer became slow.  On first bootup, connecting the Wifi - slow.  On login, the CPU would pin at 100% for long periods of time with high memory usage.  All applications would be slow to open, printing would be very slow. This is a new laptop i5, 8 GB RAM, 256 SSD.

We would remove the Intercept X and the computer would return to normal operation.  Fast bootup, fast login, apps, etc...

Now for this customer, then use Trend Micro as their primary AV.  We have Sophos Intercept X added on for the extra protection. We did not have issues previously until the Intercept X Version went up to 2.0.  Has anyone else noticed a large performance hit with Intercept X 2.0?




[locked by: SupportFlo at 11:42 PM (GMT -7) on 12 Mar 2019]
Parents
  • Our machines seemingly starting getting the Intercept X 2.0.5 update around mid-June. And ever since then, a good amount of our workstations started having boot delays and some booting with certain Windows services not starting.

    The only fix was to disable Intercept X on those machines.

    Any one else still having problems with Intercept X even after the 2.0.5 release?

  • Do you have a list of some of the services that have failed to start?

  • Well this sounds like progress, the fact you can now narrow it down to the hmpa component.

    The next check I might do to narrow it down a little further is as follows:

    1. Rename the hmpalert.sys file back so the driver loads.

    2. Given the services with issues are:

    - Wired AutoConfig

    - Wireless AutoConfig

    - Windows Audio

    Then they are all 64bit processes. 

    So if you rename C:\windows\system32\hmpalert.dll to say hmpalertoff.dll then the driver will not inject the DLL.

    Note: If they were 32-bit the DLL loaded is in C:\windows\syswow64\hmpalert.dll but it would be good to keep the 32-bit process protected if you can.

    Do you still see the services fail to start then after rebooting a few times?

    I would say that not being SSD is probably the major factor of what must be a resource problem.

    Regards,
    Jak

  • I renamed the "C:\windows\system32\hmpalert.dll" file and rebooted many times and the machine booted normally each time.

    Naming it back to it's original re-introduced the problem.

     

    I sent in to my Sophos support ticket the troubleshooting I have done and got this back:

     

    "

    Hi Matthew

    Thanks for that update, it's a very interesting solution you found and well researched.

    We do have a current known issue with servers and BSOD due to HMPA being injected at startup, but this I thought was different than you case, and why I never followed along that line of inquiry.

    "C:\Windows\System32\drivers\", renaming the driver hmpalert.sys to hmpalert.orig / old does resolve this and seems to fit somewhat with the issue we are currently pushing a fix for, but again, this is only for Servers. I will need to speak to our GES team later this morning however and then give you an update.

    Regards,

    "

  • OK, that narrows it down.  The next step for me would be to understand if a certain mitigation could be the cause.

    In Central the mitigations are controlled in the threat protection policy which consists of the following relevant sections:



    and ....



    The next test might be to disable all the options above and restart having re-introduced the driver and native hmpalert.dll.

    Does that help?  Hopefully it does.. I would then consider try re-enabling the options one at a time. E.g.

    "Enable CPU branch tracing"...

    "Protect processes"

    If you can find one or two options that helps, then at least you'd have a reasonable level of protection, starting services and support would have more to go on.


    The other option here to rule in/out mitigations  is to change settings locally in the registry at the client.  This gives very fine grained control of what is enabled.

    If you look in the log file of HMPA -  "C:\ProgramData\HitmanPro.Alert\Logs\Sophos.log", there is a log line for each process launched and the mitigations (features) applied. E.g.

    2018-07-04T21:40:19.329Z [Protected] PID 15384, Features 00072A3000000106, C:\Windows\System32\rundll32.exe
    2018-07-04T21:40:22.055Z [Protected] PID 17464, Features 00072A300000010E, C:\Windows\System32\svchost.exe

    The values 00072A3000000106 and 00072A300000010E represent the mitigations applied to the process.

    Under: 
    HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\
    Are the mitigation profiles applies to the certain class of applications.  Media, Java, Browsers, etc...

    You can see how under:
    HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\java.exe
    for example, the profile applied to the java process is the "Java" Profile.  So you can see which mitigations get applied.

    I assume that the svchost.exe processes you're having trouble with fall under the "Other" category: HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\Other\

    All the DWORD values under this key show which mitigations apply to this category.

    One option would therefore be to set the values to 0 and see what helps.  When you toggle the registry values to control features, you do need to restart the Hitman Pro Alert service for the setting to take.  That said, if you're going to be restarting the computer however I guess it doesn't matter.   Hopefull you can see in the log file mentioned about the feature value changing.

    The only thing I might do when performing this testing is to disable the "Sophos MCS Agent" service so the client doesn't apply any policies from Central during the testing.

    I'm hoping you can then narrow it down to maybe one specific mitigations by registry value name.

    Hopefully that makes sense.

    Regards,
    Jak



  • Turning off the features you mentioned in the dashboard for a specific machine did not seem to help. I can see on the machine in the Sophos client that the settings are disabled. And after many reboots, machine still has the same problems with the Wired and Wireless AutoConfig and Audio services not starting.

     

    I will attempt the registry modifications now.

     

    Thanks,

    Matt

  • So we are having this same problem with the same Services not starting. Definitely seems like a performance issue on Intercept X on startup, preventing certain services from starting. Seen this on everyone getting Intercept X 2.0.5.2, and is seen on older systems without SSD's. Any fix for this problem yet?

     

    Removing Intercept X fixes the problem completely. But this isn't a solution for us, think Sophos needs to update Intercept X to address the problem asap. Big problem for us.

  • We are also experiencing the same issue after the Intercept X 2.05 or 2.05.2 update (not sure which version).

    Lots of windows services getting blocked from starting at logon

     

    A timeout was reached (30000 milliseconds) while waiting for the AudioEndpointBuilder service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the Winmgmt service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the TrustedInstaller service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the EventSystem service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the gpsvc service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the ProfSvc service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the LanmanWorkstation service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the IKEEXT service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the PolicyAgent service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the WinHttpAutoProxySvc service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the FontCache service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the tzautoupdate service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the Themes service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the SysMain service to connect.
    A timeout was reached (30000 milliseconds) while waiting for the SENS service to connect.

  • Hi Ted Bundy,

    Do you have a ticket number so that we can have a look? 

    Otherwise, I suggest you reach out to Support and file a case for further review and troubleshooting.

    Also, to determine which version of Intercept X is installed, double click on the Sophos icon in the System tray --> About , and look for "Sophos Intercept X" under Products. You will see the version listed there. 

    Regards,

    Barb@Sophos
    Community Support Engineer | Sophos Technical Support
    Knowledge Base  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'This helped me' link.

     

  • In the registry, if you create a new DWORD called ServicesPipeTimeout under the key:

    HKLM\SYSTEM\CurrentControlSet\Control\

     

    and set the value to 60000 decimal, i.e. 60 seconds from the default of 30 seconds (30000 as mentioned in the email) does everything work when you reboot?

    I'm just curious if that even helps if they were given more time.
    Regards,
    Jak

     

     

     

  • Did this work for anyone with the problem?

  • Yep, i tried it on a PC and the audio service did eventually start, it made the logon extremely long & slow though. I wont be deploying this out as a fix as I would rather sopos fix the issue. But if your desperate, then it may be a workaround until sopos stops blocking.

Reply Children
  • Tried it myself and agree the system boot time is way too long. I also noticed this is only occurring on systems with spinning HD's and not systems with SSD's. They need to fix this problem asap ;)

  • I would like to move my (small) organization to Sophos Endpoint with Intercept X, but the performance with this release was too slow as described in this thread. It seems the time to resolution on this is way too long.

    Is that generally the speed of working at Sophos, or is this just complicated? If it is general I will look elsewhere.

  • We have Intercept X activated on all machines and don't see any impact. At least nobody complains about slow performance. Maybe because we use SSDs everywhere now?

    Regards, Jelle

    Sophos XG210-HA (SFOS 18.0.4) on SG210 appliances with Sandstorm and 1x AP55
    Sophos Central with Intercept X Advanced, Device Encryption, Phish Threat, Mobile Control Advanced

    If a post solves your question use the 'This helped me' link.

  • Yep if you have ssd drives you seem to be ok. But if you still have old spinning disks then look out! I agree the support is terrible. It goes to level 1 for ages, then level 2 for ages, then they acknowledge that it is a real problem & it finally goes to level 3. That's when they will fix it. They don't realise that enterprises cant work like this. It's too long to wait.

  • Hi All

    We've had a call with Sophos about this for quite some time with a lot of back and forth gathering logs. We have now received this update and workaround from them:

     

    This is a sequence of events leading to services timing out and not starting



    1. A service is launched, e.g. SysMain, with command line "c:\windows\system32\svchost.exe -k localsystemnetworkrestricted -p -s SysMain"

    2. Svchost.exe has "Code Integrity Guard" mitigation enabled in
    Audit mode. This feature comes from Windows Defender settings; there is a specific override for svchost.exe.

    2.This feature checks that the binaries loaded/injected into the process are signed by Microsoft, but does not block them, since it is in audit mode only.

    3. Hmpalert.dll (our driver) is injected into svchost.exe process by hmpalert.sys driver.

    4. Because of "Code Integrity Guard", Windows checks the signature of hmpalert.dll, using functions in code integrity dll (ci.dll).

    5. Windows checks both the embedded signature, and the signature in catalog (.cat) files – this causes ALL .cat files (thousands of files) to be opened.

    6. Catalog file check happens in the context of one of the services (whichever was started first); all other services are waiting on the first one to finish the signature check.

    7. The check can take a long time (due to opening thousands of files). In customer's procmons, the check was taking more than 1 minute.

    8. If the check takes longer than 30 seconds, the service will time out (and all other services waiting for it).




    We are now looking into how we can improve this behaviour from our side - but are you able to please test the below workaround and see if this helps:



    1. Open "Windows Defender Security Centre"

    2. Go to "App & Browser control"

    3. Scroll down and click on "Exploit protection settings"

    4. Click on "Program Settings". 

    5. Wait for a list with a bunch of executable names to pop up (could take more than 10 seconds)

    6. Scroll down until you find "svchost.exe"

    7. Press "Edit"

    8. Check if "Code Integrity Guard" is enabled in "Audit only" mode. If it is enabled, turn it off (uncheck "Override system settings").

    9. It might help to also disable "Arbitrary code guard" (if it is in "Audit only" mode), but it shouldn't be strictly necessary.

    10. Click on "apply" to save the changes. 

    11. Reboot the machine.



    Note that we are not making the system less secure, since these mitigations were in audit only mode.



    If this workaround helps, it believe it should possible to deploy these settings across the organization (see

    https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-exploit-guard/import-export-exploit-protection-emet-xml
    ).

    It seemed to work for us on a couple of machines I've tested on and am looking to roll this workaround out to the rest of the org.
    Hope this helps you guys.
    Cheers, Rob.

  • Hi,

    After a PS command to disable this, would this be correct? 

     

    Set-ProcessMitigation -Name svchost.exe -disable AuditMicrosoftSigned