Sophos Endpoint Intercept X 2.0 impacting Performance - slow?

On a new software build of windows 10 on a T450 Lenovo, we found that at the end we installed Sophos Endpoint Intercept X 2.0 and it significantly slowed down the computer.  All aspects of the computer became slow.  On first bootup, connecting the Wifi - slow.  On login, the CPU would pin at 100% for long periods of time with high memory usage.  All applications would be slow to open, printing would be very slow. This is a new laptop i5, 8 GB RAM, 256 SSD.

We would remove the Intercept X and the computer would return to normal operation.  Fast bootup, fast login, apps, etc...

Now for this customer, then use Trend Micro as their primary AV.  We have Sophos Intercept X added on for the extra protection. We did not have issues previously until the Intercept X Version went up to 2.0.  Has anyone else noticed a large performance hit with Intercept X 2.0?

  • In reply to kk20:

    Hello kk20,

    Please have a look at the following entries for more info regarding the Exploit Prevention tentative update dates:
    Sophos Anti-Virus: version release dates

    To see the current Exploit Prevention version and any known issues, please visit this link:
    Sophos Exploit Prevention


  • I have Intercept X 2.0.5, and my PC speed is good, but accessing web sites is very slow. It takes >40s after typing an address to load the web page, including known web sites such as google.com. Once a web site is loaded, getting the next page is normal. This happens everytime when loading a new url. I was only running intercept X, in combination with Norton Anivirus for now.

    At the firewall I notice many denied DNS heartbeat requests, not sure that has anything to do with it, but perhaps waiting for the DNS takes a long time that way.

    During de-install of the agent, at the point where the deinstall was deinstalling the Network Threat Protection, the speed of loading a web page became normal again.

     

  • In reply to Pieter van Kampen:

    Could it be that with Norton installed, that this has a process or processes which make network connections, e.g. maybe a local web proxy or some process that is performing cloud lookups.  Something that takes place when browsing essentially.  I'm sure this must be the case but I don't have Norton to check.

    The point of NTP is to check that processes aren't talking to malicious sites by performing cloud lookups.  If you check:

    "C:\ProgramData\Sophos\Sophos Network Threat Protection\Logs\SntpService.log"

    Do you see Norton processes mentioned a number of times when you see the issue?  Or any process for that matter during the slow down?

    You can make "file" exclusions in threat protection policy in Central and these processes will not be checked by MTD.  The exclusions end up in "C:\ProgramData\Sophos\Sophos Network Threat Protection\Config\policy.xml" to check they have made it.

    Regards,
    Jak

  • In reply to jak:

    Thanks Jak,

    the logs are no longer there after the deinstall. I have stopped the evaluation for now (already so much work to onboard the Sophos Xg). Once the licence for Norton expires later this year, I will do an evaluation of the whole stack.

    Regards

    Pieter

  • Our machines seemingly starting getting the Intercept X 2.0.5 update around mid-June. And ever since then, a good amount of our workstations started having boot delays and some booting with certain Windows services not starting.

    The only fix was to disable Intercept X on those machines.

    Any one else still having problems with Intercept X even after the 2.0.5 release?

  • In reply to Matthew Alley:

    Do you have a list of some of the services that have failed to start?

  • In reply to jak:

    Windows Services that intermittently would not start on some machines:

    - Wired AutoConfig

    - Wireless AutoConfig

    - Windows Audio

     

    The AutoConfig services are rather critical since it runs the 802.1x stack on a Windows machine, and we have 802.1x enabled in our network, so if that service doesnt start, that machine can't get on the network at all. Super pain in the butt.

     

    It seems that our machines are getting the Intercept X 2.0.5.2 update now. We didn't start seeing the problem until 2.0.5, so maybe this small version update will be helpful?

     

    I'll do some more testing.

  • In reply to Matthew Alley:

    There are a couple of main components that form part of Intercept X that might be worth toggling to see if it helps with the service startup.

    If you have Hitman Pro Alert (HMPA) then you will have the driver hmpalert.sys.  This driver injects the hmpalert dlls into processes.

    So test 1 might be to rename C:\windows\system32\drivers\hmpalert.sys and reboot.  This will stop all processes getting the hmpalert.dll dll.  This would rule this module out from causing issues with the services as they start.

    Note: You would need to disable Tamper Protection to do this.

    The next test would be to disable the sophosed.sys driver.  To do so,n rename "C:\windows\system32\drivers\sophosed.sys".

    I would then restart the computer a few times to see if with either of these drivers disabled the services start.

    Regards,

    Jak

  • In reply to jak:

    For all of these fixes, I left Intercept X enabled:

     

    Ok, so I tried a fix I saw referenced somewhere else: https://community.sophos.com/kb/en-us/27646

    That didnt work.

     

    I then tried what you suggested and renamed the hmpalert.sys file, and rebooted multiple times, and I had NO problems with services.

    I then renamed it back to normal, and had problems again.

     

    I then tried what you suggested and renamed the sophosed.sys file, and rebooted multiple times, and that didnt seem to change anything. Services still failed.

    I then renamed it back to normal, and still had problems.

     

    So what does it mean if my issue seems to go away after renaming the hmpalert.sys file? Is there a long term solution or am I stuck with disabling Intercept X on these machines that are affected?

     

    Worth noting: All of our machines that are experiencing this issue have spinning HDDs in them(supporting the boot delay idea?).  And all of our machines that have SSDs in them have not seen this problem at all.

  • In reply to Matthew Alley:

    Well this sounds like progress, the fact you can now narrow it down to the hmpa component.

    The next check I might do to narrow it down a little further is as follows:

    1. Rename the hmpalert.sys file back so the driver loads.

    2. Given the services with issues are:

    - Wired AutoConfig

    - Wireless AutoConfig

    - Windows Audio

    Then they are all 64bit processes. 

    So if you rename C:\windows\system32\hmpalert.dll to say hmpalertoff.dll then the driver will not inject the DLL.

    Note: If they were 32-bit the DLL loaded is in C:\windows\syswow64\hmpalert.dll but it would be good to keep the 32-bit process protected if you can.

    Do you still see the services fail to start then after rebooting a few times?

    I would say that not being SSD is probably the major factor of what must be a resource problem.

    Regards,
    Jak

  • In reply to jak:

    I renamed the "C:\windows\system32\hmpalert.dll" file and rebooted many times and the machine booted normally each time.

    Naming it back to it's original re-introduced the problem.

     

    I sent in to my Sophos support ticket the troubleshooting I have done and got this back:

     

    "

    Hi Matthew

    Thanks for that update, it's a very interesting solution you found and well researched.

    We do have a current known issue with servers and BSOD due to HMPA being injected at startup, but this I thought was different than you case, and why I never followed along that line of inquiry.

    "C:\Windows\System32\drivers\", renaming the driver hmpalert.sys to hmpalert.orig / old does resolve this and seems to fit somewhat with the issue we are currently pushing a fix for, but again, this is only for Servers. I will need to speak to our GES team later this morning however and then give you an update.

    Regards,

    "

  • In reply to Matthew Alley:

    OK, that narrows it down.  The next step for me would be to understand if a certain mitigation could be the cause.

    In Central the mitigations are controlled in the threat protection policy which consists of the following relevant sections:



    and ....



    The next test might be to disable all the options above and restart having re-introduced the driver and native hmpalert.dll.

    Does that help?  Hopefully it does.. I would then consider try re-enabling the options one at a time. E.g.

    "Enable CPU branch tracing"...

    "Protect processes"

    If you can find one or two options that helps, then at least you'd have a reasonable level of protection, starting services and support would have more to go on.


    The other option here to rule in/out mitigations  is to change settings locally in the registry at the client.  This gives very fine grained control of what is enabled.

    If you look in the log file of HMPA -  "C:\ProgramData\HitmanPro.Alert\Logs\Sophos.log", there is a log line for each process launched and the mitigations (features) applied. E.g.

    2018-07-04T21:40:19.329Z [Protected] PID 15384, Features 00072A3000000106, C:\Windows\System32\rundll32.exe
    2018-07-04T21:40:22.055Z [Protected] PID 17464, Features 00072A300000010E, C:\Windows\System32\svchost.exe

    The values 00072A3000000106 and 00072A300000010E represent the mitigations applied to the process.

    Under: 
    HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\
    Are the mitigation profiles applies to the certain class of applications.  Media, Java, Browsers, etc...

    You can see how under:
    HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\java.exe
    for example, the profile applied to the java process is the "Java" Profile.  So you can see which mitigations get applied.

    I assume that the svchost.exe processes you're having trouble with fall under the "Other" category: HKEY_LOCAL_MACHINE\SOFTWARE\HitmanPro.Alert\_profiles_\Other\

    All the DWORD values under this key show which mitigations apply to this category.

    One option would therefore be to set the values to 0 and see what helps.  When you toggle the registry values to control features, you do need to restart the Hitman Pro Alert service for the setting to take.  That said, if you're going to be restarting the computer however I guess it doesn't matter.   Hopefull you can see in the log file mentioned about the feature value changing.

    The only thing I might do when performing this testing is to disable the "Sophos MCS Agent" service so the client doesn't apply any policies from Central during the testing.

    I'm hoping you can then narrow it down to maybe one specific mitigations by registry value name.

    Hopefully that makes sense.

    Regards,
    Jak



  • In reply to jak:

    Turning off the features you mentioned in the dashboard for a specific machine did not seem to help. I can see on the machine in the Sophos client that the settings are disabled. And after many reboots, machine still has the same problems with the Wired and Wireless AutoConfig and Audio services not starting.

     

    I will attempt the registry modifications now.

     

    Thanks,

    Matt

  • In reply to jak:

    So we are having this same problem with the same Services not starting. Definitely seems like a performance issue on Intercept X on startup, preventing certain services from starting. Seen this on everyone getting Intercept X 2.0.5.2, and is seen on older systems without SSD's. Any fix for this problem yet?

     

    Removing Intercept X fixes the problem completely. But this isn't a solution for us, think Sophos needs to update Intercept X to address the problem asap. Big problem for us.