Sophos Endpoint Intercept X 2.0 impacting Performance - slow?

On a new software build of windows 10 on a T450 Lenovo, we found that at the end we installed Sophos Endpoint Intercept X 2.0 and it significantly slowed down the computer.  All aspects of the computer became slow.  On first bootup, connecting the Wifi - slow.  On login, the CPU would pin at 100% for long periods of time with high memory usage.  All applications would be slow to open, printing would be very slow. This is a new laptop i5, 8 GB RAM, 256 SSD.

We would remove the Intercept X and the computer would return to normal operation.  Fast bootup, fast login, apps, etc...

Now for this customer, then use Trend Micro as their primary AV.  We have Sophos Intercept X added on for the extra protection. We did not have issues previously until the Intercept X Version went up to 2.0.  Has anyone else noticed a large performance hit with Intercept X 2.0?

  • In reply to Sam Sarcar:

    Update
    One of the desktop admin  imaged a new desktop and installed Sophos with Intercept X


    Load desktop after entering credentials – 17 Sec
    Skype  for Business auto lunch  and log in – 37 sec.

    This device is new so it was never been in Sophos database.

    Then I did following test with me test laptop

    1)Uninstall Sophos AV
    2)Reboot the laptop and delete the device from Sophos consol.
    3)Download the installation package from Sophos console.
    4)Install Sophos AV with Intercept  X

    Now the performance seems better

    Load desktop after entering credentials  21 Sec
    Skype for Business auto lunch and Log in – 46 Sec.

    Previously

    Load desktop after entering credentials 29 Sec
    Skype for Business auto lunch and Log in – 1  min 40 Sec.


    I want to test this in few more new devices .. but Sophos might be working on the issue but they are not ready to push out to existing Client , but new installation might be getting the  fix.

  • In reply to Krystian Flemming:

    Krystian Flemming

    I'd like to ask others in this thread - are you using any other antivirus software besides Intercept X? 

    We are using the Sophos Central Endpoint Advanced AV.

    Krystian Flemming

    Could it be that Intercept X just doesn't cooperate with other AV software?  

    No, the problem also appears when using Sophos own AV.

  • Have you tried disabling Deep learning?  I recall that when we tested Invicea X (Sophos Deep Learning) on our machines last year we have experienced the same thing.  It is as if Invicea X virtualizes the whole environment so you are basically running within a VM. 

     

    How to check if Deep Learning is enabled:

     

    community.sophos.com/.../127942

  • In reply to Grammaton:

    I literally turned every single feature/policy off one by one until I had them all turned off and still had performance issues with regards to boot and login times.

     

    I've been told that our case has been raised to engineering/development now.

  • This issue seems affecting Surface devices more than others, we had to disable intercept X for several of them (Endpoint protection > Computers > Manage endpoint software) and move affected devices from Assigned to Eligible column to get back decent performances. 

  • In reply to J&P IT:

    Using Core Agent 2.0.2, Endpoint Advanced 10.8.1.1 and Intercept X 2.0.2 on 25 computers and seeing no real issues. Maybe performance is a little bit slower but nobody including me noticed a real downgrade. Deep Learning is currently not activated.

  • In reply to Jelle:

    Update, March 23 - Sophos has escalated my ticket to global escalation specialists (GES). They said I will hear from them in 1-2 weeks. Have any one has any update from your ticket?

  • In reply to Sam Sarcar:

    We have also been working very closely with GES in the UK.  We have narrowed it down to the Hitman Pro Service. Disabling that component will return the machine back to normal.  Instructions below;

    FYI, disabling HitmanPro service will disable a subset of features to stop working:

    Ransomware
    Exploit mitigations

    You will still have:

    PE file detections
    Deep Learning scanning (if enabled in policy)

     

    We have provided all logs as instructed and waiting to hear back also.

     

    1) Access the Services and stop then disable the following service:


    HitmanPro.Alert service

    2) Access the following folder:

    C:\Windows\System32\

    3) Rename hmpalert.dll to hmpalert.orig

    4) Access the following folder:

    C:\Windows\SysWOW64\

    5) Rename hmpalert.dll to hmpalert.orig

    6) Access the following folder:

    C:\Windows\System32\drivers\

    7) Rename hmpalert.sys to hmpalert.orig

    8) Reboot the computer.

  • Same issue observed on several clients (Win 10 - different brands).

    Will Sophos release a patch?

  • In reply to Stefano27383:

    Hi all,

    Faced the similarity at other customer sites, unluckily for Sophos, Microsoft has released the Spectre and Meltdown patches, and so have the hardware vendors with BIOS updates to try a protect the client system.

    Had a customers with 30 Surface 3 pro, just opening Excel would suddenly take 1-3 minutes!

    Gibson Research (Security Company), has released InSpectre, try this and disable Spectre and meltdown protection and reboot (For testing purposes), does this alter anything in performance?

    https://www.grc.com/inspectre.htm

     

  • After reading this post I wanted to test if my boot times are delayed, we had reports of slow boot times but havent been able to drill down into why.

    Machine I tested with:

    Dell Latitude E6540

    CPU: i7 3.0ghz

    RAM: 16gb

    Drive: SSD (dont know brand)

    OS: Windows 10 Pro 64bit

     

    Tests performed: 

    1. From a complete shutdown to desktop with all icons and taskbar icons showing

    2. A restart from a fully loaded desktop

     

    With Intercept X 2.0 & Core Agent 2.0

    Boot time from complete shutdown: First Attempt - 2 min 15sec. Second Attempt - 2 min 5 sec.

    Restart from desktop: First Attempt - 2 min 1 sec. Second Attempt - 2 min 7 sec

     

    Now for the shocking part.......

     

    With OUT Intercept X 2.0 & Core Agent 2.0

    Boot time from complete shutdown: First Attempt - 50 seconds. Second Attempt: 47 seconds.

    Restart from desktop: First Attempt - 52 seconds. Second Attempt: 51 seconds

     

    So it basically doubled my computers boot times, now 2 minutes is not a god awful amount of time to wait, but I could only imagine how a older machine would react to this as the machine I tested on is pretty beefed up. 

  • In reply to MatthewBosley:

    The details of the event log entries under: Applications and Services logs-Microsoft-Windows-Diagnostics-Performance/Operational might be interesting at a high level.

    Really though, I would suggest create a ETL trace of boot using Windows Performance Recorder and submit that to Sophos Support.

    Regards,

    Jak

  • In reply to jak:

    Thank you for the response Jak, I am actually in the process of capturing boot times with Windows performance analyzer. Once I have more information and data I will be opening a case with Sophos. At this time I am seeing the main cause of these machines slow boot time is from a process/service called "SophosFileScanner.exe" which according to KB 13029 is the Sophos Central Core Agent 2.0.0.

    I have also been playing with some settings with the software installed on the endpoints, by turning off services and timing boot times. If I turn of every single service, it only cuts boot time by 15 seconds. I also have been testing on a different OS, model, and specs and I am duplicating these long boot times.

    I am trying not to bash Sophos as I cannot 100% sure say its their fault as I could have some settings in my Central portal that is causing these problems, but for the life of me I cannot find what it may be from playing with settings or searching every KB I can find.

  • In reply to MatthewBosley:

    Did Sophos ever come back with a fix for this issue?

  • In reply to Peter Fraser:

    I suppose a useful and very quick test would be to make a drive exclusion for simply:

    C:

    So under the "Threat Protection" policy for the computer you're testing with set it as follows:

    Note: it says (DRIVE).

    If you remove scanning load do times improve? At least this way you know it's not just the weight of the services, the SAVService, loading virus data, etc...

    You can check the exclusion has made it in a few places at the endpoint but the value OnAccessExcludeFilePaths under:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\Scanning\Config\
    is probably the easiest.

    If it does help then in theory an exclusion or two might suffice to restore speed.  If nothing else it would be useful info to help further understand the issue.

    If it doesn't help, disable the scanning of remote files.
    If it still doesn't help disable Tamper Protection for the computer in policy.

    These 3 tests help a lot to understand the nature of the issue.

    My next test if excluding C: helps, would be try a directory exclusion for say:

    C:\windows\

    This will cover busy directories as system32, syswow64, WinSxS, Microsoft.NET, etc... and would also be useful.

    At that point it might be worth capturing just a boot Process Monitor log with a destructive filter (to improve performance and reduce the log size) for file operations on in C:\windows\.  Once collected, add the Duration column and maybe filter to just readfile, writefile operations.

    From there it might narrow it down further.  This would be a fast way to troubleshoot the issue.

    Regards,

    Jak