Multiple PCs frozen right after update.

Over the last couple weeks, since we received the Core Agent update to 2.19.8 on 10/4, we've had multiple older machines freeze completely.  Screen freezes, no keyboard or mouse, NIC unresponsive.  We have to do a hard shut down to bring them down and back up.  Not positive that this update is the culprit, but on the computers that have been freezing 2 to 3 times a day, we uninstalled Sophos and they've been behaving for a couple days now. 

Models affected:  HP xw4400, HP xw4600, Z400.  All have been running Win10 21H1 with last update back in September.  "Newer" computers (e.g. Z420, Z4 G4) have not had this problem.  Event logs show nothing out of the ordinary around the time of crash.  

Just curious if anybody else has run into this in the last week.    



[Update - Nov 23] The following KBA has been published regarding this issue - https://support.sophos.com/support/s/article/KB-000043418?language=en_US
[edited by: Qoosh at 8:41 PM (GMT -8) on 23 Nov 2021]
  • [Update - Nov 16] This issue has been escalated and is being investigated internally under reference ID: WINEP-37251
    We will update this post with additional updates as they become available.


    Hello,

    Thank you for reaching out to the Sophos Community. 

    If you were looking to troubleshoot this issue a bit further, I'd recommend trying to perform some feature isolation to see if we can narrow down what components may be playing a part in things. 

    As it seems the issues take some time to emerge, you may want to interact with the drivers on the affected device(s). I recommend only doing one of the following components at a time and observing the device as the day goes on to see if this improves the results you’re getting.

    HMPA Isolation:
    a) Access the Services and stop then disable the following service:HitmanPro.Alert service
    b) Access the following folder: C:\Windows\System32\
    c) Rename hmpalert.dll to hmpalert.orig
    d) Access the following folder: C:\Windows\SysWOW64\
    e) Rename hmpalert.dll to hmpalert.orig
    f) Reboot the device

    SAV Isolation:
    a) Access the Services and stop then disable the following service: Sophos Anti-Virus 
    b) Reboot the device

    Sophos Endpoint Defense Isolation:
    a) Access the following folder: C:\Windows\System32\drivers\
    b) Rename SophosED.sys to SophosED.sys.orig
    c) Reboot the device

    Let me know what your findings are by updating this thread and we can advise further based on the results.

    Kushal Lakhan
    Global Community Support Engineer
    Are you a Sophos Partner? | Product Documentation | @SophosSupport | Sign up for SMS Alerts
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
  • When in that state, does this work: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/forcing-a-system-crash-with-the-power-button - It would be good to know if you can force a complete/active memory dump?

    https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/forcing-a-system-crash-from-the-keyboard is another option to force a dump but the above newer power button method might be simpler and save messing about changing keys if you don't have a scroll-lock key.

    If you can set it the computer to create a Complete or ideally Active dump to reduce the size that would be of most interest to understand what has happened.

  • Our customer has same problem with  HP Compaq 8100 Elite CMT PCs since October, and now they find that the freeze often occurs when Google Chrome is opened.

    The same, it will not freeze after removing Sophos Endpoint.

  • UPDATE 4 (11/3):  I've isolated the crash happens during the live file scans.  With just that option turned off, the computers have been running fine and so I've made a temporary policy that excludes that scan and put the problem computers in it for now, though definitely not a permanent solution.  I've left two computers out of the policy and have been testing them.  They do not produce a dump whatsoever on a crash, so I've hooked them up with kernel debugging over the network.  The only thing that shows up during a crash is that "the target machine restarted without notifying the debugger."  I'm using one of the machines currently and have been just using it to RDP into my regular PC.  It froze up 3 times in one day with nothing on the debugger or dump.  I've installed the new Sophos core update (2.20.4).  We will see how it goes.  

    UPDATE 3:  The patch did not work.  I have sent SDUs to Sophos support.  They have now asked me for a dump and component isolation as recommended here.  

    UPDATE 2:  I found when renaming SophosED.sys that the system just recreates the file.  Sophos support reached out to me with a hotfix for Hitman, so I put all the computers back to original state and installed the hotfix once the users had gone home for the day.  The next day, half of those computers crashed right away in the morning with more crashes as the day goes on.  I've been uninstalling Sophos on the affected computers again, though I'm not liking having to rely on Windows Defender.   

    UPDATE:  Had a crash with SAV disabled.  Endpoint Defense isolated computer has not crashed yet.  Computers with Sophos uninstalled completely have not crashed for over a week. 

    Still working on testing with this.  So far, after a week, it still crashed after disabling Hitman, but not when disabling SAV.  I've disabled the Endpoint Defense on one computer, but it hasn't been used enough for any conclusive results yet.  

  • I'm glad I'm not the only one, though it doesn't seem to matter whether chrome is involved or not.  So far the majority of the computers that have been crashing are just used for RDP and nothing else because of their age.     

  • Are you able to force a complete or active dump of the computer when in the hung state? Say 30 seconds after it's hung? This is the information of most use here I would say. 

  • Turned on the dump on power off, but the last couple computers that crashed with didn't dump anything.  I noticed the event log lists that the unexpected shutdown is always at the time of the freeze rather than when it's actually powered off.    

  • Maybe try the crash on ctrl scroll option, or change the keys if you don't have a scroll lock. A complete dump is the best way forward for a hang otherwise you could be trying settings for weeks if it's not reproducible. 

  • No dice with that either.  Computers are completely unresponsive and take no input.  The event log considers them shutdown in this state even when they are still technically on.

  • Our customer's computers didn't dump anything too, even keyboard force dump.