Multiple PCs frozen right after update.

Over the last couple weeks, since we received the Core Agent update to 2.19.8 on 10/4, we've had multiple older machines freeze completely.  Screen freezes, no keyboard or mouse, NIC unresponsive.  We have to do a hard shut down to bring them down and back up.  Not positive that this update is the culprit, but on the computers that have been freezing 2 to 3 times a day, we uninstalled Sophos and they've been behaving for a couple days now. 

Models affected:  HP xw4400, HP xw4600, Z400.  All have been running Win10 21H1 with last update back in September.  "Newer" computers (e.g. Z420, Z4 G4) have not had this problem.  Event logs show nothing out of the ordinary around the time of crash.  

Just curious if anybody else has run into this in the last week.    



[Update - Nov 23] The following KBA has been published regarding this issue - https://support.sophos.com/support/s/article/KB-000043418?language=en_US
[edited by: Qoosh at 8:41 PM (GMT -8) on 23 Nov 2021]
Parents
  • [Update - Nov 16] This issue has been escalated and is being investigated internally under reference ID: WINEP-37251
    We will update this post with additional updates as they become available.


    Hello,

    Thank you for reaching out to the Sophos Community. 

    If you were looking to troubleshoot this issue a bit further, I'd recommend trying to perform some feature isolation to see if we can narrow down what components may be playing a part in things. 

    As it seems the issues take some time to emerge, you may want to interact with the drivers on the affected device(s). I recommend only doing one of the following components at a time and observing the device as the day goes on to see if this improves the results you’re getting.

    HMPA Isolation:
    a) Access the Services and stop then disable the following service:HitmanPro.Alert service
    b) Access the following folder: C:\Windows\System32\
    c) Rename hmpalert.dll to hmpalert.orig
    d) Access the following folder: C:\Windows\SysWOW64\
    e) Rename hmpalert.dll to hmpalert.orig
    f) Reboot the device

    SAV Isolation:
    a) Access the Services and stop then disable the following service: Sophos Anti-Virus 
    b) Reboot the device

    Sophos Endpoint Defense Isolation:
    a) Access the following folder: C:\Windows\System32\drivers\
    b) Rename SophosED.sys to SophosED.sys.orig
    c) Reboot the device

    Let me know what your findings are by updating this thread and we can advise further based on the results.

    Kushal Lakhan
    Global Community Support Engineer
    Are you a Sophos Partner? | Product Documentation | @SophosSupport | Sign up for SMS Alerts
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
  • UPDATE 4 (11/3):  I've isolated the crash happens during the live file scans.  With just that option turned off, the computers have been running fine and so I've made a temporary policy that excludes that scan and put the problem computers in it for now, though definitely not a permanent solution.  I've left two computers out of the policy and have been testing them.  They do not produce a dump whatsoever on a crash, so I've hooked them up with kernel debugging over the network.  The only thing that shows up during a crash is that "the target machine restarted without notifying the debugger."  I'm using one of the machines currently and have been just using it to RDP into my regular PC.  It froze up 3 times in one day with nothing on the debugger or dump.  I've installed the new Sophos core update (2.20.4).  We will see how it goes.  

    UPDATE 3:  The patch did not work.  I have sent SDUs to Sophos support.  They have now asked me for a dump and component isolation as recommended here.  

    UPDATE 2:  I found when renaming SophosED.sys that the system just recreates the file.  Sophos support reached out to me with a hotfix for Hitman, so I put all the computers back to original state and installed the hotfix once the users had gone home for the day.  The next day, half of those computers crashed right away in the morning with more crashes as the day goes on.  I've been uninstalling Sophos on the affected computers again, though I'm not liking having to rely on Windows Defender.   

    UPDATE:  Had a crash with SAV disabled.  Endpoint Defense isolated computer has not crashed yet.  Computers with Sophos uninstalled completely have not crashed for over a week. 

    Still working on testing with this.  So far, after a week, it still crashed after disabling Hitman, but not when disabling SAV.  I've disabled the Endpoint Defense on one computer, but it hasn't been used enough for any conclusive results yet.  

Reply
  • UPDATE 4 (11/3):  I've isolated the crash happens during the live file scans.  With just that option turned off, the computers have been running fine and so I've made a temporary policy that excludes that scan and put the problem computers in it for now, though definitely not a permanent solution.  I've left two computers out of the policy and have been testing them.  They do not produce a dump whatsoever on a crash, so I've hooked them up with kernel debugging over the network.  The only thing that shows up during a crash is that "the target machine restarted without notifying the debugger."  I'm using one of the machines currently and have been just using it to RDP into my regular PC.  It froze up 3 times in one day with nothing on the debugger or dump.  I've installed the new Sophos core update (2.20.4).  We will see how it goes.  

    UPDATE 3:  The patch did not work.  I have sent SDUs to Sophos support.  They have now asked me for a dump and component isolation as recommended here.  

    UPDATE 2:  I found when renaming SophosED.sys that the system just recreates the file.  Sophos support reached out to me with a hotfix for Hitman, so I put all the computers back to original state and installed the hotfix once the users had gone home for the day.  The next day, half of those computers crashed right away in the morning with more crashes as the day goes on.  I've been uninstalling Sophos on the affected computers again, though I'm not liking having to rely on Windows Defender.   

    UPDATE:  Had a crash with SAV disabled.  Endpoint Defense isolated computer has not crashed yet.  Computers with Sophos uninstalled completely have not crashed for over a week. 

    Still working on testing with this.  So far, after a week, it still crashed after disabling Hitman, but not when disabling SAV.  I've disabled the Endpoint Defense on one computer, but it hasn't been used enough for any conclusive results yet.  

Children
  • Finally getting somewhere.  Plugged the USB mouse/keyboard into the crashing machine while I Teamviewered into it via laptop, then ran RDP from the crasher to my work computer.  This way I could test mouse and keyboard without monitor hookup.  Within 1 to 2 hours, it crashed.  Now trying PS2 mouse and keyboard to rule things out a little further.   Anybody else testing, can try disabling their USB monitoring to see if that fixes anything also. 

  • We have found Disabling Endpoint Defense by stopping service and renaming the .sys file corrects the lockup problem but the HitmanPro and A-V settings are irrelevant.

    trying to execute a memory dump does not work with keyboard or power switch as per the MS regkeys.

  • What USB monitoring setting are you talking about?  If under peripheral control, I have that disabled as is.  If there is something else I am missing, let me know and I will try it.

  • VMAN - Your response lacks information and is kind of confusing when you say "but the hitmanpro and AV settings are irrelevant".  Can you elaborate on what you are trying to say?  And what .sys file are you talking about and where is it located?

  • We have it set to monitor USB devices when they are connected in the policy.  Don't worry about it now, I'm crashing with PS2 connections, so it's not USB.  It seems to be something to do with the keyboard and/or mouse directly connected, regardless of connection type.  Though I could be wrong and there's something else weird that goes on that I didn't trigger when I did my no-peripheral-test through Teamviewer.   

  • I guess they are referring to SophosED.sys which is the file system filter driver which is part of the "Sophos Endpoint Defense" component.  It sounds like disabling this, by renaming it and rebooting prevents the issue (maybe also stop/disable the Sophos AutoUpdate Service) to prevent it being re-instated).  I guess they concluded that if this SED driver is related, the HMPA component is not related.  Not totally convinced by that as there is some cross communication. I guess, they are also suggesting, that by not changing the threat protection policy between disabling the SophosED.sys driver, this suggests that these are not related but much of the threat protection policy configures the SophosED.sys driver.  I think I would take from his comment that SophosED.sys could be related, so you could disable that.  Doing so does disable an awful lot of the product so it's hard to say what that means.

  • Hi there,

    we have a very similar problem with the sophos endpoint on Win10 HP Z400 workstations. We have a freeze about once a week.

    We have also opened a support ticket and have already sent SDU data, but received no real help.
    We are supposed to do a memory dump, which is not possible with a freeze. You don't get any helpful answers from Sophos to direct queries, only "Standard - have you already tried this or do the following .." Yes, we did, but it doesn't work :-(

    Now we have also deactivated the real-time scanner on two computers - it seems to work. But that cannot be the solution.

    Are there any other practical solutions here in the Community?
    Thank you very much!

    P.S. Sorry for the "bad" english via google translate.