Multiple PCs frozen right after update.

[Update - Nov 16] This issue has been escalated and is being investigated internally under reference ID: WINEP-37251
We will update this post with additional updates as they become available.

Hello,

Thank you for reaching out to the Sophos Community. 

If you were looking to troubleshoot this issue a bit further, I'd recommend trying to perform some feature isolation to see if we can narrow down what components may be playing a part in things. 

As it seems the issues take some time to emerge, you may want to interact with the drivers on the affected device(s). I recommend only doing one of the following components at a time and observing the device as the day goes on to see if this improves the results you’re getting.

HMPA Isolation:
a) Access the Services and stop then disable the following service:HitmanPro.Alert service
b) Access the following folder: C:\Windows\System32\
c) Rename hmpalert.dll to hmpalert.orig
d) Access the following folder: C:\Windows\SysWOW64\
e) Rename hmpalert.dll to hmpalert.orig
f) Reboot the device

SAV Isolation:
a) Access the Services and stop then disable the following service: Sophos Anti-Virus 
b) Reboot the device

Sophos Endpoint Defense Isolation:
a) Access the following folder: C:\Windows\System32\drivers\
b) Rename SophosED.sys to SophosED.sys.orig
c) Reboot the device

Let me know what your findings are by updating this thread and we can advise further based on the results.

[Update - Nov 16] This issue has been escalated and is being investigated internally under reference ID: WINEP-37251
We will update this post with additional updates as they become available.

Hello,

Thank you for reaching out to the Sophos Community. 

If you were looking to troubleshoot this issue a bit further, I'd recommend trying to perform some feature isolation to see if we can narrow down what components may be playing a part in things. 

As it seems the issues take some time to emerge, you may want to interact with the drivers on the affected device(s). I recommend only doing one of the following components at a time and observing the device as the day goes on to see if this improves the results you’re getting.

HMPA Isolation:
a) Access the Services and stop then disable the following service:HitmanPro.Alert service
b) Access the following folder: C:\Windows\System32\
c) Rename hmpalert.dll to hmpalert.orig
d) Access the following folder: C:\Windows\SysWOW64\
e) Rename hmpalert.dll to hmpalert.orig
f) Reboot the device

SAV Isolation:
a) Access the Services and stop then disable the following service: Sophos Anti-Virus 
b) Reboot the device

Sophos Endpoint Defense Isolation:
a) Access the following folder: C:\Windows\System32\drivers\
b) Rename SophosED.sys to SophosED.sys.orig
c) Reboot the device

Let me know what your findings are by updating this thread and we can advise further based on the results.

UPDATE 4 (11/3):  I've isolated the crash happens during the live file scans.  With just that option turned off, the computers have been running fine and so I've made a temporary policy that excludes that scan and put the problem computers in it for now, though definitely not a permanent solution.  I've left two computers out of the policy and have been testing them.  They do not produce a dump whatsoever on a crash, so I've hooked them up with kernel debugging over the network.  The only thing that shows up during a crash is that "the target machine restarted without notifying the debugger."  I'm using one of the machines currently and have been just using it to RDP into my regular PC.  It froze up 3 times in one day with nothing on the debugger or dump.  I've installed the new Sophos core update (2.20.4).  We will see how it goes.  

UPDATE 3:  The patch did not work.  I have sent SDUs to Sophos support.  They have now asked me for a dump and component isolation as recommended here.  

UPDATE 2:  I found when renaming SophosED.sys that the system just recreates the file.  Sophos support reached out to me with a hotfix for Hitman, so I put all the computers back to original state and installed the hotfix once the users had gone home for the day.  The next day, half of those computers crashed right away in the morning with more crashes as the day goes on.  I've been uninstalling Sophos on the affected computers again, though I'm not liking having to rely on Windows Defender.   

UPDATE:  Had a crash with SAV disabled.  Endpoint Defense isolated computer has not crashed yet.  Computers with Sophos uninstalled completely have not crashed for over a week. 

Still working on testing with this.  So far, after a week, it still crashed after disabling Hitman, but not when disabling SAV.  I've disabled the Endpoint Defense on one computer, but it hasn't been used enough for any conclusive results yet.  

So far with no peripherals, I've been crash-free for 50 hours.  Previously was crashing twice a day.   If good all day, I'll plug peripherals back in tomorrow morning and see if it crashes again.  If it does, I'll use PS2 rather than USB connectors and see if that will make a difference.    

Finally getting somewhere.  Plugged the USB mouse/keyboard into the crashing machine while I Teamviewered into it via laptop, then ran RDP from the crasher to my work computer.  This way I could test mouse and keyboard without monitor hookup.  Within 1 to 2 hours, it crashed.  Now trying PS2 mouse and keyboard to rule things out a little further.   Anybody else testing, can try disabling their USB monitoring to see if that fixes anything also. 

Thank you for the feedback, I will be sharing this information with our Support teams to further their investigation. 

As a temporary workaround, could you try disabling the following protection feature on the affected devices? Let me know if this prevents the system crashes from occurring.

I am following cases internally and will follow up here as more information becomes available. 

Hi Qoosh

We have tried to disable "Detect malicious behavior" but still crash. 

Thank you for sharing your findings. I will let you know here if a fix or another workaround is found.

We have found Disabling Endpoint Defense by stopping service and renaming the .sys file corrects the lockup problem but the HitmanPro and A-V settings are irrelevant.

trying to execute a memory dump does not work with keyboard or power switch as per the MS regkeys.

Still crashing with PS2 connection.  Here's what I have found so far.  
1.  All connections (monitor, KB, mouse, power, NIC), PC full use, full Sophos:  Crash twice a day around noon and 5 p.m.
2.  All connections (monitor, KB, mouse, power, NIC), PC full use, Live file scanning turned off  in Sophos:  No crash.
3.  No connections except power and NIC, PC idle:  No crash - tested 1 week+
4.  All connections, PC as RDP client:  Crash twice a day same times as #1. 
5.  No connections except power and NIC, PC as RDP client:  No crash - tested 50 hours+
6.  No monitor connection, connected by USB mouse/KB, NIC, power, PC as RDP client:  Crash as #1.
7.  Connections (PS2 KB, mouse, monitor, NIC, power), PC as RDP client:  Crash. 

All in all, it seems like having the keyboard and mouse directly connected is causing the issue.  I can try connecting one or the other and narrow it down further, but I would think Sophos would have something worked out by now.  Don't they have a detailed programming change log from the bad update?  Something that they can look at and say, "hey, we changed this feature and now computers are crashing when peripherals are connected, lets roll it back and re-update."  It's been over a month now.   

What USB monitoring setting are you talking about?  If under peripheral control, I have that disabled as is.  If there is something else I am missing, let me know and I will try it.

VMAN - Your response lacks information and is kind of confusing when you say "but the hitmanpro and AV settings are irrelevant".  Can you elaborate on what you are trying to say?  And what .sys file are you talking about and where is it located?