Multiple PCs frozen right after update.

Massive update from Sophos. I have been pushing Sophos support, manager & escalation.

Apparently they have made changes with the threat control policy and how they work.

Sophos have made this change today and all should start to work better now. with no client updates needed.

We are working past our freeze window of about 3 hours and almost into our 4th hour.

Gotta say, I went the whole day without crashing on my test machine.  About time.  I'd like to know exactly what the cause was.  

Yup, me too on the cause. I pestered Sophos on various issues related to what I saw happening on the process level and got NO response from the "rep" handling my case and was passed from one case handler to another. Guess I was too much of a hot potato to deal with. Devil's in the details!! Keep pestering Sophos for the cause and maybe we'll get an answer, but it might be too deep in the process level to explain or gives away trade secrets on how they deal with threats. Been running since 13:34 PST and now is 18:19 PST w/o a freeze. Hope it holds and they haven't pushed the bubble to some other corner of the S/W. Hope others are having the same good results.

Hi Ronald, Maybe you can test my theory below? Thanks.

Will give it a try. "Process Hacker"?  My audio is on the MoBo, no separate card.  I see audiodg in the Resource Monitor but not Task Manager. Not sure how to get a dump of audiodg.

I also played music continuously on 3 different PCs since 3PM until this morning. No freezing of PCs.

USER930...  OK...Found the advanced tab of TM and tried the dump of audiodg while music playing (VLC media Player.....  Got the msg,  "Pls wait while process written to file, then the buzzing on speakers and freeze. These are on tower PCs not laptops. I never had laptop hangs when playing music, just the towers, EXCEPT the newer Optiplex 5090 desktop, which I thought, Verrrrry eeeenteresting".  Old audio chip issues???

I guess that proves it. Reading the memory of audiodg.exe process while it is playing audio is the trigger. I reproduced this on a HP Compaq 6000 with on board sound. Interestingly if I disable the on board sound card and use a USB one it’s fine. I suspected maybe a third party module in audiodg for the card but Process Explorer doesn’t suggest this is the case. All Microsoft modules. 

All MS modules, you mean the audiodg is MS module or?   So I wonder if MS did something too since May, when I was able to play music. I don't have any BUs for Sept to isolate MS or Sophos. Did the dump on audiodg in the new Opt 5090 desktop and got a momentary interruption (buzz), then it recovered after the dump completed.  I have a March '21 BU disk and I'll see if I can get to MS present baseline w/o Sophos update to see what happens and test the audiodg dump on the Mar '21 baseline BU disk before I do any updates.

All MS modules, you mean the audiodg is MS module or?   So I wonder if MS did something too since May, when I was able to play music. I don't have any BUs for Sept to isolate MS or Sophos. Did the dump on audiodg in the new Opt 5090 desktop and got a momentary interruption (buzz), then it recovered after the dump completed.  I have a March '21 BU disk and I'll see if I can get to MS present baseline w/o Sophos update to see what happens and test the audiodg dump on the Mar '21 baseline BU disk before I do any updates.

Reading What is AUDIODG.EXE? | Microsoft Docs as a bit of background, it has this statement:

The first is that there's 3rd party code that gets loaded into audiodg.exe.  Audio hardware vendors have the ability to install custom DSPs (called Audio Processing Objects or APOs) into the audio pipeline.

So it would suggest that third party non-Microsoft DLLs can be loaded into it.  But if I look on my computer with the problem, all the modules are Microsoft, it's not like there is some fancy third party audio processing going on.

I believe a quick hang is fine when you initiate a dump of the process, as that is the process being suspended to take the dump but it recovers in a fraction of a second.  The machines suffering from this issue just buzz and hang I suspect.

What is really frustrating is you can't crash the computer to get a dump file when hung.  I've tried keyboard (crash on ctrl scroll, power button holds) and remote kernel debugging over serial cable and the network interface. It's not possible to break into the machine.

SO is Sophos taking a dump of exe and dll for virus scanning? This wasn't a problem back in Mar/May 2021. I see audiodg.exe has a file date stamp of 13 Oct 2021 and was updated 9 Nov 2021, but who updated it?  Will check other BUs and see what I find. When I run the Win 10 32 bit PC, there is no freeze!

We did not use Audio to create our issue.

Our users all used Amazon AWS Workspaces (VDI's) and always just over 3 hours into user turn on and use of their systems and the connection it would freeze the system.

Yes we have the same hardware but why on the 3 hour mark?

 I believe the background memory scan runs every 3 hours, relative to the SSPService starting.

Interesting, but the timer would only start when I started to play music on my systems. Other wise I had no freezing. It seems if Sophos scanned the audiodg.exe 3 hrs after it started, that would explain the freeze. I tried to test things by stopping the music before the 3 hr mark, then start playing a little while later and it would freeze well before the next 3 hr time. Wonder if Sophos kept a log of exes that had run and the time of those starting. I was going to test things by waiting for more than 3 hrs and then starting the music, but never got around to it. Key issue is audiodg.exe, but what did Sophos do to stop the system from freezing? Did it keep some exes from being scanned? Where would that list of exceptions be kept?  Any comments back from Sophos, although it is the weekend...

nice work/analysis users!!!

I wonder what USER5115 had running on their systems that hung the system? Mine only hung when music played. Didn't see any comments about items/apps that were playing that caused freezes. Concerned now about hackers identifying  what exes are on the exception list and corrupting those exes.

No music or sounds caused it for me.  PCs were for the most part only used as dummy remote desktop clients, particularly the one I was testing on.   I usually leave it on all night and some days it would already be frozen when I got into work.  The only time I could get it to not crash was to not have anything plugged into it.  No crashes Friday though.  I'll give it a few days next week to make sure before I start turning everyone's policy back on.  

Network driver files got scanned?? causing a freeze.  I did check the two audio files, audiodg.exe, one on a 3/12/21 HDD and the other on the present HDD 11/20/21 and using Beyond Compare I compared binary wise the two versions. They were different and different vs # and dates, old one was 10.0.19041.804 digital date/size 1/28/21 585,224 bytes, the other 10.0.19041.1320   date/size 10/13/21  585,240 bytes   What sys files were used in your network connections? I have Intel 82567LM-3

So far so good, but did get a momentary 1/10th sec buzz when playing music on Youtube every now and then. Haven't been able to pin point the process running that causes it or why it's picking on Youtube.