This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Windows Insider Build 21277.1000 - Green Screen Crash

Hi Guys,

I run the Windows Insider Preview on a PC at home and after updating to Build 21277.1000 this morning the PC began constantly crashing logging into the PC (would load to login screen fine and stay there - crash was about 30 seconds after logging in) with a familiar Green Screen stop crash and bugcheck. A quick troubleshoot and I found that its definitely being caused by Sophos Intercept X Endpoint and I have disabled it in startup.

Any ideas how to fix or do I need to wait until a new version of the Intercept X Client?

Cheers



This thread was automatically locked due to age.
Parents
  • Do you have a memory dump, either in \windows\memory.dmp or \windows\minidump\ that corresponds to these? 

    If so, can you load it into WinDbg (can get it from the Store or by downloading the SDK) and attach the contents of the window?

  • Hi there, I just wanted to chime in because I'm also seeing the same error in my environment as . Computers running the Windows 10 insider build 21277 will GSOD after login, unless 'Sophos Endpoint Agent' is disabled on startup. However, if the user launches Sophos Endpoint Agent afterwards, the computer will GSOD again.

    I've attached the contents of WinDbg's analysis of both C:\Windows\MEMORY.DMP and the dump available in C:\Windows\Minidump .

    from C:\Windows\MEMORY.DMP

    Microsoft (R) Windows Debugger Version 10.0.20153.1000 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    
    
    Loading Dump File [C:\Windows\MEMORY.DMP]
    Kernel Bitmap Dump File: Active memory is available
    
    Dump completed successfully, progress percentage: 100
    
    
    ************* Path validation summary **************
    Response                         Time (ms)     Location
    Deferred                                       srv*
    Symbol search path is: srv*
    Executable search path is: 
    Windows 10 Kernel Version 21277 MP (8 procs) Free x64
    Product: WinNt, suite: TerminalServer SingleUserTS
    Edition build lab: 21277.1000.amd64fre.rs_prerelease.201207-1443
    Machine Name:
    Kernel base = 0xfffff807`6d200000 PsLoadedModuleList = 0xfffff807`6de33b30
    Debug session time: Tue Jan  5 13:32:02.391 2021 (UTC - 8:00)
    System Uptime: 0 days 0:00:57.064
    Loading Kernel Symbols
    ...............................................................
    ................................................................
    ................................................................
    .............................
    Loading User Symbols
    .....................................
    Loading unloaded module list
    ............
    For analysis of this file, run !analyze -v
    nt!KeBugCheckEx:
    fffff807`6d619060 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff90d`ca3f7810=00000000000000ef
    4: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    CRITICAL_PROCESS_DIED (ef)
            A critical system process died
    Arguments:
    Arg1: ffffd30421ee8080, Process object or thread object
    Arg2: 0000000000000000, If this is 0, a process died. If this is 1, a thread died.
    Arg3: 0000000000000000
    Arg4: 0000000000000000
    
    Debugging Details:
    ------------------
    
    
    KEY_VALUES_STRING: 1
    
        Key  : Analysis.CPU.mSec
        Value: 3437
    
        Key  : Analysis.DebugAnalysisProvider.CPP
        Value: Create: 8007007e on DESKTOP-UK4FQDV
    
        Key  : Analysis.DebugData
        Value: CreateObject
    
        Key  : Analysis.DebugModel
        Value: CreateObject
    
        Key  : Analysis.Elapsed.mSec
        Value: 3445
    
        Key  : Analysis.Memory.CommitPeak.Mb
        Value: 93
    
        Key  : Analysis.System
        Value: CreateObject
    
        Key  : Dump.Attributes.FilterDumpFile
        Value: 1
    
        Key  : WER.OS.Branch
        Value: rs_prerelease
    
        Key  : WER.OS.Timestamp
        Value: 2020-12-07T14:43:00Z
    
        Key  : WER.OS.Version
        Value: 10.0.21277.1000
    
    
    ADDITIONAL_XML: 1
    
    OS_BUILD_LAYERS: 1
    
    DUMP_FILE_ATTRIBUTES: 0x1040
      Filter Dump
    
    BUGCHECK_CODE:  ef
    
    BUGCHECK_P1: ffffd30421ee8080
    
    BUGCHECK_P2: 0
    
    BUGCHECK_P3: 0
    
    BUGCHECK_P4: 0
    
    PROCESS_NAME:  svchost.exe
    
    CRITICAL_PROCESS:  svchost.exe
    
    ERROR_CODE: (NTSTATUS) 0x376f5080 - <Unable to get error code text>
    
    BLACKBOXBSD: 1 (!blackboxbsd)
    
    
    BLACKBOXNTFS: 1 (!blackboxntfs)
    
    
    BLACKBOXPNP: 1 (!blackboxpnp)
    
    
    BLACKBOXWINLOGON: 1
    
    STACK_TEXT:  
    fffff90d`ca3f7808 fffff807`6db6c90a     : 00000000`000000ef ffffd304`21ee8080 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx
    fffff90d`ca3f7810 fffff807`6da6e6d3     : ffffd304`21ee8080 ffffd304`28ac9378 00000000`00000000 fffff807`6d4fd3ef : nt!PspCatchCriticalBreak+0x11a
    fffff90d`ca3f78b0 fffff807`6d8f819c     : ffffd304`21ee84b8 00000000`00000000 ffffd304`21ee8080 00007ffa`5908e001 : nt!PspTerminateAllThreads+0x174c8b
    fffff90d`ca3f7920 fffff807`6d8f7110     : ffffffff`ffffffff ffffd304`2a1d30c0 ffffd304`376f5080 00000000`80000001 : nt!PspTerminateProcess+0xe0
    fffff90d`ca3f7960 fffff807`6d62b2f5     : ffffd304`000004b4 ffffd304`376f5080 ffffd304`21ee8080 ffffd304`00000000 : nt!NtTerminateProcess+0xb0
    fffff90d`ca3f79e0 00007ffa`91a97174     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
    000000d2`06bff258 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtTerminateProcess+0x14
    
    
    SYMBOL_NAME:  ntdll!NtTerminateProcess+14
    
    MODULE_NAME: ntdll
    
    IMAGE_NAME:  ntdll.dll
    
    STACK_COMMAND:  .thread ; .cxr ; kb
    
    BUCKET_ID_FUNC_OFFSET:  14
    
    FAILURE_BUCKET_ID:  0xEF_svchost.exe_BUGCHECK_CRITICAL_PROCESS_376f5080_ntdll!NtTerminateProcess
    
    OS_VERSION:  10.0.21277.1000
    
    BUILDLAB_STR:  rs_prerelease
    
    OSPLATFORM_TYPE:  x64
    
    OSNAME:  Windows 10
    
    FAILURE_ID_HASH:  {1f98d8f1-0ced-5a9a-8bb5-5da6943fc861}
    
    Followup:     MachineOwner
    ---------
    
    

    from the dump available in C:\Windows\Minidump

    Microsoft (R) Windows Debugger Version 10.0.20153.1000 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    
    
    Loading Dump File [C:\Windows\Minidump\010521-9546-01.dmp]
    Mini Kernel Dump File: Only registers and stack trace are available
    
    
    ************* Path validation summary **************
    Response                         Time (ms)     Location
    Deferred                                       srv*
    Symbol search path is: srv*
    Executable search path is: 
    Windows 10 Kernel Version 21277 MP (8 procs) Free x64
    Product: WinNt, suite: TerminalServer SingleUserTS
    Edition build lab: 21277.1000.amd64fre.rs_prerelease.201207-1443
    Machine Name:
    Kernel base = 0xfffff807`6d200000 PsLoadedModuleList = 0xfffff807`6de33b30
    Debug session time: Tue Jan  5 13:32:02.391 2021 (UTC - 8:00)
    System Uptime: 0 days 0:00:57.064
    Loading Kernel Symbols
    ...............................................................
    ................................................................
    ................................................................
    .............................
    Loading User Symbols
    Loading unloaded module list
    ............
    For analysis of this file, run !analyze -v
    nt!KeBugCheckEx:
    fffff807`6d619060 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff90d`ca3f7810=00000000000000ef
    4: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    CRITICAL_PROCESS_DIED (ef)
            A critical system process died
    Arguments:
    Arg1: ffffd30421ee8080, Process object or thread object
    Arg2: 0000000000000000, If this is 0, a process died. If this is 1, a thread died.
    Arg3: 0000000000000000
    Arg4: 0000000000000000
    
    Debugging Details:
    ------------------
    
    
    KEY_VALUES_STRING: 1
    
        Key  : Analysis.CPU.mSec
        Value: 5093
    
        Key  : Analysis.DebugAnalysisProvider.CPP
        Value: Create: 8007007e on DESKTOP-UK4FQDV
    
        Key  : Analysis.DebugData
        Value: CreateObject
    
        Key  : Analysis.DebugModel
        Value: CreateObject
    
        Key  : Analysis.Elapsed.mSec
        Value: 21744
    
        Key  : Analysis.Memory.CommitPeak.Mb
        Value: 82
    
        Key  : Analysis.System
        Value: CreateObject
    
        Key  : WER.OS.Branch
        Value: rs_prerelease
    
        Key  : WER.OS.Timestamp
        Value: 2020-12-07T14:43:00Z
    
        Key  : WER.OS.Version
        Value: 10.0.21277.1000
    
    
    ADDITIONAL_XML: 1
    
    OS_BUILD_LAYERS: 1
    
    BUGCHECK_CODE:  ef
    
    BUGCHECK_P1: ffffd30421ee8080
    
    BUGCHECK_P2: 0
    
    BUGCHECK_P3: 0
    
    BUGCHECK_P4: 0
    
    PROCESS_NAME:  svchost.exe
    
    CRITICAL_PROCESS:  svchost.exe
    
    ERROR_CODE: (NTSTATUS) 0x376f5080 - <Unable to get error code text>
    
    BLACKBOXBSD: 1 (!blackboxbsd)
    
    
    BLACKBOXNTFS: 1 (!blackboxntfs)
    
    
    BLACKBOXPNP: 1 (!blackboxpnp)
    
    
    BLACKBOXWINLOGON: 1
    
    CUSTOMER_CRASH_COUNT:  1
    
    STACK_TEXT:  
    fffff90d`ca3f7808 fffff807`6db6c90a     : 00000000`000000ef ffffd304`21ee8080 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx
    fffff90d`ca3f7810 fffff807`6da6e6d3     : ffffd304`21ee8080 ffffd304`28ac9378 00000000`00000000 fffff807`6d4fd3ef : nt!PspCatchCriticalBreak+0x11a
    fffff90d`ca3f78b0 fffff807`6d8f819c     : ffffd304`21ee84b8 00000000`00000000 ffffd304`21ee8080 00007ffa`5908e001 : nt!PspTerminateAllThreads+0x174c8b
    fffff90d`ca3f7920 fffff807`6d8f7110     : ffffffff`ffffffff ffffd304`2a1d30c0 ffffd304`376f5080 00000000`80000001 : nt!PspTerminateProcess+0xe0
    fffff90d`ca3f7960 fffff807`6d62b2f5     : ffffd304`000004b4 ffffd304`376f5080 ffffd304`21ee8080 ffffd304`00000000 : nt!NtTerminateProcess+0xb0
    fffff90d`ca3f79e0 00007ffa`91a97174     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
    000000d2`06bff258 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffa`91a97174
    
    
    SYMBOL_NAME:  nt!PspCatchCriticalBreak+11a
    
    MODULE_NAME: nt
    
    IMAGE_NAME:  ntkrnlmp.exe
    
    IMAGE_VERSION:  10.0.21277.1000
    
    STACK_COMMAND:  .thread ; .cxr ; kb
    
    BUCKET_ID_FUNC_OFFSET:  11a
    
    FAILURE_BUCKET_ID:  0xEF_svchost.exe_BUGCHECK_CRITICAL_PROCESS_376f5080_nt!PspCatchCriticalBreak
    
    OS_VERSION:  10.0.21277.1000
    
    BUILDLAB_STR:  rs_prerelease
    
    OSPLATFORM_TYPE:  x64
    
    OSNAME:  Windows 10
    
    FAILURE_ID_HASH:  {a02b1c70-8dbe-b3af-aefe-af6b2a744809}
    
    Followup:     MachineOwner
    ---------
    
    

  • When you say "unless 'Sophos Endpoint Agent' is disabled on startup", are you talking about just here:

    All this does is run the Sophos UI.exe which is the tray icon and client UI.  The key being:

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

    Sophos UI.exe ="C:\Program Files\Sophos\Sophos UI\Sophos UI.exe" /hidden

    It seems bizarre that a small user mode application can cause this.

    From the info provided, without the dump file to be able to run other commands it's hard to say much.  There is a thread in a svchost.exe process seemingly terminating a CRITICAL_PROCESS: svchost.exe. From the stack it's hard to glean the origin of the stack. 

    Currently installing Windows 10 insider build 21277 to see if I see the same.

  • Yes, that's what I disabled in the Startup pane in Task Manager. If I leave it enabled then shortly after login the computer will GSOD. Alternatively if you just launch Sophos Endpoint Agent from within the Sophos folder in the Start Menu, then that will also cause a GSOD.

    Should I be running other commands against the dump file? I'm not very experienced with performing analysis on them but I can follow instructions if you have any suggestions.

  • Apologies for the delay getting back to this - I blame the Christmas/New Years period :)

    I had to disable ALL Sophos Services from Startup via the MSConfig.exe, Selective Startup and disabling all Sophos Services there for the system to boot safely.

    System still running fine up to today, just haven't had a chance to play with it ... I may have also been playing the crap out of Watch Dogs Legion ...  :)

  • Could you leave it disabled in the startup items and disable Tamper Protection on the computer, reboot and then launch "sophos ui.exe".  Does that help?

Reply Children
  • That works, thank you! Sophos can now be part of startup without incurring GSODs. However, is it feasible to leave Tamper Protection disabled long-term? What are the ramifications of doing so?

  • Well Tamper Protection stops people (or I guess malicious processes if they've got passed the execution stage) from stopping services and general fiddling.

    I did reproduce it today and I found it quite odd.

    CRITICAL_PROCESS_DIED (ef) - "A critical system process died" is the bugcheck code.

    The critical process that is dying is the svchost process that hosts the "RPCSS" service.  I.e. 'C:\WINDOWS\system32\svchost.exe -k RPCSS -p'

    A thread in this process is the the one calling kebugcheckex.  This thread seems to be throwing an exception in RPCRT4!BCACHE::Free+0x96. The disassembly has:

    00007fff`e36c52b3 488d0daeda0a00 lea rcx,[RPCRT4!`string' (00007fff`e3772d68)]
    00007fff`e36c52ba 48ff154f9f0a00 call qword ptr [RPCRT4!_imp_DbgPrint (00007fff`e376f210)]

    The string at 00007fff`e3772d68 contains:
     "RPC: BCache corruption detected at 0x%p."

    Which might offer something and quite odd that a RPC call into RPCSS can crash it.

    The thread is servicing a RPC call from the "C:\Program Files\Sophos\Sophos UI\savapi.exe" process, that is the "client" end of the RPC call into RPCSS.

    The Sophos UI.exe process is activating the COM server that is SAVAPI.exe, which is why it appears to happen when Sophos UI.exe starts up. It's more about what savapi.exe is doing as a result as it "talks" to RPCSS.

    If you disable Tamper Protection and rename savapi.exe to savapi.exe.off, re-enable Tamper Protection I think this would also work.  I'm just surprised a user mode process, issuing a standard RPC COM call can crash RPCSS which in turn brings down the computer as it's a critical process.  

    The fact that disabling TP helps suggests that the Sophosed.sys driver might be involved as that is responsible for tamper protection and being a driver, they are the components that are usually cause such issues. 

    A user mode process on it's own can't cause a bugcheck.

    I think you might have to create a ticket with Sophos.

  • I also experienced this issue on my Insider test systems  as of 21277. Turning off Tamper Protection prior to restarting the systems allowed a normal restart. I was later able to turn Tamper Protection back on without a crash. I have reported this to Sophos, although they are of course not claiming to support 21277 at this time, which is reasonable.