Server 2012 R2 BSOD on 20 + Servers, any thoughts?

Do you have a memory dump from any of the computers, full, kernel or mini might do.

Regards,

Jak

Hello,

I sure did, each and every server has the same (or very simular) bug check...

060117-31187-01.dmp 01/06/2017

13:10:11 SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

0x1000007e ffffffff`c0000005 fffff801`6feebbcb ffffd000`250fe218 ffffd000`250fda20 

x64      

C:\Windows\Minidump\060117-31187-01.dmp

 24 15 9600 287,696 01/06/2017 12:18:17 

Thanks

rob

Are you able to share out the minidump, it would be good to see the stack of the crashing thread?

Or rather than sharing the file, you could submit it to:
http://www.osronline.com/page.cfm?name=Analyze

and post the output.  The crashing stack that ended in the bugcheck should be enough I would think.

Regards,

Jak

Hello,

First, thanks for helping....

I have had a look at the mini dump but its 4GB... how can i get just the part you need?

Thanks

rob

Essentially you need to install Windbg.exe, configure symbols, load the dump and run: !analyze -v

Windbg is part of the Windows SDK.  https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk

You only need to install Debugging Tools for Windows to get Windbg.exe.

Once installed, run then in the menu you can configure Symbols, the following line will do:

SRV*c:\symbols*msdl.microsoft.com/.../symbols

After than is configured you can run

!analyze -v

I'd be interested to see the crashing stack from that.

Regards,

Jak

Here you go, thanks once again!

Microsoft (R) Windows Debugger Version 10.0.15063.400 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.

Symbol search path is: srv*
Executable search path is:
Windows 8.1 Kernel Version 9600 MP (24 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 9600.18505.amd64fre.winblue_ltsb.160930-0600
Machine Name:
Kernel base = 0xfffff800`cfc8d000 PsLoadedModuleList = 0xfffff800`cff60630
Debug session time: Thu Jun  1 13:10:11.165 2017 (UTC + 1:00)
System Uptime: 98 days 23:02:12.921
Loading Kernel Symbols
...............................................................
................................................................
................
Loading User Symbols

Loading unloaded module list
.............
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 7E, {ffffffffc0000005, fffff8016feebbcb, ffffd000250fe218, ffffd000250fda20}

*** ERROR: Module load completed but symbols could not be loaded for hmpalert.sys
Probably caused by : hmpalert.sys ( hmpalert+1478a )

Followup:     MachineOwner
---------

6: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff8016feebbcb, The address that the exception occurred at
Arg3: ffffd000250fe218, Exception Record Address
Arg4: ffffd000250fda20, Context Record Address

Debugging Details:
------------------

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING:  6.3.9600.18505 (winblue_ltsb.160930-0600)

SYSTEM_MANUFACTURER:  HP

SYSTEM_PRODUCT_NAME:  ProLiant DL380 Gen9

SYSTEM_SKU:  K8P38A

BIOS_VENDOR:  HP

BIOS_VERSION:  P89

BIOS_DATE:  03/05/2015

DUMP_TYPE:  1

BUGCHECK_P1: ffffffffc0000005

BUGCHECK_P2: fffff8016feebbcb

BUGCHECK_P3: ffffd000250fe218

BUGCHECK_P4: ffffd000250fda20

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP:
fltmgr!FltpCreateFile+c7
fffff801`6feebbcb 4c8b4b20        mov     r9,qword ptr [rbx+20h]

EXCEPTION_RECORD:  ffffd000250fe218 -- (.exr 0xffffd000250fe218)
ExceptionAddress: fffff8016feebbcb (fltmgr!FltpCreateFile+0x00000000000000c7)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000020
Attempt to read from address 0000000000000020

CONTEXT:  ffffd000250fda20 -- (.cxr 0xffffd000250fda20)
rax=ffffd000250fe500 rbx=0000000000000000 rcx=ffffd000250fe710
rdx=0000000000000000 rsi=ffffd000250fe6f8 rdi=ffffe0015abefa00
rip=fffff8016feebbcb rsp=ffffd000250fe450 rbp=ffffd000250fe629
 r8=0000000000000000  r9=0000000000000000 r10=fffff78000000008
r11=ffffd000250fe720 r12=0000000000000000 r13=0000000000000000
r14=ffffd000250fe708 r15=ffffe0015abefa00
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
fltmgr!FltpCreateFile+0xc7:
fffff801`6feebbcb 4c8b4b20        mov     r9,qword ptr [rbx+20h] ds:002b:00000000`00000020=????????????????
Resetting default scope

CPU_COUNT: 18

CPU_MHZ: 95d

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3f

CPU_STEPPING: 2

CPU_MICROCODE: 6,3f,2,0 (F,M,S,R)  SIG: 2B'00000000 (cache) 2B'00000000 (init)

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000020

FOLLOWUP_IP:
hmpalert+1478a
fffff801`717c478a 89842480000000  mov     dword ptr [rsp+80h],eax

BUGCHECK_STR:  AV

READ_ADDRESS:  0000000000000020

DEFAULT_BUCKET_ID:  NULL_CLASS_PTR_DEREFERENCE

ANALYSIS_SESSION_HOST:  MAN-APPASURE

ANALYSIS_SESSION_TIME:  06-02-2017 14:29:42.0079

ANALYSIS_VERSION: 10.0.15063.400 amd64fre

LAST_CONTROL_TRANSFER:  from fffff8016feebec8 to fffff8016feebbcb

STACK_TEXT: 
ffffd000`250fe450 fffff801`6feebec8 : ffffe001`590f28d0 ffffd000`250fe710 ffffd000`250fe708 ffffd000`250fe6f8 : fltmgr!FltpCreateFile+0xc7
ffffd000`250fe550 fffff801`717c478a : 00000000`00000000 ffffe001`590f28d0 ffffd000`250fe708 ffffd000`250fe6f8 : fltmgr!FltCreateFileEx2+0xd0
ffffd000`250fe670 fffff801`717b982d : 00000000`00000000 ffffe001`5abefa00 00000000`00000000 ffffc002`06eee160 : hmpalert+0x1478a
ffffd000`250fe780 fffff801`717b833e : ffffc002`06eee160 fffff800`00000000 ffffe801`79426c00 fffff801`00000000 : hmpalert+0x982d
ffffd000`250fe7d0 fffff801`717b6278 : ffffe001`590f28d0 ffffe001`590f28d0 ffffe801`79426c00 fffff801`717cdbae : hmpalert+0x833e
ffffd000`250fe820 fffff801`717b5fdc : ffffe001`0000000b ffffe001`53561880 00000000`00000000 00000000`00000001 : hmpalert+0x6278
ffffd000`250fe860 fffff801`717c145d : 00000000`44525652 00000000`00000000 ffffc001`f9252010 fffff801`717cf748 : hmpalert+0x5fdc
ffffd000`250fe890 fffff801`6ff04024 : ffffe001`5f1f4620 ffffe001`590f28d0 ffffe001`73cd8880 00000000`00000001 : hmpalert+0x1145d
ffffd000`250fe8e0 fffff801`6ff04268 : 00000000`00000002 fffff800`cff39240 ffffd000`2715d601 ffff6d7c`bd269193 : fltmgr!FltpDoUnloadFilter+0x16c
ffffd000`250fead0 fffff800`d020f160 : fffff800`d0138598 ffffe001`73cd8880 ffffd000`2715d680 ffffd000`2715d680 : fltmgr!FltpMiniFilterDriverUnload+0xfc
ffffd000`250feb10 fffff800`cfccbd6f : fffff800`cfd57100 ffffe001`73cd89c0 fffff800`d0138598 ffffe001`6121b9c8 : nt! ?? ::NNGAKEGL::`string'+0x610d0
ffffd000`250feb50 fffff800`cfcbdf34 : 00000000`00000000 ffffe001`73cd8880 00000000`00000080 ffffe001`73cd8880 : nt!ExpWorkerThread+0x69f
ffffd000`250fec00 fffff800`cfde19c6 : ffffd000`9b73a180 ffffe001`73cd8880 ffffe001`52daf780 ffffd000`250fed90 : nt!PspSystemThreadStartup+0x58
ffffd000`250fec60 00000000`00000000 : ffffd000`250ff000 ffffd000`250f9000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

THREAD_SHA1_HASH_MOD_FUNC:  b3e45fe0d5bf853419ada422084b0acbe084b639

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  78287a7eeaba0bc01a240dab78e8d766333aae63

THREAD_SHA1_HASH_MOD:  0af9beb6c879a65f84b8796640a5b54de50b621a

FAULT_INSTR_CODE:  80248489

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  hmpalert+1478a

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: hmpalert

IMAGE_NAME:  hmpalert.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5899cd41

STACK_COMMAND:  .cxr 0xffffd000250fda20 ; kb

BUCKET_ID_FUNC_OFFSET:  1478a

FAILURE_BUCKET_ID:  AV_hmpalert!unknown_function

BUCKET_ID:  AV_hmpalert!unknown_function

PRIMARY_PROBLEM_CLASS:  AV_hmpalert!unknown_function

TARGET_TIME:  2017-06-01T12:10:11.000Z

OSBUILD:  9600

OSSERVICEPACK:  18505

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8.1

OSEDITION:  Windows 8.1 Server TerminalServer SingleUserTS

OS_LOCALE: 

USER_LCID:  0

OSBUILD_TIMESTAMP:  2016-09-30 15:31:28

BUILDDATESTAMP_STR:  160930-0600

BUILDLAB_STR:  winblue_ltsb

BUILDOSVER_STR:  6.3.9600.18505

ANALYSIS_SESSION_ELAPSED_TIME:  975

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_hmpalert!unknown_function

FAILURE_ID_HASH:  {9453d9a1-2fd9-1a8d-d742-01cc77f6eeb0}

Followup:     MachineOwner
---------

Here you go, thanks once again!

Microsoft (R) Windows Debugger Version 10.0.15063.400 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.

Symbol search path is: srv*
Executable search path is:
Windows 8.1 Kernel Version 9600 MP (24 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 9600.18505.amd64fre.winblue_ltsb.160930-0600
Machine Name:
Kernel base = 0xfffff800`cfc8d000 PsLoadedModuleList = 0xfffff800`cff60630
Debug session time: Thu Jun  1 13:10:11.165 2017 (UTC + 1:00)
System Uptime: 98 days 23:02:12.921
Loading Kernel Symbols
...............................................................
................................................................
................
Loading User Symbols

Loading unloaded module list
.............
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 7E, {ffffffffc0000005, fffff8016feebbcb, ffffd000250fe218, ffffd000250fda20}

*** ERROR: Module load completed but symbols could not be loaded for hmpalert.sys
Probably caused by : hmpalert.sys ( hmpalert+1478a )

Followup:     MachineOwner
---------

6: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff8016feebbcb, The address that the exception occurred at
Arg3: ffffd000250fe218, Exception Record Address
Arg4: ffffd000250fda20, Context Record Address

Debugging Details:
------------------

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING:  6.3.9600.18505 (winblue_ltsb.160930-0600)

SYSTEM_MANUFACTURER:  HP

SYSTEM_PRODUCT_NAME:  ProLiant DL380 Gen9

SYSTEM_SKU:  K8P38A

BIOS_VENDOR:  HP

BIOS_VERSION:  P89

BIOS_DATE:  03/05/2015

DUMP_TYPE:  1

BUGCHECK_P1: ffffffffc0000005

BUGCHECK_P2: fffff8016feebbcb

BUGCHECK_P3: ffffd000250fe218

BUGCHECK_P4: ffffd000250fda20

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP:
fltmgr!FltpCreateFile+c7
fffff801`6feebbcb 4c8b4b20        mov     r9,qword ptr [rbx+20h]

EXCEPTION_RECORD:  ffffd000250fe218 -- (.exr 0xffffd000250fe218)
ExceptionAddress: fffff8016feebbcb (fltmgr!FltpCreateFile+0x00000000000000c7)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000020
Attempt to read from address 0000000000000020

CONTEXT:  ffffd000250fda20 -- (.cxr 0xffffd000250fda20)
rax=ffffd000250fe500 rbx=0000000000000000 rcx=ffffd000250fe710
rdx=0000000000000000 rsi=ffffd000250fe6f8 rdi=ffffe0015abefa00
rip=fffff8016feebbcb rsp=ffffd000250fe450 rbp=ffffd000250fe629
 r8=0000000000000000  r9=0000000000000000 r10=fffff78000000008
r11=ffffd000250fe720 r12=0000000000000000 r13=0000000000000000
r14=ffffd000250fe708 r15=ffffe0015abefa00
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
fltmgr!FltpCreateFile+0xc7:
fffff801`6feebbcb 4c8b4b20        mov     r9,qword ptr [rbx+20h] ds:002b:00000000`00000020=????????????????
Resetting default scope

CPU_COUNT: 18

CPU_MHZ: 95d

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3f

CPU_STEPPING: 2

CPU_MICROCODE: 6,3f,2,0 (F,M,S,R)  SIG: 2B'00000000 (cache) 2B'00000000 (init)

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000020

FOLLOWUP_IP:
hmpalert+1478a
fffff801`717c478a 89842480000000  mov     dword ptr [rsp+80h],eax

BUGCHECK_STR:  AV

READ_ADDRESS:  0000000000000020

DEFAULT_BUCKET_ID:  NULL_CLASS_PTR_DEREFERENCE

ANALYSIS_SESSION_HOST:  MAN-APPASURE

ANALYSIS_SESSION_TIME:  06-02-2017 14:29:42.0079

ANALYSIS_VERSION: 10.0.15063.400 amd64fre

LAST_CONTROL_TRANSFER:  from fffff8016feebec8 to fffff8016feebbcb

STACK_TEXT: 
ffffd000`250fe450 fffff801`6feebec8 : ffffe001`590f28d0 ffffd000`250fe710 ffffd000`250fe708 ffffd000`250fe6f8 : fltmgr!FltpCreateFile+0xc7
ffffd000`250fe550 fffff801`717c478a : 00000000`00000000 ffffe001`590f28d0 ffffd000`250fe708 ffffd000`250fe6f8 : fltmgr!FltCreateFileEx2+0xd0
ffffd000`250fe670 fffff801`717b982d : 00000000`00000000 ffffe001`5abefa00 00000000`00000000 ffffc002`06eee160 : hmpalert+0x1478a
ffffd000`250fe780 fffff801`717b833e : ffffc002`06eee160 fffff800`00000000 ffffe801`79426c00 fffff801`00000000 : hmpalert+0x982d
ffffd000`250fe7d0 fffff801`717b6278 : ffffe001`590f28d0 ffffe001`590f28d0 ffffe801`79426c00 fffff801`717cdbae : hmpalert+0x833e
ffffd000`250fe820 fffff801`717b5fdc : ffffe001`0000000b ffffe001`53561880 00000000`00000000 00000000`00000001 : hmpalert+0x6278
ffffd000`250fe860 fffff801`717c145d : 00000000`44525652 00000000`00000000 ffffc001`f9252010 fffff801`717cf748 : hmpalert+0x5fdc
ffffd000`250fe890 fffff801`6ff04024 : ffffe001`5f1f4620 ffffe001`590f28d0 ffffe001`73cd8880 00000000`00000001 : hmpalert+0x1145d
ffffd000`250fe8e0 fffff801`6ff04268 : 00000000`00000002 fffff800`cff39240 ffffd000`2715d601 ffff6d7c`bd269193 : fltmgr!FltpDoUnloadFilter+0x16c
ffffd000`250fead0 fffff800`d020f160 : fffff800`d0138598 ffffe001`73cd8880 ffffd000`2715d680 ffffd000`2715d680 : fltmgr!FltpMiniFilterDriverUnload+0xfc
ffffd000`250feb10 fffff800`cfccbd6f : fffff800`cfd57100 ffffe001`73cd89c0 fffff800`d0138598 ffffe001`6121b9c8 : nt! ?? ::NNGAKEGL::`string'+0x610d0
ffffd000`250feb50 fffff800`cfcbdf34 : 00000000`00000000 ffffe001`73cd8880 00000000`00000080 ffffe001`73cd8880 : nt!ExpWorkerThread+0x69f
ffffd000`250fec00 fffff800`cfde19c6 : ffffd000`9b73a180 ffffe001`73cd8880 ffffe001`52daf780 ffffd000`250fed90 : nt!PspSystemThreadStartup+0x58
ffffd000`250fec60 00000000`00000000 : ffffd000`250ff000 ffffd000`250f9000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

THREAD_SHA1_HASH_MOD_FUNC:  b3e45fe0d5bf853419ada422084b0acbe084b639

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  78287a7eeaba0bc01a240dab78e8d766333aae63

THREAD_SHA1_HASH_MOD:  0af9beb6c879a65f84b8796640a5b54de50b621a

FAULT_INSTR_CODE:  80248489

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  hmpalert+1478a

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: hmpalert

IMAGE_NAME:  hmpalert.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5899cd41

STACK_COMMAND:  .cxr 0xffffd000250fda20 ; kb

BUCKET_ID_FUNC_OFFSET:  1478a

FAILURE_BUCKET_ID:  AV_hmpalert!unknown_function

BUCKET_ID:  AV_hmpalert!unknown_function

PRIMARY_PROBLEM_CLASS:  AV_hmpalert!unknown_function

TARGET_TIME:  2017-06-01T12:10:11.000Z

OSBUILD:  9600

OSSERVICEPACK:  18505

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8.1

OSEDITION:  Windows 8.1 Server TerminalServer SingleUserTS

OS_LOCALE: 

USER_LCID:  0

OSBUILD_TIMESTAMP:  2016-09-30 15:31:28

BUILDDATESTAMP_STR:  160930-0600

BUILDLAB_STR:  winblue_ltsb

BUILDOSVER_STR:  6.3.9600.18505

ANALYSIS_SESSION_ELAPSED_TIME:  975

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_hmpalert!unknown_function

FAILURE_ID_HASH:  {9453d9a1-2fd9-1a8d-d742-01cc77f6eeb0}

Followup:     MachineOwner
---------

Looks like an issue with the hmpalert.sys driver during unload by the Windows Filter Manager.  The hmpalert.sys driver is a file system mini-filter so I suspect that during an update of the Cryptoguard component, in order to remove the existing driver (with a view to load the new one) it was unloaded.  

It would also get unloaded at shutdown but I assume this bugcheck didn't happen at shutdown.  I would contact Support with this thread as reference.  Hopefully they are aware of this but I don't see this happening again in the short term and the driver isn't typically unloaded unless there is an update.

Regards,

Jak

Hi Robert,

Just curious if did you get a response from Support?

Regards,

Jak

Hi , 

I would recommend you to contact support for this issue, seems to be serious and we need an engineer to collect the logs from your system.