Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 36 replies
  • Subscribers 20 subscribers
  • Views 34011 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

System.IO.IOException: Pipe is broken

DrewbleIT

Starting Dec 21st we started seeing a tremendous amount of errors on both our Server Infrastructure and Endpoint devices.  This created issues with certain .NET related applications on end users workstations that required restarting various applications. One application particularly troublesome was Mimecast For Outlook. Upon investigating we found that the only resolution to fix these errors was to completely remove Sophos (obviously that's not a solution nor a risk we're willing to take). 

Sample Errors - 

28-12-2020 09:07:41,964 ERROR [12] HOST: Domain Unhandled Exception: System.IO.IOException: The pipe is being closed.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Pipes.NamedPipeServerStream.BeginWaitForConnection(AsyncCallback callback, Object state)
   at System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(IAsyncResult asyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP). IsTerminating: True (Program)

Message=Application: msddsk.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException
   at System.IO.__Error.WinIOError(Int32, System.String)
   at System.IO.Pipes.PipeStream.WinIOError(Int32)
   at System.IO.Pipes.NamedPipeServerStream.GetImpersonationUserName()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(System.IAsyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Is anyone else seeing this? We'll certainly open a Ticket with support but wanted to also understand the scope. 



This thread was automatically locked due to age.
  • 0 in reply to DrewbleIT

    We confirmed that disabling the MTR service seemed to resolve the issue as well on workstations. Have not tested the encryption service yet. 

  • 0 in reply to DrewbleIT

    This is interesting; haven't seen these issues anywhere yet... are you seeing exceptions on your Exchange server itself, or is it on the client side (outlook) only?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Sophos Platinum Partner

    --------------------------------------

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.

  • 0 in reply to BrucekConvergent

    Search against system logs; you likely will see broken pipes affecting application communications. Now I suspect Sophos is not aware because the Pipes get re-created but some applications do NOT like it (Such as the Mimecast Plugin). We also have other errors starting to pop up but I'm comfortable saying they are related yet. 

    To clarify this issue is evident on all Windows systems and endpoints where Sophos MTR is installed. Also some errors are unrelated but I filtered those out in my search specific to our applications. 

    If you use Splunk here is a search we are using internally;

    sourcetype="WinEventLog:Application"  (IOException OR 0x8007006d)

    Here's a specific server where we disabled the MTR Service; As you can see the errors stopped after the service was stopped. 

  • 0 in reply to DrewbleIT

    Sharing one of those logs

    12/30/2020 09:23:29 AM
LogName=Application
SourceName=.NET Runtime
EventCode=1022
EventType=2
Type=Error
ComputerName=
User=NOT_TRANSLATED
Sid=
SidType=0
TaskCategory=The operation completed successfully.
OpCode=Info
RecordNumber=7087945
Keywords=Classic
Message=.NET Runtime version 4.0.30319.0 - Loading profiler failed.  Failed trying to receive from out of process a request to attach a profiler.  HRESULT: 0x8007006d.  Process ID (decimal): 14220.  Message ID: [0x250d].

  • 0 in reply to DrewbleIT

    Here's another log;

    12/30/2020 04:21:51 PM
LogName=Application
SourceName=openvpnserv
EventCode=0
EventType=2
Type=Error
TaskCategory=None
OpCode=Info
RecordNumber=476911
Keywords=Classic
Message=

Got the following information from this event: 

openvpnserv error: The pipe has been ended. (0x6d)
PeekNamedPipeAsync failed

    Again, all occurring on the 21st and stops once MTR is disabled. 

  • 0 in reply to DrewbleIT

    Interesting; I don't have any customers using Mimecast so I guess that's why I haven't heard of this.  Good sleuthing on your part.  I assume you've shared this in a Sophos Support case?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Sophos Platinum Partner

    --------------------------------------

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.

  • 0 in reply to DrewbleIT

    Hi Drew, Justin,

    I just came across this post from looking for answers!  I finally found so thanks for the post.  What is the latest with your case?  We are also MTR customer and also using the Mimecast Plugin and it started to happen around the same time for our desktops too.  I haven't looked at any of our servers for errors yet.  I have a ticket logged with Mimecast but I guess this will be for Sophos to resolve.

    Looking forward to the latest update.  Will play around with the services you mention and report back.

    Thanks and Regards (happy new years eve)

  • 0 in reply to Paul Haley

    Good day -

    Unfortunately I have not had any new information from Sophos since confirming that it's related to the MTR service. 

    Happy New years eve to you all well!

  • 0 in reply to DrewbleIT

    We are not using encryption, so that service is not present on our systems.  I can also confirm stopping the MTR service stops the msddsk.exe from repeatedly crashing.  BTW you can manually run the process to start it up but will only run for a maximum of 10-15 mins before it faults again.  My Sophos Support Case 03496588.

  • 0

    Does anyone have any useful updates yet?  Still working with support but it is slow going.

Unfiltered HTML