Advisory: Support Portal Maintenance. Login is currently unavailable, more info available here.

System.IO.IOException: Pipe is broken

Starting Dec 21st we started seeing a tremendous amount of errors on both our Server Infrastructure and Endpoint devices.  This created issues with certain .NET related applications on end users workstations that required restarting various applications. One application particularly troublesome was Mimecast For Outlook. Upon investigating we found that the only resolution to fix these errors was to completely remove Sophos (obviously that's not a solution nor a risk we're willing to take). 

Sample Errors - 

28-12-2020 09:07:41,964 ERROR [12] HOST: Domain Unhandled Exception: System.IO.IOException: The pipe is being closed.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Pipes.NamedPipeServerStream.BeginWaitForConnection(AsyncCallback callback, Object state)
   at System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(IAsyncResult asyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP). IsTerminating: True (Program)

Message=Application: msddsk.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException
   at System.IO.__Error.WinIOError(Int32, System.String)
   at System.IO.Pipes.PipeStream.WinIOError(Int32)
   at System.IO.Pipes.NamedPipeServerStream.GetImpersonationUserName()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(System.IAsyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Is anyone else seeing this? We'll certainly open a Ticket with support but wanted to also understand the scope. 

Parents Reply Children
  • This is interesting; haven't seen these issues anywhere yet... are you seeing exceptions on your Exchange server itself, or is it on the client side (outlook) only?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.

  • Search against system logs; you likely will see broken pipes affecting application communications. Now I suspect Sophos is not aware because the Pipes get re-created but some applications do NOT like it (Such as the Mimecast Plugin). We also have other errors starting to pop up but I'm comfortable saying they are related yet. 

    To clarify this issue is evident on all Windows systems and endpoints where Sophos MTR is installed. Also some errors are unrelated but I filtered those out in my search specific to our applications. 

    If you use Splunk here is a search we are using internally;

    sourcetype="WinEventLog:Application"  (IOException OR 0x8007006d)

    Here's a specific server where we disabled the MTR Service; As you can see the errors stopped after the service was stopped. 

  • Sharing one of those logs

    12/30/2020 09:23:29 AM
    LogName=Application
    SourceName=.NET Runtime
    EventCode=1022
    EventType=2
    Type=Error
    ComputerName=
    User=NOT_TRANSLATED
    Sid=
    SidType=0
    TaskCategory=The operation completed successfully.
    OpCode=Info
    RecordNumber=7087945
    Keywords=Classic
    Message=.NET Runtime version 4.0.30319.0 - Loading profiler failed.  Failed trying to receive from out of process a request to attach a profiler.  HRESULT: 0x8007006d.  Process ID (decimal): 14220.  Message ID: [0x250d].

  • Here's another log;

    12/30/2020 04:21:51 PM
    LogName=Application
    SourceName=openvpnserv
    EventCode=0
    EventType=2
    Type=Error
    TaskCategory=None
    OpCode=Info
    RecordNumber=476911
    Keywords=Classic
    Message=
    
    Got the following information from this event: 
    
    openvpnserv error: The pipe has been ended. (0x6d)
    PeekNamedPipeAsync failed

    Again, all occurring on the 21st and stops once MTR is disabled. 

  • Interesting; I don't have any customers using Mimecast so I guess that's why I haven't heard of this.  Good sleuthing on your part.  I assume you've shared this in a Sophos Support case?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.