Advisory: Support Portal Maintenance. Login is currently unavailable, more info available here.

System.IO.IOException: Pipe is broken

Starting Dec 21st we started seeing a tremendous amount of errors on both our Server Infrastructure and Endpoint devices.  This created issues with certain .NET related applications on end users workstations that required restarting various applications. One application particularly troublesome was Mimecast For Outlook. Upon investigating we found that the only resolution to fix these errors was to completely remove Sophos (obviously that's not a solution nor a risk we're willing to take). 

Sample Errors - 

28-12-2020 09:07:41,964 ERROR [12] HOST: Domain Unhandled Exception: System.IO.IOException: The pipe is being closed.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Pipes.NamedPipeServerStream.BeginWaitForConnection(AsyncCallback callback, Object state)
   at System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(IAsyncResult asyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP). IsTerminating: True (Program)

Message=Application: msddsk.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException
   at System.IO.__Error.WinIOError(Int32, System.String)
   at System.IO.Pipes.PipeStream.WinIOError(Int32)
   at System.IO.Pipes.NamedPipeServerStream.GetImpersonationUserName()
   at Mimecast.Mapi.Remote.NamedPipesServer.AcceptPipeConnection(System.IAsyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Is anyone else seeing this? We'll certainly open a Ticket with support but wanted to also understand the scope. 

  • We confirmed that disabling the MTR service seemed to resolve the issue as well on workstations. Have not tested the encryption service yet. 

  • This is interesting; haven't seen these issues anywhere yet... are you seeing exceptions on your Exchange server itself, or is it on the client side (outlook) only?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.

  • Search against system logs; you likely will see broken pipes affecting application communications. Now I suspect Sophos is not aware because the Pipes get re-created but some applications do NOT like it (Such as the Mimecast Plugin). We also have other errors starting to pop up but I'm comfortable saying they are related yet. 

    To clarify this issue is evident on all Windows systems and endpoints where Sophos MTR is installed. Also some errors are unrelated but I filtered those out in my search specific to our applications. 

    If you use Splunk here is a search we are using internally;

    sourcetype="WinEventLog:Application"  (IOException OR 0x8007006d)

    Here's a specific server where we disabled the MTR Service; As you can see the errors stopped after the service was stopped. 

  • Sharing one of those logs

    12/30/2020 09:23:29 AM
    LogName=Application
    SourceName=.NET Runtime
    EventCode=1022
    EventType=2
    Type=Error
    ComputerName=
    User=NOT_TRANSLATED
    Sid=
    SidType=0
    TaskCategory=The operation completed successfully.
    OpCode=Info
    RecordNumber=7087945
    Keywords=Classic
    Message=.NET Runtime version 4.0.30319.0 - Loading profiler failed.  Failed trying to receive from out of process a request to attach a profiler.  HRESULT: 0x8007006d.  Process ID (decimal): 14220.  Message ID: [0x250d].

  • Here's another log;

    12/30/2020 04:21:51 PM
    LogName=Application
    SourceName=openvpnserv
    EventCode=0
    EventType=2
    Type=Error
    TaskCategory=None
    OpCode=Info
    RecordNumber=476911
    Keywords=Classic
    Message=
    
    Got the following information from this event: 
    
    openvpnserv error: The pipe has been ended. (0x6d)
    PeekNamedPipeAsync failed

    Again, all occurring on the 21st and stops once MTR is disabled. 

  • Interesting; I don't have any customers using Mimecast so I guess that's why I haven't heard of this.  Good sleuthing on your part.  I assume you've shared this in a Sophos Support case?

    CTO, Convergent Information Security Solutions, LLC

    https://www.convergesecurity.com

    Advice given as posted on this forum does not construe a support relationship or other relationship with Convergent Information Security Solutions, LLC or its subsidiaries.  Use the advice given at your own risk.

  • Hi Drew, Justin,

    I just came across this post from looking for answers!  I finally found so thanks for the post.  What is the latest with your case?  We are also MTR customer and also using the Mimecast Plugin and it started to happen around the same time for our desktops too.  I haven't looked at any of our servers for errors yet.  I have a ticket logged with Mimecast but I guess this will be for Sophos to resolve.

    Looking forward to the latest update.  Will play around with the services you mention and report back.

    Thanks and Regards (happy new years eve)

  • Good day -

    Unfortunately I have not had any new information from Sophos since confirming that it's related to the MTR service. 

    Happy New years eve to you all well!

  • We are not using encryption, so that service is not present on our systems.  I can also confirm stopping the MTR service stops the msddsk.exe from repeatedly crashing.  BTW you can manually run the process to start it up but will only run for a maximum of 10-15 mins before it faults again.  My Sophos Support Case 03496588.

  • Does anyone have any useful updates yet?  Still working with support but it is slow going.