Advisory: Reflexion - Delayed delivery of outbound and inbound email

I have taken the decision to start migrating my clients quickly.

@Froader -- Completely agreed that this was a very poorly communicated outage, and has caused a lot of damaged relationships. It's clear that Reflexion has been neglected in favor of other Sophos products, and this outage helps with making the decision to look at other solutions an obvious one.

Harley: It's not going to be everyone's experience... Sorry to hear about your double whammy. Ouch, BTW. Interestingly enough since there wasn't any way to determine the aftermath of this outage other than comparing inbox and sent to every possible combination, I've learned a lot from our clients which was a phone call from me directly to assess the damage. While some were less affected than others, their requests for help never made it to our ticketing system and that's detrimental to our ability to meet our SLAs. That's a liability and since Sophos was so gracious in communicating what was going on we were unable to not only communicate via email due to the outage, but provide any status updates during our phone calls because our vendor Sophos was NOT supporting their product in our best interest and more valuable our client's best interest. Call that withholding the truth or being dishonest. Your call, but a complete falling out of trust when it comes to a vendor relationship.

One word for you to think about is liability and that means your liability as a company whom I hope puts the risk in your ball park when having a signature on the other side of your agreement. I believe that there is always room for fair error, but it must be communicated and transparent. It's literally been days without any legitimate communication from Sophos. If they did infarct communicate the truth, we would have all jumped ship much sooner an that was their Catch 22. We will NOT repeat this process in favor of our clients' best interest, end of story.

I'm only bringing some real life experience to this situation, so take it for what it's worth. My statements stand intact BTW and we have moved on. What's really interesting in the aftermath is that Reflexion is super responsive navigating the portal, which prior to this outage was spotty at times. Clearly Sophos has not been keeping up with resources needed to keep Reflexion operating and the *** finally hit the fan

We had several clients report bounced messages as well.

I think the trick to the message Reflexion sent out was the line along the lines of "Mail that was queued".  We were trying to send mail from various personal accounts and MXtoolbox.  Many times the servers never responded to the connection request.  So of course the mail wouldn't be queued at that point.

@Froader -- This has not been my experience. I did deal with an Office 365 related outage in addition to the Reflexion issues, so timing on both of these outages caused a real headache for us and our clients, but all mail that successfully left Microsoft servers and reached Reflexion servers were eventually forwarded as far as I'm aware. While I did notice a couple time-out errors come back to me for mail sent to addresses protected by Reflexion (I posted one of them in a previous comment to this thread), the messages were eventually received by the recipient as Reflexion resolved their mail-flow delay. I assume this could be due to automatic re-submission attempts however.

If I come across any differing evidence, I'll make sure to post that here.

PSA: Email has been reported to have been lost as well as many bounced and/or would not leave sender's outbox. Many many implications with the reality and again dishonesty that Sophos is telling us.

[Update] 8/1/19- 12:00pm EDT

• All Reflexion services continue to process email with no delivery queue delays. This has been the case since services had been fully restored approximately 14 hours ago. The infrastructure and engineering teams continue to monitor the service 24x7 to ensure this remains the case.
• We have identified the source of the incident which caused the excessive IO that led to this issue and we are in the process of drafting a thorough root cause analysis (RCA) that will include the details of what happened, how it happened, and steps we are taking (and have already taken) to ensure that this type of incident does not occur in the future. We will make the RCA available once it is completed.
• To reiterate, all Reflexion services are fully operational and delivering mail in real-time. The Reflexion system did not lose any email during this incident and eventually processed and delivered all inbound and outbound messages received by the service. Also, at no time did this incident affect Sophos Central or related services as the Reflexion system is completely independent of Sophos Central-managed products and services.
• We will provide an update when there is something additional to inform on.

8/1/19 - 9:00 am EDT - Update:

• All Reflexion services have been completely restored, and all queued emails have been delivered. The infrastructure and engineering teams continue to diligently monitor the service.
• While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 
• We sincerely apologize for the inconvenience this has caused and we will provide a thorough root cause analysis (RCA) once it’s completed. 
• We will continue to provide updates as there is additional information around this incident to report.

7/31/19 - [Update] 9:00 pm EDT:

• All Reflexion services have been completely restored, and all queued emails have been delivered. The infrastructure and engineering teams continue to diligently monitor the service. 
• While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 
• We sincerely apologize for the inconvenience this has caused and are continuing the investigation. We will provide a thorough root cause analysis (RCA) once it’s complete. 

We moved to Proofpoint via PAX8. PAX8 has been a great vendor and asset to work with. Highly recommended. 

Where did everybody else move to? Can we do polls in this forum? Would be great!!

Anyway, adios Reflexion. Your dishonesty and withholding of information has sure fucked this relationship up and we cannot take the risk with further liability. Plans were to move to Sophos Central Email, but that's out of the question at this point.

Reflexion/Sophos: Enjoy what you have created and I don't expect much to be learned by you!