Advisory: Reflexion - Delayed delivery of outbound and inbound email

7/31/19 - 2:30PM EDT - Update on Issues with Reflexion Email Security Service:

• As of 2:30 pm EDT, all inbound and outbound mail delivery through Reflexion is returning to normal, although some end users may periodically notice delays due to the high volume of emails we are currently processing. We expect that any delays will continue to improve as the queues are processed. We will continue to provide updates as the operation progresses back to a fully healthy state with no delays.
• Please know that all email that was queued in the last 24 hours has been safely processed and will be sent and received as the flow resumes - no emails were lost by the Reflexion operation as a result of this issue.
• We expect improvements throughout the day. The operations and development teams are diligently monitoring the service to ensure all Reflexion customer email is fully resumed. Note: This issue affected customers of the Reflexion service and did not impact any of our services in Sophos Central.
• While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure was exceeded, due to a routine maintenance process that was executed incorrectly. We are continuing the investigation and will provide a thorough root cause analysis (RCA) once complete.
• We sincerely apologize for the inconvenience this has caused and will continue to provide updates during the day. Please continue to watch this KBA page.

https://community.sophos.com/kb/en-us/134470

Any info on queue sizes?  I'm still seeing delays on the couple of customers I have still using Reflexion.  Everyone else is bypassing it for now.  I'd like to set everyone back but I don't want that to do that until I know email is back to normal.

What I want to know is, if I remove someone from the Reflexion console, so that they are no longer going through reflexion, will queued and delayed emails still reach them, or will those emails just get lost?

I am not buying the "No emails were lost". I'm getting the distinct impression that emails that were received may not have been lost, but any emails that were outright rejected are relying on the sender's email servers doing retries. How many of them will retry after 2 days of failures?

"No emails were lost".... that is like your receptionist saying "I delivered all messages I was given" but neglecting to say she didn't answer the phone all day.

http://sms.sophos.com is supposed to send out alerts.  But I just found that today.  So far I haven't gotten anything yet

Hi:

Can we get another update please? The last update was 2:30 EDT, more than 3 hours ago. Inbound and Outbound emails are still delayed more than an hour in some cases.

Thank you!
Bob H.

We moved to Proofpoint via PAX8. PAX8 has been a great vendor and asset to work with. Highly recommended. 

Where did everybody else move to? Can we do polls in this forum? Would be great!!

Anyway, adios Reflexion. Your dishonesty and withholding of information has sure fucked this relationship up and we cannot take the risk with further liability. Plans were to move to Sophos Central Email, but that's out of the question at this point.

Reflexion/Sophos: Enjoy what you have created and I don't expect much to be learned by you!

7/31/19 - [Update] 9:00 pm EDT:

• All Reflexion services have been completely restored, and all queued emails have been delivered. The infrastructure and engineering teams continue to diligently monitor the service. 
• While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 
• We sincerely apologize for the inconvenience this has caused and are continuing the investigation. We will provide a thorough root cause analysis (RCA) once it’s complete. 

8/1/19 - 9:00 am EDT - Update:

• All Reflexion services have been completely restored, and all queued emails have been delivered. The infrastructure and engineering teams continue to diligently monitor the service.
• While we are still finalizing the root cause of the problem, we have determined that during the initial incident the IO limits of our storage infrastructure were exceeded. 
• We sincerely apologize for the inconvenience this has caused and we will provide a thorough root cause analysis (RCA) once it’s completed. 
• We will continue to provide updates as there is additional information around this incident to report.

[Update] 8/1/19- 12:00pm EDT

• All Reflexion services continue to process email with no delivery queue delays. This has been the case since services had been fully restored approximately 14 hours ago. The infrastructure and engineering teams continue to monitor the service 24x7 to ensure this remains the case.
• We have identified the source of the incident which caused the excessive IO that led to this issue and we are in the process of drafting a thorough root cause analysis (RCA) that will include the details of what happened, how it happened, and steps we are taking (and have already taken) to ensure that this type of incident does not occur in the future. We will make the RCA available once it is completed.
• To reiterate, all Reflexion services are fully operational and delivering mail in real-time. The Reflexion system did not lose any email during this incident and eventually processed and delivered all inbound and outbound messages received by the service. Also, at no time did this incident affect Sophos Central or related services as the Reflexion system is completely independent of Sophos Central-managed products and services.
• We will provide an update when there is something additional to inform on.

PSA: Email has been reported to have been lost as well as many bounced and/or would not leave sender's outbox. Many many implications with the reality and again dishonesty that Sophos is telling us.