Advisory: Reflexion - Delayed delivery of outbound and inbound email

Good Evening,

Any update? I see the system status has turned green for some reason. Maybe because I see like 4 emails got delivered! Now just cough up the thousands our customers are waiting on. 

We didn't even mention how this is effecting the MSPs that actually use your product. You realize we missed nearly all ticket submissions to us today? Who knows if we'll ever get em!

At this point, seems like a waste of energy to hope this gets resolved.

Here's the MOST sad part. We just put all our client on the central platform for AV. The plan was to move them over to that, IF THEY EVER GET THE MIGRATION PLATFORM FINISHED, like they promised. 

Why pull all your resources off a product so populated with dedicated customers? You should have used your resources to make a way off this platform before you left it to die. 

Sean

Just checked several clients' mail logs - EVERYONE stopped receiving mail around 7:06pm EDT. I'm not talking they are deferred or delayed. They aren't even showing up in Reflexion.

Did this problem just get worse, or am I missing something?!?!

I've noticed that, and made the decision to jump ship. When I realized the plethora of test emails I sent never hit their queue, I lost all faith. I believe the emails in error state will get delivered, but who knows how many countless are floating in the ether. 

7/30/19

9:00PM EDT - Reflexion is still experiencing a delay in the processing of inbound and outbound email.

Delivery of outbound email flow is processing but at a slower rate as the delivery queues are catching up and working to return to normal levels.

We are still experiencing a delay in delivery of inbound email, and our infrastructure engineering team is actively working to resolve the issue and restore mail flow as quickly as possible.

We are working as fast as possible to resolve the issue. We sincerely apologize for the inconvenience this has caused and will provide an update as soon as we have additional information.

This is completely ridiculous!! I can understand an outage, but there is NO backup servers to move to while things are fixed and not giving a concern for any of the replies. We all have clients on this service as our self. Last email I received was 7:45am PST and only one single email since at 6:46pm PST.

Where is an explanation of what is going on in the first place and furthermore an explanation of "Note: No email will be lost, it is currently being queued for delivery.".

Unacceptable lack of communication!

Unacceptable lack of communication!!

Unacceptable lack of communication!!!

Unacceptable lack of communication!!!!

We are forced to move all tenants to a different service that is forthcoming no matter what the consequences. If we are not able to report to our clients because our vendor is being tight lipped and too cocky to admit what's going on and where the point of resolutions may in fact be, that's NOT a a vendor we want to do business with. This type of relationship puts us in a compromised and liable position and is NOT good for our business and reputation! Honest and transparency at the end of the day are more valuable than address on the fly as that's all I see at this point as a differentiation!!

Sophos: Get off your high horses and spill the beans!!

7/30/19

[Update]11PM EDT - All Reflexion services have been restored and all messages queued in Reflexion have been processed.

• The inbound and outbound emails that have been queued are now being received, processed, and delivered as quickly as possible.
• No email was lost, and all queued email should be delivered over the next few hours.
• Again, we sincerely apologize for the inconvenience this has caused and will provide a final update once the queues return to normal.

There also needs to be one place for customer updates.  Right now I'm checking 2 system status pages both with different statuses.  

https://monitor.reflexion.net/system-health/

https://monitor.reflexion.net/system-health/minimal/

This KB article is the third place.  Never seen ongoing updates done via KB article before.

Add to that random Twitter comments usually copy pasting the generic 'Engineering is actively working to resolve ASAP.' update. 

None of these methods have given me much except that I 'think' someone is working on it.  If you are going to use Twitter to provide updates then get somebody on Twitter and provide real updates.  Twitter is already a stupid place to update everyone but if you are going to use it then use it.  My rep doesn't know anything and stopped replying end of business day.  Why do all 'AV' companies eventually grow too fast and turn into a pile of <insert appropriate emoticon here>?  Everyone can accept there will be the occasional outage but keeping your partners in the dark is unacceptable as has been said above.  8:30p and I'm still at office dealing with this crap and haven't billed a dime all day.  Stupid.

I am sorry but this is utter BS!  Emails were lost, every customer account has reported emails that bounced back, never were received, and some that came through with blank message bodies...  A lot of email was lost in this severe screw up on Sophos' end.  Completely unacceptable!  Then the communication from Sophos....  Even worse!

NEW inbound emails are NOT being queued in Reflexion.  I'm sending test email after test email  to my Reflexion customers and none of them show up anywhere queued inbound in Reflexion.  They are sitting my outgoing queue of O365 servers in a 'delayed' state.  I currently have all Reflexion routing turned off on my side so I can test and communicate.  I've sent over half a dozen emails to different customers and none of them receive the email nor are they visible in the queue waiting to be delivered.  It is NOT fixed.

Unfortunately, email was lost. I'm sure MSPs will lose customers. I'm even more sure Sophos will lose customers. 

Losing day all around