SPAM (confirmed) - Problems with Cyren Database or bad pattern?

This has started again on May. 6th.

On three UTM 9.509-3 appliances in Norway. A large amount of emails from senders on Office 365 is rejected with "SPAM Confirmed" these are obviously false positives. And it is causing huge problems. Businesses are stopping up because of emails not being received. It has already had a cost of delayed shipments and lost orders. 

I had to add an exeption where SPAM check was bypassed on all emails coming from Office 365 Outbound Security. This is a horrible solution but was necessary. Now I have to manually all day long scavenge the logs for emails that should not have passed and manually delete these.

What is going on?

Stig

This has started again on May. 6th.

On three UTM 9.509-3 appliances in Norway. A large amount of emails from senders on Office 365 is rejected with "SPAM Confirmed" these are obviously false positives. And it is causing huge problems. Businesses are stopping up because of emails not being received. It has already had a cost of delayed shipments and lost orders. 

I had to add an exeption where SPAM check was bypassed on all emails coming from Office 365 Outbound Security. This is a horrible solution but was necessary. Now I have to manually all day long scavenge the logs for emails that should not have passed and manually delete these.

What is going on?

Stig

Guys, if this is being blocked as "spam" (the exact reason will be in the logs as Sachin mentioned) then there is an issue with the Cyren anti-spam system.  Most are not familiar with how this works; there is no pattern db, etc. referenced on the UTM when doing the spam check; instead an algorithm is run against the email, and a "signature" is generated.  This is compared with a real-time lookup on Cyren's (maybe Sophos hosts some mirrors, but I can't recall, been a while) spam database.  Cyren uses a number of methods to update their spam DB constantly... I have at times noted issues with their system (as happens with any anti-spam system), and pointing it out to Sophos (or in some cases, I've taken it direct to Cyren... back when they were known as Commtouch).  They will need to work with Cyren to resolve the issue.

The most likely reason for this is I am now seeing spammers leveraging Office 365 and other cloud mail services to spam folks (most likely via hijacked accounts) and so that's probably why this is happening.   I would open a case with Sophos Support instead of posting on this forum to get this issue moving.

Just hoping someone would know anything and hopefully that it will be solved soon.

I really don't want to open a support case. Every time I do it takes forever to get resolved.

<rant>

First time after buying and using Sandstorm only to find out after 6 months that it was not working at all, I opened a support case. It took 3 months for support to solve the case.

Then later when an email with cryptovirus got by they used another two months to solve that.
Then with all these bad firmware pathes last months creating all sorts of hell.

Also Sophos recommended some of my customers to Upgrade to XG from SG and replace UTM Endpoint with Sophos Cloud only to discover after that actual license cost went up with several hundred percent. One Case Sophos UTM Endpoint Protection cost 1500$, Sophos Cloud Ended up costing 5600$ Customers were furious. Greedy licensing and repeated serious issues is getting on the customers and my nerves.

Bought some XG firewalls for site-to-site IPsec to branch offices in China, USA, Lithuania and Thailand. Needed Network Protection only to discover after buying that Support and firmware updates was not included anymore on XG like it was on SG. Sales representative did not mention this.
Ended up losing money on this sale.

I will recommend to replace Sophos with other brand in near future. Getting tired of all these problems.

</rant>

Stig

We are having the exact same problem at several firewalls. Legitimate emails are being stopped with "Spam confirmed" as reason. No other explenation. It started monday morning may 7th.

I reported this to Sophos support and received the following answer:

"I have spoken to our global escalation team in regards to the pattern update and they have confirmed that there is no issues within the pattern.
They also mentioned that Cyren do not block domains or RBL but the IP's that are associated to the domains. In addition to this they have said that getting samples to us is the best way to get this resolved for you."

But how am I supposed to retrieve samples of emails that is never saved anywhere? And this is of such massive magnitude that I refuse to believe it is related to single senders and senders domains. I really hope Sophos will look further into this with Cyren.

One of my addresses is subscribed to a listelixr.net mailing list.  The first one I see incorrectly blocked was on 4/30.  It's accelerated now.

We also occasionally buy/sell something on eBay.  Here's the result of grepping the SMTP logs since last year:

zgrep 'mailx1.ebay.com' /var/log/smtp/2017/*/*|grep -oP '".*?"'|sort -n|uniq -c|sort -n
      2 "email rejected"
      5 "email quarantined"
    115 "email passed"
zgrep 'mailx1.ebay.com' /var/log/smtp/2018/01/*|grep -oP '".*?"'|sort -n|uniq -c|sort -n
      4 "email passed"
zgrep 'mailx1.ebay.com' /var/log/smtp/2018/02/*|grep -oP '".*?"'|sort -n|uniq -c|sort -n
      8 "email passed"
zgrep 'mailx1.ebay.com' /var/log/smtp/2018/03/*|grep -oP '".*?"'|sort -n|uniq -c|sort -n
     18 "email passed"
zgrep 'mailx1.ebay.com' /var/log/smtp/2018/04/*|grep -oP '".*?"'|sort -n|uniq -c|sort -n
      4 "email rejected"
      7 "email passed"

Enough were rejected last month that none were sent this month.

This is a definite problem that needs to get escalated immediately.  I will enter a case with Support and ring bells at Sophos...

Cheers - Bob

I have been on the phone with Sagar Dave at Sophos support for a couple of hours today. He remoted in to my screen and after a LOT of explenation I was able to make him understand what the problem was. He spoke with his Senior technician and after a while I got this promising answer:

I consulted my senior team and have passed the details regarding faulty signature causing issues with legit inbound emails.
It might take some time to remove the signatures as we don't have sample emails available currently.
Also if we could change the ACTON of confirmed quarantine from DROP to QUARANTINE maybe we can get those samples.
I also verified the logs and it's been confirmed that Anti-Spam engine is marking them as spam. For now you can create exceptions in order to avoid the issue.

If any of you contact support about this it will not hurt if you tell them to look at my ticket with id #8097489

I hope this gets fixed quite soon. This is so big in scale that my paranoia triggers. What if this is a targeted attack/manipulation of filters to make organizations switch off antispam on their firewalls.

Hei stig, Hallo Rolf-Arn and welcome both to the UTM Community!

I have set 'Reject at SMTP time: Off' and 'Confirmed spam action: Quarantine' so that I can try to capture one of the emails that have been being rejected.  If anyone else can do that and capture an example or two, PM me to get my email address and I'll forward your capture to the Escalation Engineer that's working with Cyren right now.

Cheers - Bob

Hi Community,

If you affected by this issue, please follow the directions as advised above by BAlfson to set your spam action to Quarantine to properly capture these emails. We will be needing these samples to further the investigation.

Best,

Do you wonder if this is a problem for you?

Here's a quick way to get a list of all email senders to you (user@domain.com) that were blocked as confirmed spam this year:

zgrep 'reason="as" extra="confirmed"' /var/log/smtp/2018/*/*|grep 'to="user@domain.com"'|grep -oP 'from=".*?"'|sort -n|uniq -c|sort -n

Or, just the ones this month:

zgrep 'reason="as" extra="confirmed"' /var/log/smtp/2018/05/*|grep 'to="user@domain.com"'|grep -oP 'from=".*?"'|sort -n|uniq -c|sort -n

Cheers - Bob

I can also confirm strange issues with spam confirmed at customer sites in the last days.

Glad to hear that Sophos now really tries to fix/analyse this issue instead of just telling "false positives"! Of course there can be "false positives" but not in this way...

regards

I've captured four so far. Still need others to open a ticket and submit or to submit via me.

Cheers - Bob