Sophos Firewall: v19.0 MR1: Feedback and experiences

Hi,

I received an email about 10 minutes ago, it has a number of attachments which are all blank, further the email does not show in logviewer -> email. There a e two entries at the time of the email with no details and show possible spam.

Ian

Further to the message above, I sent the originator a question about the attachment and he advised other recipient were able to view it. I took the iPhone out of the wifi and was able to see the attachment which is an mp4 file.. So something is broken in the latest mail functions in v19.0.1.

I have not made any changes to my mail settings, least of all have the ability to hide messages from XG log viewer -> email report. I have received other messages which do not show up in log viewer. This is an issue that was fixed a couple of versions ago.

Ian

Ian

Just for my interest. Which SASI Version does the customer currently use? 

2022-08-02.11:40:15 MESSAGE [Main] [ precompile.cpp:583] Downloaded file /sdisk/sasi/asdb.delta is verified with checksum..
2022-08-02.11:40:17 MESSAGE [Main] [ engine.cpp:845] Database loaded of version: 2022.8.2.91519
2022-08-02.11:40:17 MESSAGE [Main] [ precompile.cpp:761] [Precompile thread]: Signatures are reloaded with latest delta and verified with checksum
of new signatures.

That is my latest version (extracted by /log/sasi.log). 

Here you go:

2022-08-02.11:40:20 MESSAGE    [Main] [          precompile.cpp:583] Downloaded file /sdisk/sasi/asdb.delta is verified with checksum..
2022-08-02.11:40:23 MESSAGE    [Main] [              engine.cpp:845] Database loaded of version: 2022.8.2.91519
2022-08-02.11:40:23 MESSAGE    [Main] [          precompile.cpp:761] [Precompile thread]: Signatures are reloaded with latest delta and verified with checksum of new signatures.

Update.

Hi, 

For me it looks like this :(

19.0 MR1

Well, without a comparison to 18.5 MR2 it's not expressive. If you route a lot clean messages through the firewall, it will always look like this:

Dreamcatcher Could you attach the SupportAccessID to your case please? 

And also: To report False-Negative, you can forward the email to : https://support.sophos.com/support/s/article/KB-000033422?language=en_US

You do not need to send them to support. 

YOu can find the access ID in case 05553951. Support asked me to send them spam mails that got through, so I did.

1 week 18.5 MR2 vs. 1 week 19.0 MR1.

Not only is the overall detection rate significantly lower, "Confirmed spam" vs. "Probable spam" is pretty much useless too now with the new engine, since there literally is no "confirmed spam" anymore:

How something like this makes it to production state is beyond me.

I did look into the UTM reviews as well. There is no real feedback about the bad positioning of SASI in UTM, i could found now. No escalations either. And UTM uses the same SASI version / pattern like SFOS. It is correct, in UTM you can write your own "blocklists" based on words etc. But is this the workaround "all utm customers" use? I found it hard to believe, that all UTM customers (home vs business) have no raise any attention to the SASI engine. 

See: https://community.sophos.com/utm-firewall/f/mail-protection-smtp-pop3-antispam-and-antivirus/128189/9-706---anti-spam-engine-changed-to-sasi Most feedback was about the old hardware and the support of the engine. 

BTW: SASI works differently on Probable Spam vs Spam. Confirmed does not exist in SASI, so the labels simply switched. 

From my observations, SASI is not as aggressive on the spectrum of interaction with certain "Bulk" as Cyren is. Bulk Emails (Getting Marketing Email Campaign) seems to be a problem in your installation? If you look into the Emails going through, could a additional RBL help? You can add RBLs as well to the List, if you want. 

I did not review your case (as i am not a support engine nor product manager, i do not have the permissions to do so), so i want to simply give some insights. You should continue discussing this further within the case. 

If SASI is not using probable vs. confirmed, then this is a problem as well. We usually set confirmed spam to drop and probable spam to quarantine. This worked pretty well, since sometimes you get false prositives and probable spam wasn't so much to look through. Now we are forced to send practically everything to quarantine, since there is no confirmed spam anymore, so you get all the most obvious spam mails in your quarantine and have to look through everything of it. Our customers are not happy about that.

Speaking of "bulk", you're probably right. Our customers get a lot bulk spam messages, I've send some to the support and they basically said "that's not spam, that's marketing". Now what should I tell our customers? From their point of view (and mine as well) the spam engine is just worse. It doesn't really matter why an unwanted marketing mail is getting through, because it is still unwanted and was successfully  filtered out before. It's not this particular customer either, since I see the very same lower detection rate with my home version, with our internal own XG and on our other customers with mail protection licenses. Sure, the drop in detection rate is not the same with all of them, since they do not receive spam in the same amount, but overall it is visible if you just check the same appliance with both firmwares. Some get 10% more spam and some over 50%.

In regards of UTM I've just see something like this: https://community.sophos.com/utm-firewall/f/mail-protection-smtp-pop3-antispam-and-antivirus/129655/spam-recognition-really-bad-since-sasi

We do not have many customers left with UTM and mail protection. Basically it's just one and he "complained" as well. I should say that this customer is really relaxed in this regard, he just said that they are getting more spam. Other than that I've heard from two other admins in bigger installations (both running UTM 230 active clusters) that they receive a lot more spam since the engine changed, but that they kinda got it sorted out with own rules, regex etc. but that shouldn't be the solution, should it?

In regards of RBL: I had zen.spamhaus.org included additionally, but was forced to remove it since a lot of legitimate mails got dropped with it. I'm not sure why, though.

If SASI is not using probable vs. confirmed, then this is a problem as well. We usually set confirmed spam to drop and probable spam to quarantine. This worked pretty well, since sometimes you get false prositives and probable spam wasn't so much to look through. Now we are forced to send practically everything to quarantine, since there is no confirmed spam anymore, so you get all the most obvious spam mails in your quarantine and have to look through everything of it. Our customers are not happy about that.

Speaking of "bulk", you're probably right. Our customers get a lot bulk spam messages, I've send some to the support and they basically said "that's not spam, that's marketing". Now what should I tell our customers? From their point of view (and mine as well) the spam engine is just worse. It doesn't really matter why an unwanted marketing mail is getting through, because it is still unwanted and was successfully  filtered out before. It's not this particular customer either, since I see the very same lower detection rate with my home version, with our internal own XG and on our other customers with mail protection licenses. Sure, the drop in detection rate is not the same with all of them, since they do not receive spam in the same amount, but overall it is visible if you just check the same appliance with both firmwares. Some get 10% more spam and some over 50%.

In regards of UTM I've just see something like this: https://community.sophos.com/utm-firewall/f/mail-protection-smtp-pop3-antispam-and-antivirus/129655/spam-recognition-really-bad-since-sasi

We do not have many customers left with UTM and mail protection. Basically it's just one and he "complained" as well. I should say that this customer is really relaxed in this regard, he just said that they are getting more spam. Other than that I've heard from two other admins in bigger installations (both running UTM 230 active clusters) that they receive a lot more spam since the engine changed, but that they kinda got it sorted out with own rules, regex etc. but that shouldn't be the solution, should it?

In regards of RBL: I had zen.spamhaus.org included additionally, but was forced to remove it since a lot of legitimate mails got dropped with it. I'm not sure why, though.

Most countries have some law regulations to be able to opt-out of Bulk (DSGVO in Germany) So your customer should be able to easily (one click) opt-out of Bulk emails. This should be the better approach than rely on a Spam Engine to find Bulk marketing emails in the first place. 

I just wanted to point out: Cyren is more aggressive in its pattern compared to SASI. While Cyren seems to strike most Bulk as "spam", which is odd to me in the first place, you should be able to resolve this by manually opt-out of those Bulk lists. This is an issue from back in the days, where you could not opt-out. Correct me, if i am wrong, but on those examples, Labs told you "this is a marketing email", can you opt-out with a Link? If the customer does not want to get those emails, the opt-out mechanism should be a better solution. Then we can focus on the False-negative Spams detection and check, what is undetected. 

And my point is: Nobody escalated this at this point back in the day with UTM 9.607. This is more than a year ago. 
My suggestion would be: Let the customer, you have with those numbers first opt-out of those marketing Emails and then we look at the remaining false-negative Emails and check them for some pattern. 

LuCar Toni please look at good SASI engine and tell me if this for you look like marketing email or SPAM email? For me and all our users this are SPAM or PHISHING emails:

With 18.5 MR2 we don't get emails like this.

We use "Standard RBL Services", "Premium RBL Services" and additional RBL lists but do not help.

We use Spamhouse:

• zen.spamhaus.org
• sbl.spamhaus.org
• pbl.spamhaus.org
• xbl.spamhaus.org

And Baracudacentral:

b.barracudacentral.org

Some mails have unsubscribe options, others don't. Sure, I can ask if they are willing to unsubscribe from everything where it is possible, but be prepared that they will complain about unsubscribing from dozens of mailing lists that were never subscribed in the first place. An option to set the level of aggression used by the spam engine would be of more help I guess.

Some customers had other spam solutions in use before we switched them to Sophos. All was fine, then you guys changed the engine (again: nobody asked for this!) and now we, as your partner, have to deal with customer complaints about how much better everything was "before we switched to Sophos", thank you, really.

To circle back on this: A vendor will likely use there own engine, if possible. After looking at the recent years, i could not spot a problem in any sense to perform such a change. 

Dreamcatcher feel free to discuss this in more detail with your sales rep. I am quite sure, we can do something like a migration to Central Email for your customers. 

About your feedback DejanBukovec I cannot understand nor say anything about your examples due the fact i cannot understand this language. You should highlight those examples to Labs, they can give you a proper statement about those emails. 

I already explained this. Central Email is not an option for our customers. We will not offer them the same solution (email protection) from the same company that first took their money and then, without any explanation, degraded the product they had already purchased, just to sell them the same thing for even more. If we have to sell a dedicated product anyway, I see no point in staying with Sophos at all.

I can only highly recommend you to look out the open discussion about the opportunity. I cannot assist anymore than offer the dialog. 

Hi Dreamcatcher

do you have some news to share on the spam detection topic?

No, nothing new. They will investigate the issue and get back to me, but so far they haven't even admitted that there is a problem at all.

I had opened support case about spam, which has been closed yesterday by tehnician with reason inactivity(I send last email response on friday and he close it on monday)... For us case is not closed until engine will not work as before with older engine which has been reason to use and BUY this subscription but Sophos know that better than us.

After upgrade to MR1 AND Sophos Lab actions to include our samples into engine spam detection in last 10 days has been a lot of better. We receive only national(slovenian) spams. but today we start receiving some phishing mails which are not in national language:

Hi Dejan Bukovec, sorry to hear your case was closed prior to resolution. Reviewing the case, the email response you sent on Friday did not come through. Please let me know and I can reopen a case.