Sophos Firewall: v19.0 MR1: Feedback and experiences

Hi,

I received an email about 10 minutes ago, it has a number of attachments which are all blank, further the email does not show in logviewer -> email. There a e two entries at the time of the email with no details and show possible spam.

Ian

Further to the message above, I sent the originator a question about the attachment and he advised other recipient were able to view it. I took the iPhone out of the wifi and was able to see the attachment which is an mp4 file.. So something is broken in the latest mail functions in v19.0.1.

I have not made any changes to my mail settings, least of all have the ability to hide messages from XG log viewer -> email report. I have received other messages which do not show up in log viewer. This is an issue that was fixed a couple of versions ago.

Ian

Ian

No, and I don't see why I even should open a case anyway? Did you change anything to improve the spam detection rate and therefore think that the bad detection rate us and our customers experiencing is a bug? If you didn't, then the only thing opening a case with you guys does is stealing a lot of my time, like on almost every case I ever opened with you before.

I'm going to open a case next week after seeing that 19.0 MR1 is still bad compared to 18.5 MR2, which I have absolutely no doubt about, as the way you answered me before shows me that no one at Sophos even acknowledges that there is indeed a massive problem with the spam detection rate. I'm pretty sure that is going to be "nice". Me doing your work, me being your beta-tester, me getting angry since you won't change anything, me giving you support access for many days, just to see nobody even connected to the appliance at all. Yeah, it's going to be a lot of "fun", as always with you guys.

After you create the Case, feel free to post your Case ID to this thread, so we can looking into this particular case. 

Oh I will, you can count on me.

I feel you... You speak from the bottom of my heart. Thank you dreamcatcher!

Spam detection on 19.0 MR1 seems to work as expected, meaning it's as bad as with 18.5 MR4. I've checked the timeframe since I've upgraded from 18.5 MR2 to 19.0 MR1 and compared it to the last days.

considered period is 19:30-10:30 each.

18.5 MR2
24.07.-25.07. -> 30 spam mails detected
25.07.-26.07. -> 23 spam mails detected
26.07.-27.07. -> 37 spam mails detected

19.0 MR1
27.07.-28.07. -> 6 spam mails detected

Wow. Seems like I don't even need to wait one week to open a case, good work Sophos, good work.

Here you go: 05553951

Hi Dreamcatcher, thank you for sharing your case ID. Please allow me to follow up with our engineering and development team and I will post any updates on this thread as we know more.

Hey Karlos, sure, do as you like.

I checked the last day and compared it to the last full day with 18.5 MR2 running, here are the results:

My customer already got back to me, saying that since the update much more spam is going through. This is not funny by any means!

Just for my interest. Which SASI Version does the customer currently use? 

2022-08-02.11:40:15 MESSAGE [Main] [ precompile.cpp:583] Downloaded file /sdisk/sasi/asdb.delta is verified with checksum..
2022-08-02.11:40:17 MESSAGE [Main] [ engine.cpp:845] Database loaded of version: 2022.8.2.91519
2022-08-02.11:40:17 MESSAGE [Main] [ precompile.cpp:761] [Precompile thread]: Signatures are reloaded with latest delta and verified with checksum
of new signatures.

That is my latest version (extracted by /log/sasi.log). 

Just for my interest. Which SASI Version does the customer currently use? 

2022-08-02.11:40:15 MESSAGE [Main] [ precompile.cpp:583] Downloaded file /sdisk/sasi/asdb.delta is verified with checksum..
2022-08-02.11:40:17 MESSAGE [Main] [ engine.cpp:845] Database loaded of version: 2022.8.2.91519
2022-08-02.11:40:17 MESSAGE [Main] [ precompile.cpp:761] [Precompile thread]: Signatures are reloaded with latest delta and verified with checksum
of new signatures.

That is my latest version (extracted by /log/sasi.log). 

Here you go:

2022-08-02.11:40:20 MESSAGE    [Main] [          precompile.cpp:583] Downloaded file /sdisk/sasi/asdb.delta is verified with checksum..
2022-08-02.11:40:23 MESSAGE    [Main] [              engine.cpp:845] Database loaded of version: 2022.8.2.91519
2022-08-02.11:40:23 MESSAGE    [Main] [          precompile.cpp:761] [Precompile thread]: Signatures are reloaded with latest delta and verified with checksum of new signatures.

Update.

Hi, 

For me it looks like this :(

19.0 MR1

Well, without a comparison to 18.5 MR2 it's not expressive. If you route a lot clean messages through the firewall, it will always look like this:

Dreamcatcher Could you attach the SupportAccessID to your case please? 

And also: To report False-Negative, you can forward the email to : https://support.sophos.com/support/s/article/KB-000033422?language=en_US

You do not need to send them to support. 

YOu can find the access ID in case 05553951. Support asked me to send them spam mails that got through, so I did.

1 week 18.5 MR2 vs. 1 week 19.0 MR1.

Not only is the overall detection rate significantly lower, "Confirmed spam" vs. "Probable spam" is pretty much useless too now with the new engine, since there literally is no "confirmed spam" anymore:

How something like this makes it to production state is beyond me.

I did look into the UTM reviews as well. There is no real feedback about the bad positioning of SASI in UTM, i could found now. No escalations either. And UTM uses the same SASI version / pattern like SFOS. It is correct, in UTM you can write your own "blocklists" based on words etc. But is this the workaround "all utm customers" use? I found it hard to believe, that all UTM customers (home vs business) have no raise any attention to the SASI engine. 

See: https://community.sophos.com/utm-firewall/f/mail-protection-smtp-pop3-antispam-and-antivirus/128189/9-706---anti-spam-engine-changed-to-sasi Most feedback was about the old hardware and the support of the engine. 

BTW: SASI works differently on Probable Spam vs Spam. Confirmed does not exist in SASI, so the labels simply switched. 

From my observations, SASI is not as aggressive on the spectrum of interaction with certain "Bulk" as Cyren is. Bulk Emails (Getting Marketing Email Campaign) seems to be a problem in your installation? If you look into the Emails going through, could a additional RBL help? You can add RBLs as well to the List, if you want. 

I did not review your case (as i am not a support engine nor product manager, i do not have the permissions to do so), so i want to simply give some insights. You should continue discussing this further within the case. 

If SASI is not using probable vs. confirmed, then this is a problem as well. We usually set confirmed spam to drop and probable spam to quarantine. This worked pretty well, since sometimes you get false prositives and probable spam wasn't so much to look through. Now we are forced to send practically everything to quarantine, since there is no confirmed spam anymore, so you get all the most obvious spam mails in your quarantine and have to look through everything of it. Our customers are not happy about that.

Speaking of "bulk", you're probably right. Our customers get a lot bulk spam messages, I've send some to the support and they basically said "that's not spam, that's marketing". Now what should I tell our customers? From their point of view (and mine as well) the spam engine is just worse. It doesn't really matter why an unwanted marketing mail is getting through, because it is still unwanted and was successfully  filtered out before. It's not this particular customer either, since I see the very same lower detection rate with my home version, with our internal own XG and on our other customers with mail protection licenses. Sure, the drop in detection rate is not the same with all of them, since they do not receive spam in the same amount, but overall it is visible if you just check the same appliance with both firmwares. Some get 10% more spam and some over 50%.

In regards of UTM I've just see something like this: https://community.sophos.com/utm-firewall/f/mail-protection-smtp-pop3-antispam-and-antivirus/129655/spam-recognition-really-bad-since-sasi

We do not have many customers left with UTM and mail protection. Basically it's just one and he "complained" as well. I should say that this customer is really relaxed in this regard, he just said that they are getting more spam. Other than that I've heard from two other admins in bigger installations (both running UTM 230 active clusters) that they receive a lot more spam since the engine changed, but that they kinda got it sorted out with own rules, regex etc. but that shouldn't be the solution, should it?

In regards of RBL: I had zen.spamhaus.org included additionally, but was forced to remove it since a lot of legitimate mails got dropped with it. I'm not sure why, though.

Most countries have some law regulations to be able to opt-out of Bulk (DSGVO in Germany) So your customer should be able to easily (one click) opt-out of Bulk emails. This should be the better approach than rely on a Spam Engine to find Bulk marketing emails in the first place. 

I just wanted to point out: Cyren is more aggressive in its pattern compared to SASI. While Cyren seems to strike most Bulk as "spam", which is odd to me in the first place, you should be able to resolve this by manually opt-out of those Bulk lists. This is an issue from back in the days, where you could not opt-out. Correct me, if i am wrong, but on those examples, Labs told you "this is a marketing email", can you opt-out with a Link? If the customer does not want to get those emails, the opt-out mechanism should be a better solution. Then we can focus on the False-negative Spams detection and check, what is undetected. 

And my point is: Nobody escalated this at this point back in the day with UTM 9.607. This is more than a year ago. 
My suggestion would be: Let the customer, you have with those numbers first opt-out of those marketing Emails and then we look at the remaining false-negative Emails and check them for some pattern.