Currently all incoming emails which matches a RBL are blocked. Can this be changed to accept but warn using the SPAM marker?

If you check your spam settings under Email Protection > SMTP > Antispam, if you have Reject at SMTP time set to anything other than "Off" then it will just drop the messages it marks as spam. Confirmed spam are the RBLs, Spam is both confirmed and the emails the UTM decides as spam when it does it's own checks.

In the Spam Filter, set both settings to Warn and create your Spam marker, or use default.

I advise keeping at least HELO, Greylisting and BATV turned on in the advanced antispam features. Strict RDNS and SPF check can false positive often because of setup on the senders side.

Reject at SMTP time is Off. Both actions are set to WARN. RBL still get rejected at SMTP time.

How much spam do you get with the recommended RBLs turned off? Is it like a nuclear spam missile?

lol, I would be scared to test that.

SMTP last 24 hours [ 2231 messages delivered. 2619 messages blocked (54%) ]
Malware quarantined/rejected: 3 SPF rejects: 1
Spam quarantined/rejected: 0 RBL rejects: 1570
Blacklist rejects: 0 BATV rejects: 0
Address verification rejects: 35 RDNS/HELO rejects: 1010

Phwoar, that's a fair bit of spam, definitely don't turn that off!

Only thing I can think of is to add the email domains your workforce mainly deal with (if you can find out/know) as an exception to the RBL checks. Unless you use other RBLs than the recommended, there's not much you can do to prevent the legitimate companies sending you emails getting rejected if they get added to an RBL.

A possible test, which would increase load on your system, is to turn off the RBLs and dial up the Reject at SMTP time to Spam. Then you're relying on the UTM to detect and identify the spam within those 2619 emails. Could be a worthy test for 15 minutes?

Confirmed spam doesn't seem to be the RBLs it seems.

Any idea on a command or script I could run to gather a list of domains that are frequently received or sent to? I could use something like that to build exceptions for RBL.

The logs you see are based inside the SQL database and I'm not even sure you can access it for exportation.

What you can do, you can set the amount of entries to 1000 in the mail manager and then highlight the tables text, copy it and paste it into an excel workbook. Do that however many times necessary to get all the data. Then doing some crafty sorting after removal of the data preceding the @domain.blah you might be able to structure a reasonable evaluation of your commonly contacted domains and create an explicit exception for them from the rbl.

Once you've removed the preceding data, sort by a-z then you can do an =countif(A:A,A1) on the column with domains. A:A would be the column your domains are stored in and A1 would be the current cell being Count-IFed. I use this a lot so I can manipulate marketing data manually to see what poor soul would be high on a call list because every action they take, every move they make, Google is watching them...

Hope that helps!

The logs you see are based inside the SQL database and I'm not even sure you can access it for exportation.

What you can do, you can set the amount of entries to 1000 in the mail manager and then highlight the tables text, copy it and paste it into an excel workbook. Do that however many times necessary to get all the data. Then doing some crafty sorting after removal of the data preceding the @domain.blah you might be able to structure a reasonable evaluation of your commonly contacted domains and create an explicit exception for them from the rbl.

Once you've removed the preceding data, sort by a-z then you can do an =countif(A:A,A1) on the column with domains. A:A would be the column your domains are stored in and A1 would be the current cell being Count-IFed. I use this a lot so I can manipulate marketing data manually to see what poor soul would be high on a call list because every action they take, every move they make, Google is watching them...

Hope that helps!

wow, thanks! very helpful!

Emile, I like to use the 'Search Log Files' tab to get the data to work on with your spreadsheet technique.  For example, to get just the lines with email addresses that have sent mail to our organization, I search on:

"email passed".*mediasoftusa.com" subject

To see the emails we have sent, I search on:

@mediasoftusa.com> R=<.* > Accepted: from relay

If you’re comfortable at the command line, you don’t need the spreadsheet for much. For example, the following will give a list of all sending domains from whom we received mail this December (thanks to AlanT for the last grep trick):

cd /home

zgrep '"email passed".*mediasoftusa.com" subject' /var/log/smtp/2019/*/*.gz >receivedlines

grep -oP '@.*?" to="' receivedlines|sort -n|uniq -c >senders

 Cheers - Bob

Bob, that is so cool and so swift, writing that down, much better than my horrifically manual procedure!

Fantastic as always!

If anyone cares I had a guy here who is expert at shell scripting come up with the perfect working solution.  You want to get all the domains you've ever sent email to instead of received from because received from will be a lot of mailing list and marketing crap.

as root, create the file /root/build_whitelist.bash with these lines but replace @yourdomain.com in the 4 places:

rm -f /root/master_whitelist
rm -f /root/master_whitelist.csv

zgrep 'F=.*@yourdomain.com> R='  /var/log/smtp/20*/*/smtp-* |grep -oP 'R=.*@.*> Acc' |grep -oP '@.*' |sed s/\>\ Acc// |sort |uniq -c |sort -nr >> /root/master_whitelist

echo "subject:Master Whitelist" > master_whitelist.csv
echo "To:you@yourdomain.com,someone@yourdomain.com" >> master_whitelist.csv
cat /root/master_whitelist |awk '{printf "%s,*%s\n",$1,$2}' >> master_whitelist.csv


cat /root/master_whitelist.csv | sendmail -F "UTM - Whitelist" -f noreply@yourdomain.com -t

Now when you rush "bash build_whitelist.bash"  it will build a list of all the domains you have received email from, then count and sort the list so that each domain appears once with a count in front seperated by a comma with the *@domain.com so you can easily paste into excel to use for import into UTM once you have a quick once over.  You obviously will get thousands of lines, but you can ignore like everything with a count below like 20 or 50.  This script will email you the results so you dont have to scp or try to get the file yourself.

Sample output:

4952,*@us.ibm.com
3187,*@vnet.ibm.com
1852,*@customer1.org
1644,*@avnet.com
1223,*@lenovo.com
960,*@ingrammicro.com
849,*@brocade.com
604,*@customer2.org
545,*@gmail.com
517,*@customer3.com
512,*@customer4.com
450,*@customer5.com
442,*@customer6.com
411,*@customer7.com
351,*@customer8.com
324,*@customer9.com
276,*@customer10.com
269,*@aerohive.com
264,*@customer11.com
262,*@customer12.com
257,*@synnex.com
250,*@sbcglobal.net
246,*@nutanix.com
223,*@customer13.com
203,*@aol.com
193,*@customer14.com
190,*@customer15.com
186,*@yahoo.com
163,*@customer16.com
160,*@customer17.com
158,*@bluechasm.com
155,*@vmware.com
151,*@customer18.com
141,*@customer19.com
136,*@zerto.com
...
...

continued... goes on and on...

Now the whole reason I'm adding these to whitelist for just RBL is due to the ever increasing outsourcing of email to large hosting providers like messagelabs and mictrosoft office 365.  I'm getting an increasing amount of people come to me each day and say customers are telling me that some of their emails get rejected but most go through.  Well you ask them which domains and they say, well I've had 5 different companies tell me that.  Turns out these large hosting providers always seem to have a small few of their sever IPs on known blacklist like sorbs.net on any given day.  When the users sends email it goes out via one server ip in the pool and like 2% of the time it goes out one that's on a known blacklist.  Evidently these hosting providers aren't checking to see if their servers are blacklisted publicly and removed them from their outgoing pool.  They normally get the blacklist removed, but it seems to take hours, and by that time they have a couple other which have become blacklisted.  I don't know of any other solution to this issue.

Thanks, Jason!  Nice touches with the grep on multiple years and months, the sed "surgery" and sendmail.  This is going into my personal list of command line tricks with the following changes...

A star 'nix admin I know once told me never to put anything in /root that didn't need to be there.  If you look at the various KnowledgeBase articles that recommend writing a new file, it's always put into the /home directory.  I know that she would recommend starting the script with cd /home, deleting all occurrences of /root/ in the script and putting create_whitelist.bash in /home.  I just ran this successfully on our UTM.

I don't think I'll be using this whitelist technique with any of our customers though.  I'd rather have the folks whose emails are getting bounced complain to their email providers. [;)]

Cheers - Bob