Currently all incoming emails which matches a RBL are blocked. Can this be changed to accept but warn using the SPAM marker?

If anyone cares I had a guy here who is expert at shell scripting come up with the perfect working solution.  You want to get all the domains you've ever sent email to instead of received from because received from will be a lot of mailing list and marketing crap.

as root, create the file /root/build_whitelist.bash with these lines but replace @yourdomain.com in the 4 places:

rm -f /root/master_whitelist
rm -f /root/master_whitelist.csv

zgrep 'F=.*@yourdomain.com> R='  /var/log/smtp/20*/*/smtp-* |grep -oP 'R=.*@.*> Acc' |grep -oP '@.*' |sed s/\>\ Acc// |sort |uniq -c |sort -nr >> /root/master_whitelist

echo "subject:Master Whitelist" > master_whitelist.csv
echo "To:you@yourdomain.com,someone@yourdomain.com" >> master_whitelist.csv
cat /root/master_whitelist |awk '{printf "%s,*%s\n",$1,$2}' >> master_whitelist.csv


cat /root/master_whitelist.csv | sendmail -F "UTM - Whitelist" -f noreply@yourdomain.com -t

Now when you rush "bash build_whitelist.bash"  it will build a list of all the domains you have received email from, then count and sort the list so that each domain appears once with a count in front seperated by a comma with the *@domain.com so you can easily paste into excel to use for import into UTM once you have a quick once over.  You obviously will get thousands of lines, but you can ignore like everything with a count below like 20 or 50.  This script will email you the results so you dont have to scp or try to get the file yourself.

Sample output:

4952,*@us.ibm.com
3187,*@vnet.ibm.com
1852,*@customer1.org
1644,*@avnet.com
1223,*@lenovo.com
960,*@ingrammicro.com
849,*@brocade.com
604,*@customer2.org
545,*@gmail.com
517,*@customer3.com
512,*@customer4.com
450,*@customer5.com
442,*@customer6.com
411,*@customer7.com
351,*@customer8.com
324,*@customer9.com
276,*@customer10.com
269,*@aerohive.com
264,*@customer11.com
262,*@customer12.com
257,*@synnex.com
250,*@sbcglobal.net
246,*@nutanix.com
223,*@customer13.com
203,*@aol.com
193,*@customer14.com
190,*@customer15.com
186,*@yahoo.com
163,*@customer16.com
160,*@customer17.com
158,*@bluechasm.com
155,*@vmware.com
151,*@customer18.com
141,*@customer19.com
136,*@zerto.com
...
...

continued... goes on and on...

Now the whole reason I'm adding these to whitelist for just RBL is due to the ever increasing outsourcing of email to large hosting providers like messagelabs and mictrosoft office 365.  I'm getting an increasing amount of people come to me each day and say customers are telling me that some of their emails get rejected but most go through.  Well you ask them which domains and they say, well I've had 5 different companies tell me that.  Turns out these large hosting providers always seem to have a small few of their sever IPs on known blacklist like sorbs.net on any given day.  When the users sends email it goes out via one server ip in the pool and like 2% of the time it goes out one that's on a known blacklist.  Evidently these hosting providers aren't checking to see if their servers are blacklisted publicly and removed them from their outgoing pool.  They normally get the blacklist removed, but it seems to take hours, and by that time they have a couple other which have become blacklisted.  I don't know of any other solution to this issue.

Thanks, Jason!  Nice touches with the grep on multiple years and months, the sed "surgery" and sendmail.  This is going into my personal list of command line tricks with the following changes...

A star 'nix admin I know once told me never to put anything in /root that didn't need to be there.  If you look at the various KnowledgeBase articles that recommend writing a new file, it's always put into the /home directory.  I know that she would recommend starting the script with cd /home, deleting all occurrences of /root/ in the script and putting create_whitelist.bash in /home.  I just ran this successfully on our UTM.

I don't think I'll be using this whitelist technique with any of our customers though.  I'd rather have the folks whose emails are getting bounced complain to their email providers. [;)]

Cheers - Bob