Outbound mails fail with greylisting - no second attempt

Please tell us how your configuration differs from Basic Exchange setup with SMTP Proxy.

Cheers - Bob

The smart host setting in the SMTP Connector in Exchange Manager must point to the "Internal (Address)" of the Astaro. If you already had a different setting in Exchange, pointing at an external smart host that you must use, you must transfer that to the Astaro's 'Smarthost settings' at the bottom of the 'Advanced' tab.

Check

Other than that, here's the basic Exchange installation by tab:

• - 'Global': "Simple mode"  Check
• - 'Routing': Add yourdomain.com to 'Domains', choose 'Route by' "Static host list" and add the host definition for your Exchange server. Check 'Verify recipients' "with callout." I have "Verify in AD", but that should not be a problem here
• - 'AntiVirus': should be OK as delivered
• - 'AntiSpam': 'Reject at SMTP Time' "Confirmed Spam."  Check Check 'Use recommended RBLs'. Check For your 'Spam filter' selections, click on the ? at the top of the page to read the help and decide for yourself. All of the 'Advanced anti-spam features' should be selected. I usually deselect 'Greylisting', but others here like it.  I have "Invalifd HELO/RDNS" and "SPF"; "Greylisting" and "BATV" are unchecked
• - 'Exceptions': should be OK as delivered Well, these have substantially grown over the years and are the reason why a few of the advanced features above have meanwhile been disabled (consequently, it should be possible to prune the exception list again - insome long winter evening perhaps)
• - 'Relaying': If your Exchange server also receives mail via an upstream host, you'll need to add the upstream host to the list at the top. N/A  Add the host definition for Exchange to 'Host-based relay';Check don't include your internal network. Check Do leave 'Authenticated relay' empty. Check At the bottom, select to have outgoing mail scanned. This is unchecked
  
• - 'Advanced': Don't select 'Use transparent mode'! In 'Advanced settings', modify if needed the 'SMTP hostname' and/or 'Postmaster address' Check

Don't forget to disable any DNAT that was forwarding inbound SMTP to Exchange or to a different anti-spam device as that takes precedence over the SMTP Proxy. If you want outbound mail to leave with the IP of an Additional Address named "Mail," you will need to 'SNAT : Any -> SMTP -> Internet : from External [Mail] (Address)'. Check

The smart host setting in the SMTP Connector in Exchange Manager must point to the "Internal (Address)" of the Astaro. If you already had a different setting in Exchange, pointing at an external smart host that you must use, you must transfer that to the Astaro's 'Smarthost settings' at the bottom of the 'Advanced' tab.

Check

Other than that, here's the basic Exchange installation by tab:

• - 'Global': "Simple mode"  Check
• - 'Routing': Add yourdomain.com to 'Domains', choose 'Route by' "Static host list" and add the host definition for your Exchange server. Check 'Verify recipients' "with callout." I have "Verify in AD", but that should not be a problem here
• - 'AntiVirus': should be OK as delivered
• - 'AntiSpam': 'Reject at SMTP Time' "Confirmed Spam."  Check Check 'Use recommended RBLs'. Check For your 'Spam filter' selections, click on the ? at the top of the page to read the help and decide for yourself. All of the 'Advanced anti-spam features' should be selected. I usually deselect 'Greylisting', but others here like it.  I have "Invalifd HELO/RDNS" and "SPF"; "Greylisting" and "BATV" are unchecked
• - 'Exceptions': should be OK as delivered Well, these have substantially grown over the years and are the reason why a few of the advanced features above have meanwhile been disabled (consequently, it should be possible to prune the exception list again - insome long winter evening perhaps)
• - 'Relaying': If your Exchange server also receives mail via an upstream host, you'll need to add the upstream host to the list at the top. N/A  Add the host definition for Exchange to 'Host-based relay';Check don't include your internal network. Check Do leave 'Authenticated relay' empty. Check At the bottom, select to have outgoing mail scanned. This is unchecked
  
• - 'Advanced': Don't select 'Use transparent mode'! In 'Advanced settings', modify if needed the 'SMTP hostname' and/or 'Postmaster address' Check

Don't forget to disable any DNAT that was forwarding inbound SMTP to Exchange or to a different anti-spam device as that takes precedence over the SMTP Proxy. If you want outbound mail to leave with the IP of an Additional Address named "Mail," you will need to 'SNAT : Any -> SMTP -> Internet : from External [Mail] (Address)'. Check

I think the configuration problem may be in the mail server.

Cheers - Bob

How so?

Does that mean "451 4.7.1 Greylisting in action, please come back later" is not a suitable reply?
Sifting through the logs, I do find a few successful deliveries after a "451 4.7.1". But indeed, the success seems always to be withoin a few seconds (i.e., try mx1 => "451 4.7.2", try mx2 => success - as if the secondary MX learns his greylisting database immediately from the primary). I have not yet found a case where this code is successfully used with "real" greylisting" (i.e., with success only after a significantly later queue run)

I am not all the way through my logs, but it seems that "really" successful greylisting work only with ifferent extended status codes, e.g. as below: Here we see a first round throgh two mail servers who both reply with "451 4.3.2"; after a few minutes, a second round of attempts has the same result; finally, the first attempt of the next round succeeds. Note that several "retry time not reached" events happen inbetween.

2020:06:10-10:09:58 firewall-2 exim-in[28468]: 2020-06-10 10:09:58 [10.x.x.x] F=<SOMEONE@HERE> R=<USER@ELSEWHERE> Accepted: from relay
2020:06:10-10:10:21 firewall-2 smtpd[28080]: SCANNER[28080]: id="1000" severity="info" sys="SecureMail" sub="smtp" name="email passed" srcip="10.x.x.x" from="SOMEONE@HERE" to="USER@ELSEWHERE" subject="redacted" queueid="1jivoS-0007Iu-K8" size="754243"
2020:06:10-10:13:48 firewall-2 exim-out[29937]: 2020-06-10 10:13:48 1jivoS-0007Iu-K8 SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host spamgateway.ELSEWHERE [88.x.x.78]: 451 4.3.2 Please try again later
2020:06:10-10:13:49 firewall-2 exim-out[29937]: 2020-06-10 10:13:49 1jivoS-0007Iu-K8 SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host mail.ELSEWHERE [88.x.x.75]: 451 4.3.2 Please try again later
2020:06:10-10:13:49 firewall-2 exim-out[29936]: 2020-06-10 10:13:49 1jivoS-0007Iu-K8 == USER@ELSEWHERE R=dnslookup T=remote_smtp defer (-44): SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host mail.FARAWAY [217.7.3.18]: 451 4.3.2 Please try again later
2020:06:10-10:14:26 firewall-2 exim-out[30337]: 2020-06-10 10:14:26 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:15:04 firewall-2 exim-out[30804]: 2020-06-10 10:15:04 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:17:33 firewall-2 exim-out[32531]: 2020-06-10 10:17:33 1jivoS-0007Iu-K8 SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host spamgateway.ELSEWHERE [88.x.x.78]: 451 4.3.2 Please try again later
2020:06:10-10:17:33 firewall-2 exim-out[32531]: 2020-06-10 10:17:33 1jivoS-0007Iu-K8 SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host mail.ELSEWHERE [88.x.x.75]: 451 4.3.2 Please try again later
2020:06:10-10:17:33 firewall-2 exim-out[32530]: 2020-06-10 10:17:33 1jivoS-0007Iu-K8 == USER@ELSEWHERE R=dnslookup T=remote_smtp defer (-44): SMTP error from remote mail server after RCPT TO:<USER@ELSEWHERE>: host mail.FARAWAY [217.7.3.18]: 451 4.3.2 Please try again later
2020:06:10-10:17:34 firewall-2 exim-out[32563]: 2020-06-10 10:17:34 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:17:36 firewall-2 exim-out[32585]: 2020-06-10 10:17:36 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:18:02 firewall-2 exim-out[32678]: 2020-06-10 10:18:02 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:19:00 firewall-2 exim-out[417]: 2020-06-10 10:19:00 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:19:25 firewall-2 exim-out[510]: 2020-06-10 10:19:25 1jivoS-0007Iu-K8 == USER@ELSEWHERE routing defer (-51): retry time not reached
2020:06:10-10:20:01 firewall-2 exim-out[700]: 2020-06-10 10:20:01 1jivoS-0007Iu-K8 => USER@ELSEWHERE P=<SOMEONE@HERE> R=dnslookup T=remote_smtp H=spamgateway.ELSEWHERE [88.x.x.78]:25 X=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256 C="250 2.0.0 05A8K0fi013568-05A8K0fk013568 Message accepted for delivery"

So does this mean that "451 4.3.2" works for greylisting and "451 4.7.1" does not? As they share the same 451 basic status code, the difference must come from the extended error code.
According to RFC3463, section 2, a 4.xxx.xxx code means "Persistent Transient Failure [...] some temporary condition has caused abandonment or delay of attempts to send the message. [...] sending in the future may be successful." To me, this suggests that UTM should try future delivery until the retry timeout kicks in.
In section 3.4, we learn that "X.3.2   System not accepting network messages:  The host on which the mailbox is resident is not accepting messages.  Examples of such conditions include an immanent shutdown, excessive load, or system maintenance.  This is useful for both permanent and persistent transient errors."
And in section 3.8, we learn "X.7.1   Delivery not authorized, message refused: The sender is not authorized to send to the destination.  This can be the result of per-host or per-recipient filtering.  This memo does not discuss the merits of any such filtering, but provides a mechanism to report such.  This is useful only as a permanent error."
Note that the last sentence only opines on usefulness, there is no MUST or the like involved. Also, the sentence does not say that every X.7.1 I see coming in from a remote end has to be treated by me as permanent error; rather the text says that the sender of a X.7.1 status could may want to consider this useful only in the form of a 5.7.1 instead of a 4.7.1 or 2.7.1. Alas, they do send it as 4.7.1, and so we should treat it as 4.x.x

Two questions arise:

• Why does UTM apparently allow the "7.1" part in a "451 4.7.1" to implicitly override the severity to a "5xx 5.x.x" and abandon retries?
• Why is "451 4.7.1" so widespread in the wild with greylisting implementations

Or is it? I made a little statistic about all "4xx 4.x.x" codes in recent logs:

      1 421 4.1.1
      1 421 4.4.1
      1 450 4.1.7
      1 451 4.1.8
      1 451 4.4.25
      1 452 4.2.1
      2 450 4.7.3
      2 451 4.4.27
      4 452 4.1.1
      6 421 4.7.1
      6 451 4.1.1
      6 451 4.2.1
     14 450 4.1.0
     22 450 4.2.1
     24 421 4.4.0
     24 451 4.3.0
     30 450 4.7.25
     40 451 4.7.0
     45 450 4.5.8
     46 451 4.3.5
     56 450 4.1.8
     88 451 4.7.5
     93 450 4.0.0
    109 451 4.5.0
    112 422 4.7.1
    131 450 4.3.2
    161 451 4.3.2
    186 421 4.7.0
    190 450 4.1.1
    246 421 4.3.2
   1148 451 4.7.1
   1655 454 4.7.1
   1877 450 4.7.1
   2441 450 4.2.0
   4403 452 4.3.1

As indicated, 4.7.1 is apparently the most logged extended code (though with different basic codes 450, 454, 451, and others), whereas 4.3.2 is logged much fewer time.
Judging by the cleart text messages, it seems that "450 4.2.0" is also used for greylisting (and as successfully as "451 4.3.2"). The winner, "452 4.3.1" is usually accompanied by texts like "Insufficient system resources", so probably not used as greylisting - but it does lead to retries as expected.

In my opinion, UTM should treat any "4xx 4.x.x" as temporary failure that warrants a retry.
At any rate, it seems that I cannot use the descriptions from RFC3463 to argue with the remote postmaster that they are not compliant and ought to reconfigure their system to use "451 4.7.1" ...

If this is happening with only a single company, maybe they could skip greylisting for email from your company's mail server.

If it's more than one company, maybe the SMTP PostgreSQL data base is broken.  Eight years ago, the following rebuilt it - I'm not sure today, but you might try it...

/var/mdw/scripts/smtp stop
dropdb -U postgres smtp
createdb -U postgres smtp
/var/mdw/scripts/smtp start

Any luck with that?

Cheers - Bob