mails rejected "server certificate not trusted" (dnssec behind utm)

Update: It seems that the DANE authentication request is not put through UTM to the mail server behind; zone informations such as _443._tcp.mail.mailserver.com or 25._tcp.mail.mailserver.com can not be received from the mail server in order that the clients can validate the certificates when DNSSEC is enabled. It "stops" somewhere at the UTM and communicates the UTM internal self-signed certs. External clients or mail services do x-check those values and sometimes deny the communication. How can this be avoided? In short: how can UTM do its Email-Protection while in front of a fully set up and working mail server with DNSSEC enabled on itself? 

Hi, Mike, and welcome to the UTM Community!

To the rest of the world, your UTM is the MTA for you, not your mail server.  Do you have DNSSEC configured for it?

Cheers - Bob

Hi, Bob! Thanks a lot for your warm welcome!

Well, its kind of both - in some way. UTM at least communicates with the outside world. But: Everything is configured on the mail server itself, including certificates and DNSSEC, zone infos etc. So DNSSEC deployment is based on the mail server, not UTM - UTM was put later in front of the mail server as a bridge (transparent mode / etc. (see above)) to filter mail traffic - does this make any sense? Although not an expert at all, i have the feeling that the "chain of trust" is broken. The goal would be that the "outside world" check the certificates and entries on the mail server itself in regards to DNSSEC / DANE (and not already on the UTM), but this two settings may exclude themselves, as DNSSEC / DANE is intended to also avoid DNS takeover and UTM is in some way just doing "man-in-the-middle" tasks ...

I wonder if there is a digital signature on the packet, and utm is altering the packet in a way tbat breaks the signature

DouglasFoster I think this indeed could be the case in some way. DNSSEC is new territory for me - but an interesting one. As far as I understood, if DNSSEC is enabled, some other domains/E-Mail providers (such as gmx) have a DANE SMTP validation implemented, thus check the actual certificate of the primary MX host against the DANE TLSA records. If it does not match, the system seems "compromised" and validation of certification fails. It seems that it is now x-checking the UTM certificate against the DANE TLSA records, resulting in a mismatch. So i can only think of the following solutions: I. Implement DNSSEC on UTM directly (honestly no idea how this should be done) II. Disable DNSSEC completely or III. in some way set-up the UTM that it can check the traffic without interfering in certificates. I would prefer III but any creative idea how this could be done? As far as i can think, the UTM would in some way need to allow requests (from outside) to pass on to (the directory on) the mail server where the CA keys are stored - and thus not using the keys of the UTM itself...

brunomc thanks. In this case just implementing a commercial certificate for the UTM would not work I guess, as long as the DANE TLSA records would still not match the certificate.

You are ahead of me on this, because I have only read about DNS SEC.   But I hope that I can help you break the problem into segments.

I will assume that your MX is mail.mycompany.com and the other MX is mail.theircompany.com.

My understanding is that you have DNS SEC enabled for your domain.   That means that your MX has to have a commercial certificate for mail.mycompany.com, or others may not trust you.   You said this was all working.

If UTM is entered into the equation, I can imagine two valid configurations for retaining that trust:

• Transparent mode.   UTM and your MX share an external IP address.   Your certificate for mail.mycompany.com has to be replicated (complete with private key) to UTM, and UTM has to be configured to offer that certificate on email connections.   Your DNS SEC records should be unchanged, because they link the DNS name (mail.mycompany.com) to the external IP address, all of which are unchanged and all of which are validated by the commercial certificate.   This could fall apart, as I said before, if UTM fails to provide a correct certificate chain (with intermediate cert and without root cert)
  
  
• Proxy mode - option 1.  UTM could take over the external IP and name of your MX.   Again, no DNS SEC changes should be required.
  
  
• Proxy mode, option 2 - UTM operates with a different DNS Name and matching commercial certificate, and a different external IP.   You publish the UTM information as an additional MX record.   You control which MX device is available to the public by enabling or disabling an external IP.   Other organizations should trust your outbound traffic whether it comes from your MX directly or UTM.

For inbound traffic:

• UTM begins checking for DNS SEC validity simply by enabling the feature, and ensuring that it sends all DNS SEC lookups to a DNS SEC-enabled name server. 
• However for your MX, all traffic will now be coming from UTM, and because UTM is doing man-in-the-middle decrypt-and-scan,  the connection from UTM to MX will be encrypted using a UTM-issued certificate, which will not pass DNS SEC inspection.   Therefore, your MX needs to be configured to ignore DNS SEC, probably by directing it to a DNS server which does not handle DNS SEC.

So I am guessing that your problem is that your MX is still checking DNS SEC, and when it sees an incoming connection from UTM claiming to be mail.theircompany,com, it detects the fraud and rejects the connection.

DouglasFoster said:

So I am guessing that your problem is that your MX is still checking DNS SEC, and when it sees an incoming connection from UTM claiming to be mail.theircompany,com, it detects the fraud and rejects the connection.

@DouglasFoster You are right! And not sure at all if I am ahead of you - would say you are at least on the same level. What I did some hours ago before i saw your post is exactly option one (transparent mode) you mentioned:

I replicated the complete certificate (with private key and the intermediate certificate, installed root cert etc.), created the *.p12 file and uploaded it to UTM. DNSSEC has been unchanged, as this would involve and be done at e.g. the DN registrar anyway. Did some initial testing and so far it looks better. Some mail providers that do DANE SMTP validation start to deliver mails.

Interim Conclusion: Inbound mail from DANE SMTP mail providers now seem to work. Outbound mail can still not be delivered, but the reply changed from "not trusted" to "not verified": Diagnostic-Code: X-Postfix; Server certificate not verified.

Probably missed something out.

Will do some more testing the coming week. Now it is time for Kubrick`s space odyssey 2001.

I confirm that, besides making some progress, the error "Server certificate not verified" persists and outbound mail to providers that do DANE SMTP validation get rejected with this message.

I would think that if both ends are DNSSEC-enabled, that there would be some mutual validation on both incoming and outgoing connections.   Consequently, I am surprised that you only have problems one direction.

Sometimes when debugging, it is important to go back to basics.   At the risk of sounding trite, here are some things to check:

Send to a non-DNSSEC recipient and check the email header logs.

1) Is outbound mail going out on the IP address that you expect?   

2) Are the assserted SMTP HELO hostname, certificate name, and reverse-DNS identities all consistent and what you intend?  

3) Are you using a wildcard or a named certificate?   If a wildcard, are you incorrectly trying to use it to cover multiple levels?  

(NOTE:  "*.mycompany.com" is not valid for "host1.division1.mycompany.com")

Use a downloaded OpenSSL kit to test SMTP with STARTTLS from the outside:

3) Do you get the certificate chain that you expect?  

4) Does it connect?  What ciphersuite is used for the connection, and is it a current one?

Hope this helps.

Miscellaneous, which requires contacting a communication partner:

4) For the sites that are rejecting you, do they recognize and trust your Certificate Authority, or is the root certificat the problem?

Really appreciate your efforts DouglasFoster. As Conan Doyle once said: "There is nothing more deceptive than an obvious fact." Your approach is fully justified. 

Although with the risk to complicate things, i need to give again the following information: the mail server (mail.mailserver.com) is behind the UTM (UTM in bridge mode and SMTP in transparent mode with host name mailserver.com).

To answer your question:

1) Is outbound mail going out on the IP address that you expect?  
Yes. The IP Address is the mail servers IP.

2) Are the assserted SMTP HELO hostname, certificate name, and reverse-DNS identities all consistent and what you intend?  
No. I think here lies a problem. I will get into that more detailed below. (*)

3) Are you using a wildcard or a named certificate?   If a wildcard, are you incorrectly trying to use it to cover multiple levels?  
No wildcard. Let`s Encrypt as a CA for mail.mailserver.com, mailserver.com and www.mailserver.com (3 single certs).

Use a downloaded OpenSSL kit to test SMTP with STARTTLS from the outside:
3) Do you get the certificate chain that you expect?  
SSL Tool gives all (three) green light and no errors or warnings.

4) Does it connect?  What ciphersuite is used for the connection, and is it a current one?
Yes. Ciphersuite: TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384


(*) Going back to point two again:

We remember that I replicated (completely) the certs for mail.mailserver.com from the mail server to UTM and UTM is offering those now. This helped me to get rid of the error "server certificate not trusted" and the error changed to "server certificate not verified". Indeed, outbound mails to Email providers with implemented DANE SMTP validation started to get through, but no inbound messages from them ever arrived ("server certificate not verified"). Btw it works perfectly fine in both directions with Email providers that do not have DANE SMTP validation).

Having said that, I continued with some IP checks for sending mail servers and sent an email from the mail server over to MultiRBLValli. FCrDNS Test all fine (green), Sender Address Tests all fine (green), but with "HELO/EHLO" Tests I run into a FAIL:

The HELO/EHLO string "mailserver.com" is a valid host- or domainname OK
IP Addresses for "mailserver.com"  :   99.999.99.999 OK
At least one IP address of the DNS lookup for "mailserver.com" matches to the connecting IP 99.999.99.999 OK
The HELO/EHLO string "mailserver.com" doesn't match to one of the names in the rDNS from connecting IP Failed
 
While not sure if this is the final problem, it could be a lead. So whats happening here?

The rDNS 99.999.99.999 points to the mail server behind the UTM at mail.mailserver.com - that's correct imho. When using MxToolbox, it safely says for mailserver.com that SMTP Reverse DNS is fine - 99.999.99.999 resolves to mail.mailserver.com.

But when using the MultiRBLValli to check outbound mails directly, the "fail" above shows up. Found out UTM hostname is set to mail.mailserver.com (so the certs do match) and in E-Mail Protection (Advanced - Advanced Settings - SMTP hostname), the hostname is set to mailserver.com (and not: mail.mailserver.com).

Interim Conclusion: There is a mismatch between the rDNS entry and the "received from" in the mail header.Guess this is because UTM is "intercepting", acts as a MTA as we know and modifies the information.

No solution so far: Can not change the SMTP hostname on UTM to mail.mailserver.com - this results immediately in an "Undelivered Mail Returned to Sender" answer from the mail server MTA: "mail for emailaddress@email.com loops back to myself". It could be that third party mail servers who enabled DNSSEC do strict checking and this mismatch could result in the "server certificate not verified". But this goes beyond my knowledge for sure. So still stuck with the rejected mails.

Really appreciate your efforts DouglasFoster. As Conan Doyle once said: "There is nothing more deceptive than an obvious fact." Your approach is fully justified. 

Although with the risk to complicate things, i need to give again the following information: the mail server (mail.mailserver.com) is behind the UTM (UTM in bridge mode and SMTP in transparent mode with host name mailserver.com).

To answer your question:

1) Is outbound mail going out on the IP address that you expect?  
Yes. The IP Address is the mail servers IP.

2) Are the assserted SMTP HELO hostname, certificate name, and reverse-DNS identities all consistent and what you intend?  
No. I think here lies a problem. I will get into that more detailed below. (*)

3) Are you using a wildcard or a named certificate?   If a wildcard, are you incorrectly trying to use it to cover multiple levels?  
No wildcard. Let`s Encrypt as a CA for mail.mailserver.com, mailserver.com and www.mailserver.com (3 single certs).

Use a downloaded OpenSSL kit to test SMTP with STARTTLS from the outside:
3) Do you get the certificate chain that you expect?  
SSL Tool gives all (three) green light and no errors or warnings.

4) Does it connect?  What ciphersuite is used for the connection, and is it a current one?
Yes. Ciphersuite: TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384


(*) Going back to point two again:

We remember that I replicated (completely) the certs for mail.mailserver.com from the mail server to UTM and UTM is offering those now. This helped me to get rid of the error "server certificate not trusted" and the error changed to "server certificate not verified". Indeed, outbound mails to Email providers with implemented DANE SMTP validation started to get through, but no inbound messages from them ever arrived ("server certificate not verified"). Btw it works perfectly fine in both directions with Email providers that do not have DANE SMTP validation).

Having said that, I continued with some IP checks for sending mail servers and sent an email from the mail server over to MultiRBLValli. FCrDNS Test all fine (green), Sender Address Tests all fine (green), but with "HELO/EHLO" Tests I run into a FAIL:

The HELO/EHLO string "mailserver.com" is a valid host- or domainname OK
IP Addresses for "mailserver.com"  :   99.999.99.999 OK
At least one IP address of the DNS lookup for "mailserver.com" matches to the connecting IP 99.999.99.999 OK
The HELO/EHLO string "mailserver.com" doesn't match to one of the names in the rDNS from connecting IP Failed
 
While not sure if this is the final problem, it could be a lead. So whats happening here?

The rDNS 99.999.99.999 points to the mail server behind the UTM at mail.mailserver.com - that's correct imho. When using MxToolbox, it safely says for mailserver.com that SMTP Reverse DNS is fine - 99.999.99.999 resolves to mail.mailserver.com.

But when using the MultiRBLValli to check outbound mails directly, the "fail" above shows up. Found out UTM hostname is set to mail.mailserver.com (so the certs do match) and in E-Mail Protection (Advanced - Advanced Settings - SMTP hostname), the hostname is set to mailserver.com (and not: mail.mailserver.com).

Interim Conclusion: There is a mismatch between the rDNS entry and the "received from" in the mail header.Guess this is because UTM is "intercepting", acts as a MTA as we know and modifies the information.

No solution so far: Can not change the SMTP hostname on UTM to mail.mailserver.com - this results immediately in an "Undelivered Mail Returned to Sender" answer from the mail server MTA: "mail for emailaddress@email.com loops back to myself". It could be that third party mail servers who enabled DNSSEC do strict checking and this mismatch could result in the "server certificate not verified". But this goes beyond my knowledge for sure. So still stuck with the rejected mails.