mails rejected "server certificate not trusted" (dnssec behind utm)

Update: It seems that the DANE authentication request is not put through UTM to the mail server behind; zone informations such as _443._tcp.mail.mailserver.com or 25._tcp.mail.mailserver.com can not be received from the mail server in order that the clients can validate the certificates when DNSSEC is enabled. It "stops" somewhere at the UTM and communicates the UTM internal self-signed certs. External clients or mail services do x-check those values and sometimes deny the communication. How can this be avoided? In short: how can UTM do its Email-Protection while in front of a fully set up and working mail server with DNSSEC enabled on itself? 

Hi, Mike, and welcome to the UTM Community!

To the rest of the world, your UTM is the MTA for you, not your mail server.  Do you have DNSSEC configured for it?

Cheers - Bob

There is a known issue that the utm certificate published for userportal and webadmin does not include the intermediate certificate.  There is one complicated remedy for this problem, posted elsewhere in this forum, but probably unknown to most Sophos support staff.  I am inferring that you have a commercial certificate, and that your certificate chain needs to be correct.

I also asume that the smtp proxy uses the same certificate as webportal and will have the same chain problem.   This may or may not be your problem as I dont know how DNSSEC fits into all this.

Hi, Bob! Thanks a lot for your warm welcome!

Well, its kind of both - in some way. UTM at least communicates with the outside world. But: Everything is configured on the mail server itself, including certificates and DNSSEC, zone infos etc. So DNSSEC deployment is based on the mail server, not UTM - UTM was put later in front of the mail server as a bridge (transparent mode / etc. (see above)) to filter mail traffic - does this make any sense? Although not an expert at all, i have the feeling that the "chain of trust" is broken. The goal would be that the "outside world" check the certificates and entries on the mail server itself in regards to DNSSEC / DANE (and not already on the UTM), but this two settings may exclude themselves, as DNSSEC / DANE is intended to also avoid DNS takeover and UTM is in some way just doing "man-in-the-middle" tasks ...

I wonder if there is a digital signature on the packet, and utm is altering the packet in a way tbat breaks the signature

hi,

you will need a commercial certificate or the utm as it is functioning as an MTA. (server authenticates against utm not your mailserver) mailcheap has cheap certs or use one of the free ones and your issue should be resolved) or disable tls (i guess in your case you do not wish to do that).

as for dnssec is does not seem to be your issue as your issue is server certificate not trusted / Diagnostic-Code: X-Postfix; Server certificate not trusted (in the maillogs) 

However like balfson mentioned to be correct your mx should just point to the UTM (its a mta in your mailserver chain now) (ie internet MTA UTM > MTA mailserver)

DouglasFoster I think this indeed could be the case in some way. DNSSEC is new territory for me - but an interesting one. As far as I understood, if DNSSEC is enabled, some other domains/E-Mail providers (such as gmx) have a DANE SMTP validation implemented, thus check the actual certificate of the primary MX host against the DANE TLSA records. If it does not match, the system seems "compromised" and validation of certification fails. It seems that it is now x-checking the UTM certificate against the DANE TLSA records, resulting in a mismatch. So i can only think of the following solutions: I. Implement DNSSEC on UTM directly (honestly no idea how this should be done) II. Disable DNSSEC completely or III. in some way set-up the UTM that it can check the traffic without interfering in certificates. I would prefer III but any creative idea how this could be done? As far as i can think, the UTM would in some way need to allow requests (from outside) to pass on to (the directory on) the mail server where the CA keys are stored - and thus not using the keys of the UTM itself...

brunomc thanks. In this case just implementing a commercial certificate for the UTM would not work I guess, as long as the DANE TLSA records would still not match the certificate.

You are ahead of me on this, because I have only read about DNS SEC.   But I hope that I can help you break the problem into segments.

I will assume that your MX is mail.mycompany.com and the other MX is mail.theircompany.com.

My understanding is that you have DNS SEC enabled for your domain.   That means that your MX has to have a commercial certificate for mail.mycompany.com, or others may not trust you.   You said this was all working.

If UTM is entered into the equation, I can imagine two valid configurations for retaining that trust:

• Transparent mode.   UTM and your MX share an external IP address.   Your certificate for mail.mycompany.com has to be replicated (complete with private key) to UTM, and UTM has to be configured to offer that certificate on email connections.   Your DNS SEC records should be unchanged, because they link the DNS name (mail.mycompany.com) to the external IP address, all of which are unchanged and all of which are validated by the commercial certificate.   This could fall apart, as I said before, if UTM fails to provide a correct certificate chain (with intermediate cert and without root cert)
  
  
• Proxy mode - option 1.  UTM could take over the external IP and name of your MX.   Again, no DNS SEC changes should be required.
  
  
• Proxy mode, option 2 - UTM operates with a different DNS Name and matching commercial certificate, and a different external IP.   You publish the UTM information as an additional MX record.   You control which MX device is available to the public by enabling or disabling an external IP.   Other organizations should trust your outbound traffic whether it comes from your MX directly or UTM.

For inbound traffic:

• UTM begins checking for DNS SEC validity simply by enabling the feature, and ensuring that it sends all DNS SEC lookups to a DNS SEC-enabled name server. 
• However for your MX, all traffic will now be coming from UTM, and because UTM is doing man-in-the-middle decrypt-and-scan,  the connection from UTM to MX will be encrypted using a UTM-issued certificate, which will not pass DNS SEC inspection.   Therefore, your MX needs to be configured to ignore DNS SEC, probably by directing it to a DNS server which does not handle DNS SEC.

So I am guessing that your problem is that your MX is still checking DNS SEC, and when it sees an incoming connection from UTM claiming to be mail.theircompany,com, it detects the fraud and rejects the connection.

DouglasFoster said:

So I am guessing that your problem is that your MX is still checking DNS SEC, and when it sees an incoming connection from UTM claiming to be mail.theircompany,com, it detects the fraud and rejects the connection.

@DouglasFoster You are right! And not sure at all if I am ahead of you - would say you are at least on the same level. What I did some hours ago before i saw your post is exactly option one (transparent mode) you mentioned:

I replicated the complete certificate (with private key and the intermediate certificate, installed root cert etc.), created the *.p12 file and uploaded it to UTM. DNSSEC has been unchanged, as this would involve and be done at e.g. the DN registrar anyway. Did some initial testing and so far it looks better. Some mail providers that do DANE SMTP validation start to deliver mails.

Interim Conclusion: Inbound mail from DANE SMTP mail providers now seem to work. Outbound mail can still not be delivered, but the reply changed from "not trusted" to "not verified": Diagnostic-Code: X-Postfix; Server certificate not verified.

Probably missed something out.

Will do some more testing the coming week. Now it is time for Kubrick`s space odyssey 2001.

I confirm that, besides making some progress, the error "Server certificate not verified" persists and outbound mail to providers that do DANE SMTP validation get rejected with this message.