Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 8 replies
  • Answers 1 answer
  • Subscribers 5 subscribers
  • Views 10612 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Smarthost TLS problem

kerobra_retired

I've had a very strange issue today...

Our customer has a UTM working as a transparent proxy (https: URL filtering only) in the configuration: LAN -> UTM -> transfer network -> other firewall -> internet

The firewall rule on the UTM is Any:Any:Any sincte the other firewall is handling the external traffic. On the UTM the POP3 proxy is active, SMTP proxy is NOT used.


Customer has an Exchange 2013 Server that is collecting it's mail with POP-Beamer, using the POP3 Proxy of the UTM. Sending of emails worked via smarthost send connector that used "SMTP (port 25). His mail provider changed his smtp server to SMTP (port 587) and the exchange server stopped sending mails because of certificate validation issues. I found a blog post that was handling the same error message and the author managed to get it up again by enabling netshell winhttp proxy settings. It seemed to have something to do with ocsp verification for the TLS certificate of the new mail server.

I tried excluding the exchange server's traffic completely via skip list entry, no luck. What should I say, I gave it a try and set the netshell proxy to use the (transparent) proxy as dedicated proxy and voilá, the sh..t worked! I'm fine with the "solution" for the moment but there remains a big big question about it...

I can't understand why the verification of the TLS certificate doesn't work in transparent proxy mode with a skiplist entry. Can anybody explain this to me?



This thread was automatically locked due to age.
Parents
  • 0

    Coarse language is not motivating.   Please write professionally.

    You said SMTP proxy is off, so it should be no surprise that the SMTP skip list had no effect.  SMTP proxy is not intended for authenticated SMTP, and I am doubtful that it could work, especially if encryption is also implemented.  It has no effect on POp proxy.

    Here is my theory:

    The SMTP change activated encryption for the tirst time.m, so it us encryption that created the problem, not the served name or IP.

    With POP inspection, UTM connects to the server and the client (Exchange send connector) connects to UTM, which impersonates the remote server using a UTM-generated certificate.

    With encryoted SMTP, the client connects directly to the remote server and sees the real server certificate.   Exchange is apparently clever enough to see that if it gets two very different certificates when only one server is supposed to be involved, then something is amiss.

    By putting Exchange into proxy mode, both connections are going through UTM and as a result Exchange is happy.

    Overall, it is a very nonstandard configuration and one should always expect trouble when using three independent products in ways that were unlikely to have been intended or tested during development of any of those products.

    Winhttp proxy is a pretty hidden feature of Windows.   Responding to an explicit proxy request for SMTP client traffic is an apoarently hidden and  undocumented feature of UTM.   Using Exchange as an email client is a rare implementation of Exchange, even if it is supported.   Using traffic filtering between server and client is an unusual configuration for the remote mail server.  All of those anomalies are why I do not like the configuration.

    However, I guess cudos are in order for you, because you found a way to keep the contraption working.

Reply
  • 0

    Coarse language is not motivating.   Please write professionally.

    You said SMTP proxy is off, so it should be no surprise that the SMTP skip list had no effect.  SMTP proxy is not intended for authenticated SMTP, and I am doubtful that it could work, especially if encryption is also implemented.  It has no effect on POp proxy.

    Here is my theory:

    The SMTP change activated encryption for the tirst time.m, so it us encryption that created the problem, not the served name or IP.

    With POP inspection, UTM connects to the server and the client (Exchange send connector) connects to UTM, which impersonates the remote server using a UTM-generated certificate.

    With encryoted SMTP, the client connects directly to the remote server and sees the real server certificate.   Exchange is apparently clever enough to see that if it gets two very different certificates when only one server is supposed to be involved, then something is amiss.

    By putting Exchange into proxy mode, both connections are going through UTM and as a result Exchange is happy.

    Overall, it is a very nonstandard configuration and one should always expect trouble when using three independent products in ways that were unlikely to have been intended or tested during development of any of those products.

    Winhttp proxy is a pretty hidden feature of Windows.   Responding to an explicit proxy request for SMTP client traffic is an apoarently hidden and  undocumented feature of UTM.   Using Exchange as an email client is a rare implementation of Exchange, even if it is supported.   Using traffic filtering between server and client is an unusual configuration for the remote mail server.  All of those anomalies are why I do not like the configuration.

    However, I guess cudos are in order for you, because you found a way to keep the contraption working.

Children
  • 0 in reply to DouglasFoster

    I apologize for the coarse language of yesterday evening. I've had two 14h days of troubleshooting behind me and my nerves were at their limits.

     

    I'm not happy with this kind of setup, too but the customer defines the budget for his IT, not me. The customer is located right in the middle of nowhere where no good internet connection ist available. So routing MX record directly to him, an email gateway or the Sophos SMTP proxy simply is no option.

    Today the problem popped up on several other customers that all use the same mailbox provider (1&1, also called IONOS). I investigated the problem with a bit more calmness today. I tried the same things I did yesterday at another customer with a nearly identical setup. The dedicated proxy mode or winhttp proxy is NOT involved, because one customer where it appeared has no web protection license at all.

     

    I was able to isolate the problems to the send connector of the Exchange Server. If this is configured as "SmartHostAuthMechanism: BasicAuthRequireTLS" the mail sending fails completely. In the logfile for the send connector I see this:

    Remote certificate
    "CN=smtp.1und1.de, L=Montabaur, S=Rheinland-Pfalz, O=1&1 Telecommunication SE, C=DE",Certificate subject
    "CN=TeleSec ServerPass Class 2 CA, STREET=Untere Industriestr. 20, L=Netphen, PostalCode=57250, S=Nordrhein Westfalen, OU=T-Systems Trust Center, O=T-Systems International GmbH, C=DE",Certificate issuer name
    289B2D8A7DE5FEEEB5E117131748CC27,Certificate serial number
    ECAE7BFD0F1CF5F33853D6D81E9A37A90B6D9674,Certificate thumbprint
    smtp.1und1.de,Certificate alternate names
    "TLS protocol SP_PROT_TLS1_2_CLIENT negotiation succeeded using bulk encryption algorithm CALG_AES_256 with strength 256 bits, MAC hash algorithm CALG_SHA1 with strength 160 bits and key exchange algorithm CALG_ECDHE with strength 521 bits"
    Received certificate
    ECAE7BFD0F1CF5F33853D6D81E9A37A90B6D9674,Certificate thumbprint
    UntrustedRoot,Chain validation status

    which leads to a

    Remote Server at smtp.1und1.de (212.227.15.167) returned '451 4.4.0 Primary target IP address responded with: "454 4.7.5 Certificate validation failure." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was 212.227.15.167:25'

    error in the Exchange's mail queue. Using port 587 instead does not change anything. If I disable the auth option to 'BasicAuth' the mails can be delivered.


    What I still wasn't able to point out are the similarities between the customers. The mailbox provider would be the obvious target since we have different MS Exchange versions, different OS versions. But what stays against that theory is my customer from yesterday evening. There 'BasicAuthRequireTLS' is still active and mails are delivered.

    Does anyone know if there is some kind of "known thumbprints cache" on an Exchange server or how to verify the certificate chain manually? If this was be a https proxy problem I would say it's an "unable to get local issuer certificate" problem and I would know how to solve it, but not for mail :-(

    Gruß / Regards,

    Kevin
    Sophos CE/CA (XG+UTM), Gold Partner

  • 0 in reply to kerobra_retired

    Suggest you optain a copy of OpenSSL.   The official site only has source code, but the community forum has links to several people who have compiled versions.    I have used those compiled versions without known problems, but you need to decide whom to trust.   Install binaries into the program directory so you can have multiple copies installed at once if desired.

    The commsnd you want is

    openssl s_client -connect fqdn: port

    You may want to add

    -starttls SMTP

    UTM 9.5 and later uses 1.0.2j-fips, if my memory has not failed.   There are newer versions but they lack a fips option, which I assume is part of the reason that newer versions are not yet used in UTM.

    use -? for more command options.   The openssl website has full manual oages as well.

    You need to find out the certificate chain used by the server.  It may not be configured correctly.

    Unfortunately, I have not found any certificate test web site that supports alternate port testing.  Ask your certificate vendor if it can be done using their test tool.  If so, you can delay learning OpenSSL.

  • 0 in reply to kerobra_retired

    Hey, thanks for pointing out a solution!

    So I was working on exactly the same problem today.
    Error messages were exactly the same in my logs.

    It all came down to this certificate problem, that the SMTP Server at 1&1 didn't accept the authentication via TLS.
    Disabling the BasicAuthRequireTLS was helping me out here.
    In the logs the auth-procedure still looks exactly the same as before, but is not canceled once the BasicAuthRequiereTLS is deactivated.

    Had nothing to do with Sophos (it's not even installed on that system)

    The certificate we were using here was a simple self signed one. Maybe IONOS  / 1&1 policy towards those certificates changed?

    Haven't tried yet to sign a certificate with Let's Encrypt or any other service.
    Maybe someone wants to try that first and tell us the result? ;-)

    Next step in my case is to ask 1&1 what happened on their side of the connection.

    Btw: Timestamp when the first mail was stuck in  the mailing queu: 2019-01-17T11:15:01.913Z  (that's 10:10 German time)
    Maybe that helps finding out what happend to all your other customers?

    Cheers
     Alex

    ###  Keywords so this post shows up better in Google

    Exchange 2010 2013 2016 2019 smarthost 1&1 mails can't be sent not being delivered stuck in mail queue TLS SSL port 587  



Unfiltered HTML