Hello Community,
we have a very nice and puzzling issue with our Sophos UTM and SMTP relay to a smarthost over an IPsec tunnel.
Problem - short version:
Our Exchange server forwards mails to the UTM as smarthost. The UTM forwards the mails to a second (non-public) smarthost via an IPsec tunnel. As we have to Uplinks (UPLINK1 & UPLINK2) everything works, if the IPsec tunnel is build up via UPLINK2. If we use UPLINK1 for this tunnel, no email will be forwarded.
Environment:
- Our Mailserver Exchange 2010
- Sophos UTM SG310 (HA-Clustered active/passive) - V9.506-2
- external Mailgateway: Proventia Mail Gateway - IP: 10.11.22.33
Uplinks:
- UPLINK1:
Telekom Business DSL with fixed IP
BinTEC xDSL-Bridge
Sophos: PPPoE connection on eth5 - fixed IP
- UPLINK2:
Vodafone Cable Business with fixed IP
Compal Cable-Router (small /30 transfernetwork with public IPs - so called "Bridgemode at Vodafone")
Sophos: Ethernet connection on eth1 - fixed IP and RouterIP as gateway-address
VPN-Tunnel:
IPsec site-2-site, AES-256, PSK bound to the interface (I need a nother routing metric for that tunnel)
Routing entries to other sites networks available and working
SNAT:
Due to the fact, we have to uplinks in case of an outage of one, and the fact, the IPsec tunnel doesn't allow direkt traffic of the IP-addresses of uplink-initerfaces I created a SNAT rule, which the UTM uses, if the smtp proxy forwards mails into the tunnel to the external mail-gateway.
SNAT: "Uplink Primary Addresses" -> any -> remote-network (uses IP 172.22.55.66, which is allowed via the tunnel)
Mail routing as smarthost - NOT transparent(!):
Exchange 2010 -[LAN]-> Sophos UTM -[IPsec]-> Proventia remote mailgateway
1st scenario:
Exchange 2010 -[LAN]-> Sophos UTM -[IPsec via UPLINK2]-> Proventia remote mailgate
Everything works as exspected!
2nd scenario:
Exchange 2010 -[LAN]-> Sophos UTM -[IPsec via UPLINK1]-> Proventia remote mailgate
Mails stuck in smtp-spool queue and will not be forwarded
LOGS:
SMTP-Proxy Live-Log:
--------------------
2018:02:02-08:03:51 ptgfw001-1 exim-out[15154]: 2018-02-02 08:03:51 1ehUia-0001GE-92 == john@doe.de R=smarthost_route T=smarthost_smtp defer (-18): Remote host 10.11.22.33 [10.11.22.33] closed connection in response to end of data
2018:02:02-08:04:00 ptgfw001-1 exim-out[15317]: 2018-02-02 08:04:00 Start queue run: pid=15317
2018:02:02-08:04:00 ptgfw001-1 exim-out[15319]: 2018-02-02 08:04:00 1ehUia-0001GE-92 == john@doe.de R=smarthost_route T=smarthost_smtp defer (-53): retry time not reached for any host
2018:02:02-08:04:00 ptgfw001-1 exim-out[15317]: 2018-02-02 08:04:00 End queue run: pid=15317
2018:02:02-08:04:00 ptgfw001-2 exim-out[22820]: 2018-02-02 08:04:00 Start queue run: pid=22820
2018:02:02-08:04:00 ptgfw001-2 exim-out[22820]: 2018-02-02 08:04:00 End queue run: pid=22820
SMTP-Debug Log:
---------------
2018-02-02 08:03:50 MASTER[15110]: 0 messages in work queue, 0 scanners are running
2018-02-02 08:03:52 QMGR[15117]: 1ehUia-0001GE-92 deferred - updated msglog in messages table
2018-02-02 08:04:00 MASTER[15110]: 0 messages in work queue, 0 scanners are running
2018-02-02 08:04:00 MASTER[15110]: 1 messages in output queue with age >2 minutes, running additional queue runner (max. 2)
2018-02-02 08:04:01 MASTER[15110]: queue runner 15317 terminated.
2018-02-02 08:04:10 MASTER[15110]: 0 messages in work queue, 0 scanners are running
2018-02-02 08:04:20 MASTER[15110]: 0 messages in work queue, 0 scanners are running
tcpdump -vvXXeni any host 172.22.55.66:
---------------------------------------
08:03:51.476258 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59709, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e93d 4000 3f06 1734 0a0a 4f4f E..(.=@.?..4..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:03:51.729742 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59710, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e93e 4000 3f06 1733 0a0a 4f4f E..(.>@.?..3..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:03:52.249914 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59711, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e93f 4000 3f06 1732 0a0a 4f4f E..(.?@.?..2..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:03:53.296608 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59712, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e940 4000 3f06 1731 0a0a 4f4f E..(.@@.?..1..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:03:55.381357 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59713, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e941 4000 3f06 1730 0a0a 4f4f E..(.A@.?..0..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:03:59.553365 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59714, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e942 4000 3f06 172f 0a0a 4f4f E..(.B@.?../..OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
08:04:07.905048 In ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 63, id 59715, offset 0, flags [DF], proto TCP (6), length 40)
10.11.22.33.25 > 172.22.55.66.36909: Flags [F.], cksum 0xefe4 (correct), seq 206, ack 129, win 913, length 0
0x0000: 0000 0200 0000 0000 0000 0000 0000 0800 ................
0x0010: 4500 0028 e943 4000 3f06 172e 0a0a 4f4f E..(.C@.?.....OO
0x0020: 0a09 d7fc 0019 902d 94ba 97df e814 dc09 .......-........
0x0030: 5011 0391 efe4 0000 P.......
What have I testet so far?
- telnet via port 25 from Exchange server to UTM smarthost:
Email created on UTM will be forwarded without any issues - telnet via port 25 from UTM to Proventia mailgateway:
no issues - Sending mails directly from Exchange to Proventia Mail-gateway via IPsec tunnel
Mail-Routing:
Exchange 2010 -[IPsec]-> Proventia remote mailgateway
-> Working without issues - Replaced the xDSL-Modem with a router, doing the PPPoE-connection and forwarding all incoming connections to the UTM (exposed host)
-> Mail forwarding via IPsec tunnel no issue (that was my former networksetup, but the router had to less power in case of a complete failover
Conclusion:
Everything (every service) is working well over the IPsec connection. But if we want to forward mails over the UTM as smarthost, it will only work, if the uplink-interface is UPLINK2. On UPLINK1 (PPPoE session), the mails stuck.
Has any one an idea ?
Is there an issue with ppp-interfaces?
Many thanks!
BR,
Florian
Changes:
5.2.2018: removed empty lines (cleanup); added Mailrouting as smarthost
This thread was automatically locked due to age.
