Firewall Traffic gets routed in full-tunnel IPsec VPN

Hi Tobias Kuhnert : Thank you for reaching out to Sophos community team. What is the route precedence set over XG as of now? 

The default route precedence in 18.0 is set to static, SD-WAN policy routes, and VPN. Can you please check the below KBA for reference steps and set the route precedence to it's default V18 settings, if it not set.
 
https://support.sophos.com/support/s/article/KB-000037964?language=en_US

https://docs.sophos.com/nsg/sophos-firewall/18.0/Help/en-us/webhelp/onlinehelp/nsg/sfos/learningContents/ConfiguringRoutePrecedence.html

Note: You may test it in odd hours for safer side to avoid any outage or issue.

console> system route_precedence show
Routing Precedence:
1. Static routes
2. SD-WAN policy routes
3. VPN routes

Hi Tobias Kuhnert : Thanks for sharing this output. Please share below command output here.

1) #route -n

2) console> sy ipsec route show

SG230_WP02_SFOS 18.0.4 MR-4# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.64.0.0 0.0.0.0 255.255.255.0 U 0 0 0 PortE0
10.64.1.0 0.0.0.0 255.255.255.0 U 0 0 0 PortE0.2
10.64.2.0 0.0.0.0 255.255.255.0 U 0 0 0 PortE0.3
10.64.3.0 0.0.0.0 255.255.255.0 U 0 0 0 PortE0.4
10.255.0.0 0.0.0.0 255.255.255.0 U 0 0 0 GuestAP
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 PortE3
172.16.64.144 0.0.0.0 255.255.255.248 U 0 0 0 PortE1

console> sy ipsec_route show
tunnelname host/network netmask
... list is empty

Tobias Kuhnert : Thanks for sharing the above requested output. How many active WAN gateway present under "WAN Link Manager" over XG ? Please share snapshot if possible. (Navigation Path : Configure > Network > WAN Link Manager)

Can you please share below output as well? 

#ip route show table multilink 

That will help us to confirm default route is via which interface and gateway on XG currently.

first is of course to have a WAN interface configured

What is the way the packets go when you do the packet capture in GUI Diagnostic and ping google DNS? are they really sent into the tunnel?

we only have 1 wan gateway.

Interface:

SG230_WP02_SFOS 18.0.4 MR-4# ip route show table multilink
default via 172.16.64.145 dev PortE1 proto static

of course we have a wan interace.

yes, they are sent into the tunnel.

of course we have a wan interace.

yes, they are sent into the tunnel.

Hi Tobias Kuhnert,

Please share the screenshot of the local and remote network configured with the IPsec connection. 

If you added any in the remote network, that doesn't require IPsec system routes to send ICMP traffic from the firewall through the tunnel. 

As LHerzog said, run packet capture from the GUI and share the screenshot with us. 

Thanks,

i can't share any more screenshot with sensitive data. please see text below:

i have configured the following networks in the VPN connection. This is the only VPN connection configured on the device.
remote subnet: any
local subnet: 10.64.0.0/21

no additional/manual routes have been added to the system.

packet capture (10.64.0.1 is the IP of the firewall):

SG230_WP02_SFOS 18.0.4 MR-4# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss

as already mentioned route lookup from diagnostics page:

The problem is, SFOS is exactly doing what you tell the system to do. Your Traffic matches the IPsec route: 10.64.0.1 to ANY. Therefore it gets routed to the tunnel. 

You would have to create a new SNAT Rule, MASQ and use another IP, which does not match the IPsec route, this could resolve this issue. 

Check the System for IPsec NAT Rules. https://support.sophos.com/support/s/article/KB-000035848?language=en_US

you can narrow it down from the CLI also:

Checked this on a small XG with probably similar configuration as yours:

XG106_XN01_SFOS 18.0.4 MR-4# ip route get 8.8.8.8
8.8.8.8 via 10.1.254.2 dev Port2 table gw1 src 10.1.254.1 uid 0
    cache

So WAN in on Port 2 and Gw ist 10.1.254.2  -> look uid0

XG106_XN01_SFOS 18.0.4 MR-4# ip rule list
0:      from all lookup local            <- uid0
51:     from all fwmark 0x4001 lookup gw1
52:     from all fwmark 0x200 lookup routeipsec0
53:     from all lookup main
150:    from all fwmark 0x8001 lookup gw1
151:    from 10.1.254.1 lookup wanlink1
220:    from all iif lo lookup 220
221:    from all lookup multilink
32766:  from all lookup main
32767:  from all lookup default

XG106_XN01_SFOS 18.0.4 MR-4# ip route show table 0
default via 10.1.254.2 dev Port2 table wanlink1 proto static src 10.1.254.1
prohibit default table wanlink1 proto static metric 1
default via 10.1.254.2 dev Port2 table gw1 proto static
prohibit default table gw1 proto static metric 1
default dev ipsec0 table routeipsec0 scope link
default via 10.1.254.2 dev Port2 table multilink proto static

So yes, probably there is a MASQ rule and SNAT missing.

Fw Rule:

NAT rule 4:

The problem is:

With SFOS 17.5.x everything worked. - since 18.0.x the problem exists as described.

i want the firewall initiated traffic to get sent out to the default gateway (not in the tunnel) as it worked before. With the current behavior i cannot use the firewall as DNS server, because it is trying to resolve everything trough the tunnel and the firewall on the other side does not allow this communication. I cannot change this!

Not sure how your KB article will help here.

and with this version change the NAT binding has changed. So you need to review your settings in NAT.

we were lucky - our migrated NAT rules worked after migration

i get your point, but why would i need to configure NAT rules for firewall initiated traffic.

we had those "sys-traffic-nat" CLI commands for that earlier - are they obsolet now?

regarding the packet capture - why is the firewall even initiating the traffic from the LAN interface when the shortest way would be to iniate from the WAN interface.

btw:

of course i do have a default NAT rule:

I did run the same ip commands as you. they look different:

SG230_WP02_SFOS 18.0.4 MR-4# ip route get 8.8.8.8
8.8.8.8 dev ipsec0 table 220 src 10.64.0.1 uid 0
cache

SG230_WP02_SFOS 18.0.4 MR-4# ip rule list
0: from all lookup local
50: from all lookup main
51: from all fwmark 0x4001 lookup gw1
52: from all fwmark 0x200 lookup routeipsec0
150: from all fwmark 0x8001 lookup gw1
151: from 172.16.64.148 lookup wanlink1
220: from all iif lo lookup 220
221: from all lookup multilink
32766: from all lookup main
32767: from all lookup default

SG230_WP02_SFOS 18.0.4 MR-4# ip route show table 0
default via 172.16.64.145 dev PortE1 table wanlink1 proto static src 172.16.64.148
prohibit default table wanlink1 proto static metric 1
default via 172.16.64.145 dev PortE1 table gw1 proto static
prohibit default table gw1 proto static metric 1
default dev ipsec0 table routeipsec0 scope link
default via 172.16.64.145 dev PortE1 table multilink proto static
default dev ipsec0 table 220 scope link src 10.64.0.1 <-maybe this is the issue?

So you have a NAT route. Thats good.

what does  

ip route show table wanlink1

show you?

And what happens, if you edit and re-safe the config of PortE1 without change? Be aware of a VPN reconnect.

"default dev ipsec0 table 220 scope link src 10.64.0.1 <-maybe this is the issue?"

that is the cause of your issue

Edit: but this is only for table 220 - the IPSec Networks (ip route show table 220 (don't post the result))