Firewall Traffic gets routed in full-tunnel IPsec VPN

Hi Tobias Kuhnert : Thank you for reaching out to Sophos community team. What is the route precedence set over XG as of now? 

The default route precedence in 18.0 is set to static, SD-WAN policy routes, and VPN. Can you please check the below KBA for reference steps and set the route precedence to it's default V18 settings, if it not set.
 
https://support.sophos.com/support/s/article/KB-000037964?language=en_US

https://docs.sophos.com/nsg/sophos-firewall/18.0/Help/en-us/webhelp/onlinehelp/nsg/sfos/learningContents/ConfiguringRoutePrecedence.html

Note: You may test it in odd hours for safer side to avoid any outage or issue.

console> system route_precedence show
Routing Precedence:
1. Static routes
2. SD-WAN policy routes
3. VPN routes

Hi Tobias Kuhnert,

Please share the screenshot of the local and remote network configured with the IPsec connection. 

If you added any in the remote network, that doesn't require IPsec system routes to send ICMP traffic from the firewall through the tunnel. 

As LHerzog said, run packet capture from the GUI and share the screenshot with us. 

Thanks,

i can't share any more screenshot with sensitive data. please see text below:

i have configured the following networks in the VPN connection. This is the only VPN connection configured on the device.
remote subnet: any
local subnet: 10.64.0.0/21

no additional/manual routes have been added to the system.

packet capture (10.64.0.1 is the IP of the firewall):

SG230_WP02_SFOS 18.0.4 MR-4# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss

as already mentioned route lookup from diagnostics page:

The problem is, SFOS is exactly doing what you tell the system to do. Your Traffic matches the IPsec route: 10.64.0.1 to ANY. Therefore it gets routed to the tunnel. 

You would have to create a new SNAT Rule, MASQ and use another IP, which does not match the IPsec route, this could resolve this issue. 

Check the System for IPsec NAT Rules. https://support.sophos.com/support/s/article/KB-000035848?language=en_US

you can narrow it down from the CLI also:

Checked this on a small XG with probably similar configuration as yours:

XG106_XN01_SFOS 18.0.4 MR-4# ip route get 8.8.8.8
8.8.8.8 via 10.1.254.2 dev Port2 table gw1 src 10.1.254.1 uid 0
    cache

So WAN in on Port 2 and Gw ist 10.1.254.2  -> look uid0

XG106_XN01_SFOS 18.0.4 MR-4# ip rule list
0:      from all lookup local            <- uid0
51:     from all fwmark 0x4001 lookup gw1
52:     from all fwmark 0x200 lookup routeipsec0
53:     from all lookup main
150:    from all fwmark 0x8001 lookup gw1
151:    from 10.1.254.1 lookup wanlink1
220:    from all iif lo lookup 220
221:    from all lookup multilink
32766:  from all lookup main
32767:  from all lookup default

XG106_XN01_SFOS 18.0.4 MR-4# ip route show table 0
default via 10.1.254.2 dev Port2 table wanlink1 proto static src 10.1.254.1
prohibit default table wanlink1 proto static metric 1
default via 10.1.254.2 dev Port2 table gw1 proto static
prohibit default table gw1 proto static metric 1
default dev ipsec0 table routeipsec0 scope link
default via 10.1.254.2 dev Port2 table multilink proto static

So yes, probably there is a MASQ rule and SNAT missing.

Fw Rule:

NAT rule 4:

The problem is:

With SFOS 17.5.x everything worked. - since 18.0.x the problem exists as described.

i want the firewall initiated traffic to get sent out to the default gateway (not in the tunnel) as it worked before. With the current behavior i cannot use the firewall as DNS server, because it is trying to resolve everything trough the tunnel and the firewall on the other side does not allow this communication. I cannot change this!

Not sure how your KB article will help here.

and with this version change the NAT binding has changed. So you need to review your settings in NAT.

we were lucky - our migrated NAT rules worked after migration

i get your point, but why would i need to configure NAT rules for firewall initiated traffic.

we had those "sys-traffic-nat" CLI commands for that earlier - are they obsolet now?

regarding the packet capture - why is the firewall even initiating the traffic from the LAN interface when the shortest way would be to iniate from the WAN interface.

btw:

of course i do have a default NAT rule:

I did run the same ip commands as you. they look different:

SG230_WP02_SFOS 18.0.4 MR-4# ip route get 8.8.8.8
8.8.8.8 dev ipsec0 table 220 src 10.64.0.1 uid 0
cache

SG230_WP02_SFOS 18.0.4 MR-4# ip rule list
0: from all lookup local
50: from all lookup main
51: from all fwmark 0x4001 lookup gw1
52: from all fwmark 0x200 lookup routeipsec0
150: from all fwmark 0x8001 lookup gw1
151: from 172.16.64.148 lookup wanlink1
220: from all iif lo lookup 220
221: from all lookup multilink
32766: from all lookup main
32767: from all lookup default

SG230_WP02_SFOS 18.0.4 MR-4# ip route show table 0
default via 172.16.64.145 dev PortE1 table wanlink1 proto static src 172.16.64.148
prohibit default table wanlink1 proto static metric 1
default via 172.16.64.145 dev PortE1 table gw1 proto static
prohibit default table gw1 proto static metric 1
default dev ipsec0 table routeipsec0 scope link
default via 172.16.64.145 dev PortE1 table multilink proto static
default dev ipsec0 table 220 scope link src 10.64.0.1 <-maybe this is the issue?

So you have a NAT route. Thats good.

what does  

ip route show table wanlink1

show you?

And what happens, if you edit and re-safe the config of PortE1 without change? Be aware of a VPN reconnect.

"default dev ipsec0 table 220 scope link src 10.64.0.1 <-maybe this is the issue?"

that is the cause of your issue

Edit: but this is only for table 220 - the IPSec Networks (ip route show table 220 (don't post the result))

"default dev ipsec0 table 220 scope link src 10.64.0.1 <-maybe this is the issue?"

that is the cause of your issue

Edit: but this is only for table 220 - the IPSec Networks (ip route show table 220 (don't post the result))

i think there is nothing sensitive in it:
SG230_WP02_SFOS 18.0.4 MR-4# ip route show table 220
default dev ipsec0 scope link src 10.64.0.1

any thoughts what to check or try next?

220 should print out all remote networks of your IPSec tunnel. But maybe in your case you do not need access from there to remote networks (e.g. HQ networks), only the other way around. So it may be empty.

The only difference in your config I could find is the routin precedence:

Default routing Precedence:
1.  SD-WAN policy routes
2.  VPN routes
3.  Static routes

yours:
Routing Precedence:
1. Static routes
2. SD-WAN policy routes
3. VPN routes

We have no SD-WANs

We have VPN routes (but only to the 220 networks, no WAN routes)

-> system ipsec_route show

You say your static routes are empty. And so are ours.

It's worth a try to change that precedence.

Maybe LuCar Toni ca have a review on your NAT/SNAT config?

and please be sure that you don't have bad SNAT here:

console> show advanced-firewall

...

NAT policy for system originated traffic
---------------------
Destination Network Destination Netmask Interface SNAT IP

interesting stuff here

I am not sure right now, which will be applied first. If you change the Traffic via a NAT Rule, if the traffic will hit the IPsec0 Interface or if the routing decision will be applied after the NAT. I assume its the NAT part. Having said, this is rather a problematic construct, as the firewall is doing what you tell the firewall to do. It NATs the traffic to ANY into the tunnel. 

I would assume, you should think about the setup in the first place, if remote subnet should be ANY and cannot be more specific? 

one thing to point out: We do not apply NAT to any traffic in the tunnel.

we have a local subnet 10.64.0.0/21 and the remote subnet is any or 0.0.0.0/0 - no NAT applied

we need the full tunnel because web-security is currently applied in the HQ.

the only NAT rule i have is from LAN to WAN zone for direct internet access if the tunnel fails.

console> show advanced-firewall
NAT policy for system originated traffic
---------------------
Destination Network Destination Netmask Interface SNAT IP

... list is empty

Somehow the XG uses 0.1 as a outgoing IP. This is configured by somebody. Please review the NAT page and check all rules. 

please see the NAT rules below. i can't see anyhting wrong here.

you may ignore rule 1 - this is a workaround to have the DNS server function working for 1 client - does not make difference if i delete/disable it.

Do you use NAT on the IPSec setup? The Checkbox? 

PS: What is the issue, if you wants to use the IPsec Tunnel for everything? I mean, if Security is done on the peer site, you can also give the DNS etc. to them? 

not checked

sure i could allow the XG traffic on the peer side - that would be the easy way.

but i don't get why an update of the firewall changes the routing in a way that all my security policies on the peer side need to be edited.

and i would expect the XG firewall system to be configurable in the way i want it... and since is was working before in that way, i would rather find the cause of the issue and solve it