18.04 route_precedence not working for VPN Routes

There are three different independent routing tables. 

The RFC standard to use the subnet mask to find the matching route (Like use /24 over /8) does not work across those routing tables. 

Is your source device one of both 1:1 NAT subnet masks or not? I can see there are two networks as 1:1 Networks. Those are the only routes, applied by XG. See Ipsec status. XG will only apply Source + destination of local and remote subnet and ignore everything else. 

You can, as you already did, set a own route via CLI, which will be considered additionally to your ipsec Routes. 

so if I get that right, there are three different routing tables

- static

- pbr (sd-wan)

- vpn

each routing table works within itself "best match"   but they don't work "best match" across each other, thats why you can configure route_precedence to change the order like so:

1. VPN

2. static 

3. pbr

in this case, if there is a VPN route eg. /16 it will win against a more specific static route eg. /24, because it has the higher priority, right?

so lets go back to the customer setup:

Yes the original source networks are source natted

for simplicity let's go with on combination only:

Tunnel original  Subnet: 10.1.1.0/24   Tunnel local subnet (SNAT): 10.2.2.0/24    Tunnel Remote: 10.150.70.40

from ipsec status i can see that the route is installed as:

10.2.2.0 <-> 10.150.70.40

before I configured IPSEC Static routes, i did not see this route in the static routing tables, when doing a "route" from the adv. CLI.

So I assume, what the command:

 system ipsec_route add net x.x.x.x/x tunnelname xxxx

essentially does, it puts a route into the static routing table to point to the IPSEC routing table. 

That would totally make sense, if the static routing table has the higher priority, than the VPN routing table.

but why is it necessary if the route_precedency is set to VPN, static, pbr?

I mean the tunnel works as expected including the SNAT as long as there is no static or PBR routing, containing the same subnets.

regardless of the orgiginal source or the natted source.

why is a static route able to overule the ipsec route, if "VPN" has the highest priortiy.

looking a the flow chart below, i don't get how  SNAT'ted VPN traffic matches any ACL at all, since SNAT is done after routing decition.

There is a false assumption: The CLI Command to add a route will not place this route into the Static routing, instead into the VPN category. 

Got that. 

But I still don't understand why I need to configure the IPSEC Route statically via console.

It does not work without static IPSEC Routes.

Routing Precedence is set to:
1. VPN routes
2. Static routes
3. SD-WAN policy routes

when doing a:  

ip route show table all 

I can see the following routes are present:

VPN

10.150.0.0/22 dev ipsec0 table 220 scope link 

10.150.0.0/22 dev ipsec0 scope link 

static Route

10.0.0.0/8 via 10.116.35.1 dev LAG.97 proto zebra metric 255 

but without the static IPSEC route:

system ipsec_route add net 10.150.0.0/255.255.0.0 tunnelname KIVBF_de 

the firewall always preferes the static 10.0.0.0/8 over the VPN Route. 

system ipsec_route add net 10.150.0.0/255.255.0.0 tunnelname KIVBF_de  Is a VPN Route, not a static route. 

Yes I know,  that's what I'm saying.

When I referred to static VPN route, I meant I had to manually  ("statically") configure this route via Console

And the question is, why is it necessary to configure this route from the CLI. 

As i explained the route precedency is set to VPN / Static / Sd-WAN, but a soon a I configure a static route, which contains the subnets from the VPN Network, it overrules the VPN Routes and traffic is sent to the local router.

the firewall simply ignores the route precedency, if there is no IPSEC_Route configured from the CLI, even if I can clearly see the route being automatically configured by the VPN Tunnel. 

The pre defined IPsec routes of your setup seems not to match, therefore they will be ignored. 

They only will be applied, if the source and destination network 100% apply to the traffic, which seems not to be the case in your setup. Therefore they will be ignored, and the next routing table is static, which apply the static route matching. 

Alright, so in short

if you have a source NAT applied to a VPN Tunnel, you always need additional ipsec_routes, in case you have an overlapping static or sd-wan rule. 

ignoring the fact, that there will always be a overlapping default route 0.0.0.0/0, but i guess that one is handled differently.

thx, makes sense now.

Alright, so in short

if you have a source NAT applied to a VPN Tunnel, you always need additional ipsec_routes, in case you have an overlapping static or sd-wan rule. 

ignoring the fact, that there will always be a overlapping default route 0.0.0.0/0, but i guess that one is handled differently.

thx, makes sense now.