Routing Precedence not working as expected

Likely you are doing something with the traffic. Means you are doing some sort of NAT or something? Which means, the traffic is not Local/Remote Subnet anymore? 

No, NAT is not involved in any case. To show some more detail I will take another remote subnet that causes less alerts in our monitoring when it is not reachable.

First my settings in pcap, I am trying to reach the WebAdmin of a firewall on the remote site (Sophos SG but makes no difference if it is a XG/XGS).

This is how it looks when only the route to the ASA is active (192.168.0.0/16) and no more specific route is pointing to the WAN interface:

Traffic is routed to the Cisco ASA on VLAN17, no NAT applies, FW rule is 20.

Now this is what happens when I create the more specific route:

The connection directly is active and I get replies.

The pcap now looks like this:

The route I added above is this, where "TK-DCIP" is the name of my WAN interface where the IPSEC tunnels are connected to the interface address (no alias IP included in VPN tunnels).

And as I said above, my routing precedence is VPN. Static, SD-WAN.

Can you show us the IPsec Tunnel configuration and the SAs? 

Sure, but there is nothing special:

At the moment I get the failure with every S2S tunnel that I connect to the XGS firewall, simple ones as the one above, and more complex configurations with 10 or more subnets on the remote side.

If nobody has an idea my plan for next week will be creating routes for all "ASA tunnels" and not using the private classes for routing anymore. On our old Sophos SG I did not use the RFC1918 subnets. There I had routings for every customer's subnet, all together in one static rule.

But it was much more transparent since I could use groups for static routing, to be exact groups in groups. I had a "all customer vpn networks" group, within that for every customer an own group "customer xyz networks" and within that group I had the real network definitions for the subnets. But that groups grew over the last 10 years and now I have something between 100 and 200 network definitions.

My plan was to avoid creating all those rules again, because I just need them temporary, when all tunnels have moved to the XGS I don't need those rules anymore. Since I can't give those rules names I have to manually search each customer's networks when moving/deleting them. Because of that I thought routing the RFC1918 networks would be a clever idea for the temporary phase, hoping that the route precedence will kick in if a network is attached "directly" through ipsec.

Sure, but there is nothing special:

At the moment I get the failure with every S2S tunnel that I connect to the XGS firewall, simple ones as the one above, and more complex configurations with 10 or more subnets on the remote side.

If nobody has an idea my plan for next week will be creating routes for all "ASA tunnels" and not using the private classes for routing anymore. On our old Sophos SG I did not use the RFC1918 subnets. There I had routings for every customer's subnet, all together in one static rule.

But it was much more transparent since I could use groups for static routing, to be exact groups in groups. I had a "all customer vpn networks" group, within that for every customer an own group "customer xyz networks" and within that group I had the real network definitions for the subnets. But that groups grew over the last 10 years and now I have something between 100 and 200 network definitions.

My plan was to avoid creating all those rules again, because I just need them temporary, when all tunnels have moved to the XGS I don't need those rules anymore. Since I can't give those rules names I have to manually search each customer's networks when moving/deleting them. Because of that I thought routing the RFC1918 networks would be a clever idea for the temporary phase, hoping that the route precedence will kick in if a network is attached "directly" through ipsec.

If you say, there are ipsec errors, it sounds like, the SAs are not correctly buildup on the system and you are workaround this.

What if you check the status on ipsec status all on CLI and check the ip route command? 

No I don't get IPSEC errors, the connection - if established once - stays until I disconnect it.

But tonight a complete other problem occured... the primary appliance went into a faulty state but not enough to cause a failover to the second appliance.

I went to office to troubleshoot. At the devices the secondary showed the right HA status, the primary display freezed when I went to the HA menu. All LEDs seemed to be normal, but the primary appliance did not pass any traffic from or to any zone.

Because I had that strange problem with my disappearing default ssl/tls rule earlier I decided to re-image both appliances completely with 19.5 iso, installed the backup and recreated the HA cluster. Maybe something did go wrong with my initial setup/firmware upgrade last time. It was the first XG/XGS appliance where I got so many issues that I never got on another appliance.

I deleted the RFC1918 routes (and those I had to create to make the Sophos-terminated tunnels work). Then I began to create all route entries manually and am finished now. No nice solution at all but I guess it will be better on the long run.

Regards,

Kevin