XG v18 MR-3 SDWAN Policy Routing

Do you have a SD-WAN Rule on SiteB? 

The important part to understand is: SD-WAN PBR will break with the network expectation of the last decades. You can "force" XG to route those packets to other interfaces.

If you enable "reply-packets", it will even route the packets, based on a existing connection, to another Interface and not to the interface, which the packets originated. 

For years, this setup was feared as asymmetric routing and bad practice. But starting with SD-WAN, you actually do not care, how those packets get to the destination, as long they arrive there. 

Yes, I have SDWAN on both sides. Originating traffic from each LAN is routing as expected. However, return traffic (PING response) is not being direct properly. Reply traffic is going out the WAN instead of the VTI. 

Yes, I have SDWAN on both sides. Originating traffic from each LAN is routing as expected. However, return traffic (PING response) is not being direct properly. Reply traffic is going out the WAN instead of the VTI. 

Disable Reply traffic. That should do the trick on both ends. 

I started with that, it was off by default as these were v17 upgrades. Reply traffic went out the WAN. Enabled reply, reply traffic still went out the WAN.  These configs are very straight forward but with SDWAN PBR the traffic won't go back. As soon as I make a static route, everything is fine but I can't do tunnel failover with static routes.

The question is:  Originating traffic follows PBR, but why doesn't the reply traffic follow the PBR rules, with or without reply traffic enabled in the CLI?

There is an HR pair at both ends. 

Reply Traffic switch should be disabled. This should give you the desired behavior.

What you can do: Set it to disable and flush the conntrack (conntrack -F). This will reset all connections. Based on this, it should be working. 

The switch can actually be applied to new traffic. 

PS: Do you have any migrated SD-WAN Rules still there? 

We don't have any migrated SDWAN rules. We were using were using straight IPSEC site-to-site tunnels. We are moving them to VTI for better failover control.

I will try this tonight, but I don't expect it it will make a difference. That is how it was before.

I did notice last night that when I switched HA to the passive device (failed over), traffic did work as expected for a time. I have to wonder if VTI/PBR works as expected in HA setups.

I have a question. Do you MASQ/NAT the traffic on one XG for the tunnel? 

Not intentionally. I haven't gone thru the NAT rules after the v18 upgrade though.

Site A - 192.168.x.0/24,  Site B - 192.168.y.0/24

Static route works... I Would expect a NAT/MASQ issue would show up there as wll.

Do you have a NAT Rule from Site A to Site B using SNAT? Anything to masq the traffic? 

No

You should get in touch with Sophos Support to get your conntracks etc. analysed. What you could do, try to disable firewall acceleration to check, if this will actually have an impact. 

Where is that?