Sophos XG BGP dynamic routing over 2 RED tunnels (for failover)

Question

Hi, 
 I'm trying to set up a reliable connection between two offices of the same company using RED tunnels instead of VPNs. There are limitations which are not relevant now, but the fact is that we can't use IPSEC VPNs and RED tunnels work fine. 
 
 This is the scenario:

Office 1 
 Office 2

XG1 SFOS 18.5.1 MR-1-Build326 
 
 XG2 SFOS 18.5.1 MR-1-Build326

WAN1.1 
 
 WAN2.1

WAN1.2 
 
 WAN 2.2

XG1 red Tunnels (main and backup)

XG2 red Tunnels (main and backup)

BGP CONFIGURATION XG1

BGP CONFIGURATION XG2

BGP Summary at XG1 
 BGP router identifier 192.168.0.76, local AS number 500
RIB entries 7, using 448 bytes of memory
Peers 2, using 4968 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.254.2 4 501 84 87 0 0 0 01:17:08 2
10.0.254.5 4 501 1023 1031 0 0 0 00:15:36 2

Total number of neighbors 2

BGP Summary at XG2 
 BGP router identifier 192.168.64.101, local AS number 501
RIB entries 7, using 448 bytes of memory
Peers 2, using 4968 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.254.1 4 500 82 83 0 0 0 01:15:36 2
10.0.254.6 4 500 1018 1026 0 0 0 00:14:03 2

Total number of neighbors 2

BGP routes at XG1 BGP table version is 0, local router ID is 192.168.0.76
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
 r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 10.81.234.0/24 0.0.0.0 0 32768 i
*> 172.27.240.0/24 10.0.254.5 0 20 501 i
* 10.0.254.2 0 0 501 i
*> 192.168.0.0/18 0.0.0.0 0 32768 i
*> 192.168.64.0/18 10.0.254.5 0 20 501 i
* 10.0.254.2 0 0 501 i

Total number of prefixes 4

BGP routes at XG2 
 BGP table version is 0, local router ID is 192.168.64.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
 r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 10.81.234.0/24 10.0.254.6 0 20 500 i
* 10.0.254.1 0 0 500 i
*> 172.27.240.0/24 0.0.0.0 0 32768 i
*> 192.168.0.0/18 10.0.254.6 0 20 500 i
* 10.0.254.1 0 0 500 i
*> 192.168.64.0/18 0.0.0.0 0 32768 i

Total number of prefixes 4

BGP Neighbors at XG1 
 
 BGP neighbor is 10.0.254.2, remote AS 501, local AS 500, external link
 BGP version 4, remote router ID 192.168.64.101
 BGP state = Established, up for 01:47:58
 Last read 00:00:58, hold time is 180, keepalive interval is 60 seconds
 Neighbor capabilities:
 4 Byte AS: advertised and received
 Route refresh: advertised and received(old & new)
 Address family IPv4 Unicast: advertised and received
 Message statistics:
 Inq depth is 0
 Outq depth is 0
 Sent Rcvd
 Opens: 3 0
 Notifications: 0 1
 Updates: 5 5
 Keepalives: 109 108
 Route Refresh: 0 0
 Capability: 0 0
 Total: 117 114
 Minimum time between advertisement runs is 30 seconds
 Default weight 10

For address family: IPv4 Unicast
 Community attribute sent to this neighbor(both)
 2 accepted prefixes

Connections established 1; dropped 0
 Last reset never
Local host: 10.0.254.1, Local port: 179
Foreign host: 10.0.254.2, Foreign port: 52066
Nexthop: 10.0.254.1
Read thread: on Write thread: off

BGP neighbor is 10.0.254.5, remote AS 501, local AS 500, external link
 BGP version 4, remote router ID 192.168.64.101
 BGP state = Established, up for 00:46:26
 Last read 00:00:26, hold time is 180, keepalive interval is 60 seconds
 Neighbor capabilities:
 4 Byte AS: advertised and received
 Route refresh: advertised and received(old & new)
 Address family IPv4 Unicast: advertised and received
 Message statistics:
 Inq depth is 0
 Outq depth is 0
 Sent Rcvd
 Opens: 4 1
 Notifications: 1 1
 Updates: 5 5
 Keepalives: 1052 1047
 Route Refresh: 0 0
 Capability: 0 0
 Total: 1062 1054
 Minimum time between advertisement runs is 30 seconds
 Default weight 20

For address family: IPv4 Unicast
 Community attribute sent to this neighbor(both)
 2 accepted prefixes

Connections established 3; dropped 2
 Last reset 00:46:39, due to BGP Notification send
Local host: 10.0.254.6, Local port: 49014
Foreign host: 10.0.254.5, Foreign port: 179
Nexthop: 10.0.254.6
Read thread: on Write thread: off

BGP Neighbors at XG2 
 
 BGP neighbor is 10.0.254.1, remote AS 500, local AS 501, external link
 BGP version 4, remote router ID 192.168.0.76
 BGP state = Established, up for 01:47:56
 Last read 00:00:56, hold time is 180, keepalive interval is 60 seconds
 Neighbor capabilities:
 4 Byte AS: advertised and received
 Route refresh: advertised and received(old & new)
 Address family IPv4 Unicast: advertised and received
 Message statistics:
 Inq depth is 0
 Outq depth is 0
 Sent Rcvd
 Opens: 1 1
 Notifications: 0 0
 Updates: 5 5
 Keepalives: 109 108
 Route Refresh: 0 0
 Capability: 0 0
 Total: 115 114
 Minimum time between advertisement runs is 30 seconds
 Default weight 10

For address family: IPv4 Unicast
 Community attribute sent to this neighbor(both)
 2 accepted prefixes

Connections established 1; dropped 0
 Last reset never
Local host: 10.0.254.2, Local port: 52066
Foreign host: 10.0.254.1, Foreign port: 179
Nexthop: 10.0.254.2
Read thread: on Write thread: off

BGP neighbor is 10.0.254.6, remote AS 500, local AS 501, external link
 BGP version 4, remote router ID 192.168.0.76
 BGP state = Established, up for 00:46:23
 Last read 00:00:23, hold time is 180, keepalive interval is 60 seconds
 Neighbor capabilities:
 4 Byte AS: advertised and received
 Route refresh: advertised and received(old & new)
 Address family IPv4 Unicast: advertised and received
 Message statistics:
 Inq depth is 0
 Outq depth is 0
 Sent Rcvd
 Opens: 3 1
 Notifications: 1 0
 Updates: 4 4
 Keepalives: 1050 1045
 Route Refresh: 0 0
 Capability: 0 0
 Total: 1058 1050
 Minimum time between advertisement runs is 30 seconds
 Default weight 20

For address family: IPv4 Unicast
 Community attribute sent to this neighbor(both)
 2 accepted prefixes

Connections established 2; dropped 1
 Last reset 00:46:37, due to BGP Notification send
Local host: 10.0.254.5, Local port: 179
Foreign host: 10.0.254.6, Foreign port: 49014
Nexthop: 10.0.254.5
Read thread: on Write thread: off

route -n at XG1 
 XG310_WP02_SFOS 18.5.1 MR-1-Build326# netstat -nr|grep red 
 10.0.254.0 0.0.0.0 255.255.255.252 U 0 0 0 reds20 
 10.0.254.4 0.0.0.0 255.255.255.252 U 0 0 0 reds21 
 172.27.240.0 10.0.254.5 255.255.255.0 UG 0 0 0 reds21 
 192.168.64.0 10.0.254.5 255.255.192.0 UG 0 0 0 reds21

route -n at XG2 
 XG135_XN03_SFOS 18.5.1 MR-1-Build326# netstat -nr |grep red 
 10.0.254.0 0.0.0.0 255.255.255.252 U 0 0 0 reds20 
 10.0.254.4 0.0.0.0 255.255.255.252 U 0 0 0 reds21 
 10.81.234.0 10.0.254.6 255.255.255.0 UG 0 0 0 reds21 
 
 192.168.0.0 10.0.254.6 255.255.192.0 UG 0 0 0 reds21

ip route -n at XG1 
 
 XG310_WP02_SFOS 18.5.1 MR-1-Build326# ip route |grep reds2 
 10.0.254.0/30 dev reds20 proto kernel scope link src 10.0.254.1 
 10.0.254.4/30 dev reds21 proto kernel scope link src 10.0.254.6 
 172.27.240.0/24 via 10.0.254.5 dev reds21 proto zebra 
 192.168.64.0/18 via 10.0.254.5 dev reds21 proto zebra

ip route at XG2 
 
 XG135_XN03_SFOS 18.5.1 MR-1-Build326# ip route |grep reds 
 10.0.254.0/30 dev reds20 proto kernel scope link src 10.0.254.2 
 10.0.254.4/30 dev reds21 proto kernel scope link src 10.0.254.5 
 10.81.234.0/24 via 10.0.254.6 dev reds21 proto zebra 
 10.254.251.0/30 dev reds500 proto kernel scope link src 10.254.251.1 
 10.254.252.0/30 dev reds1 proto kernel scope link src 10.254.252.1 
 192.168.0.0/18 via 10.0.254.6 dev reds21 proto zebra

What happens: Everything seems to work fine pinging, trace-routing from one network to another.. However if I disable one of the RED tunnels, the routing does not commute to the active RED tunnel. 
 
 So, should it commute? 
 If not, what could I do to automatically conmmute? 
 
 At the end of the day, what we need is a failover connectivity between these two sites.. and no IPSEC VPN can be used..

Thanks in advance!

Santiago Crespo Guillen · Accepted Answer

Lucar.. Thanks a lot for your help! - I'll take a look at the link and will come back with more feedback! 
 
 Update: 
 
 I've set-up the following parameters via CLI:

router bgp 500 
 bgp router-id 192.168.0.76 
 network 10.81.234.0/24 
 network 192.168.0.0/18 
 timers bgp 3 10 
 neighbor 10.0.254.2 remote-as 501 
 neighbor 10.0.254.2 weight 21 
 neighbor 10.0.254.5 remote-as 501 
 neighbor 10.0.254.5 weight 20 
 maximum-paths 4 
 !

router bgp 501 
 bgp router-id 192.168.64.101 
 network 172.27.240.0/24 
 network 192.168.64.0/18 
 timers bgp 3 10 
 neighbor 10.0.254.1 remote-as 500 
 neighbor 10.0.254.1 weight 21 
 neighbor 10.0.254.6 remote-as 500 
 neighbor 10.0.254.6 weight 20 
 maximum-paths 4 
 !

done a write to save changes and things look the same so far.. I don't know if there's a way to restart the BGP service and check if it makes any difference.. 
 
 After a while... I shutted down one of the RED tunnels and then automatically the connection started working using the other one. So it looks the solution is working. :-)

Sophos XG BGP dynamic routing over 2 RED tunnels (for failover)

Top Replies