Our failover system (vpn-1 and vpn-2) has been running stable for several days now (on 8.103) so I risked taking a look at the logs. And what I discovered is a bit alarming:
2011:09:07-14:51:08 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-14:51:08 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-17:22:58 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-17:22:58 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-22:03:37 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-22:03:37 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-22:03:37 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: ACTIVE -> UNLINKED"
2011:09:07-22:03:38 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: UNLINKED -> ACTIVE"
2011:09:07-22:05:28 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-22:05:28 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-22:05:28 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: ACTIVE -> UNLINKED"
2011:09:07-22:05:29 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: UNLINKED -> ACTIVE"
2011:09:07-22:06:02 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-22:06:02 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-22:06:02 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: ACTIVE -> UNLINKED"
2011:09:07-22:06:03 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: UNLINKED -> ACTIVE"
2011:09:07-22:10:33 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="Lost link on interface lag0"
2011:09:07-22:10:33 vpn-2 ha_daemon[5332]: id="38A1" severity="warn" sys="System" sub="ha" name="All interfaces with link again!"
2011:09:07-22:10:33 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: ACTIVE -> UNLINKED"
2011:09:07-22:10:34 vpn-1 ha_daemon[5348]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: UNLINKED -> ACTIVE"
It seems like the LAG is giving the backup unit quite some trouble.
Some background info:
Two ports of each unit are connected to switch 1, two ports to switch 2. Each unit therefore has full redundancy in case of a hardware failure of one of the switches (using bond mode 1 - active/backup). Both lags have an arp_ip_target set which corresponds to the particular default gateway on each side.
I tried to figure out what the problem might be and came up with the following possibilities:
a) the arp_ip_target is not reachable sometimes (this is impossible, because the default gateways are layer 3 switches with more than enough capacity and almost no load)
b) the source ip of the backup unit is the same as on the master (this should also be no reason, because they are communicating at layer 2 using ARP)
c) the arp_interval is chosen too aggressive (that may be but why does this happen randomly over time? If that were the problem it should be constantly and reproducable)
d) It's a connectivity problem (I ruled that out, because after a forced failover by rebooting the master the new slave seems to have the exact same problems and they disappear on the new master)
e) ...?
Anyone got an idea or had the same problem using arp_ip_target?
Cheers and good night
Manuel
This thread was automatically locked due to age.
