2010:09:01-12:48:52 (none)-2 ha_factory_reset[3558]: id="38K0" severity="info" sys="System" sub="ha" name="Cant read ha sync password from local confd!"
2010:09:01-12:48:54 (none)-2 ha_zeroconf[3636]: id="3800" severity="info" sys="System" sub="ha" name="Set ASG appliance to 320"
2010:09:01-12:48:54 (none)-2 ha_zeroconf[3636]: id="3800" severity="info" sys="System" sub="ha" name="set zeroconf interface to eth3"
2010:09:01-12:48:54 (none)-2 ha_zeroconf[3637]: id="3800" severity="info" sys="System" sub="ha" name="Waiting for interface to come up..."
2010:09:01-12:48:56 (none)-2 ha_zeroconf[3637]: id="3800" severity="info" sys="System" sub="ha" name="Interface eth3 is up! Searching for active HA system!"
2010:09:01-12:48:56 (none)-2 ha_zeroconf[3637]: id="3800" severity="info" sys="System" sub="ha" name="Using temporary ip 198.19.250.196"
2010:09:01-12:48:59 host ha_zeroconf[3637]: id="3800" severity="info" sys="System" sub="ha" name="Found master 198.19.250.1 on eth3, getting HA configuration..."
2010:09:01-12:48:59 host ha_zeroconf[3637]: id="3800" severity="info" sys="System" sub="ha" name="Using temporary ip 198.19.250.157"
2010:09:01-12:49:17 host ha_daemon[4628]: id="38A0" severity="info" sys="System" sub="ha" name="Cluster mode activated!"
2010:09:01-12:49:17 host ha_daemon[4628]: id="38A0" severity="info" sys="System" sub="ha" name="Autojoin disabled!"
2010:09:01-12:49:17 host ha_daemon[4628]: id="38A0" severity="info" sys="System" sub="ha" name="Reading cluster configuration"
2010:09:01-12:49:17 host ha_daemon[4628]: id="38A0" severity="info" sys="System" sub="ha" name="Set ASG version to 7.507"
2010:09:01-12:49:17 host ha_daemon[4628]: id="38A0" severity="info" sys="System" sub="ha" name="Set ASG appliance to 320"
2010:09:01-12:49:17 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Starting ASG HA daemon v0.9.8 in universal time-sharing mode on interface eth3 with name Node2"
2010:09:01-12:49:17 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="--- Node is disabled ---"
2010:09:01-12:49:17 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Monitoring interface for link beat: eth0"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Node 1 joined cluster"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Node 1 joined with version 7.507"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38C0" severity="info" sys="System" sub="ha" name="Node 1 is alive!"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Node 1 changed state: DEAD -> ACTIVE"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Node 1 changed mode: WORKER -> MASTER"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="cluster mode: set master id to 1"
2010:09:01-12:49:18 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Found master node 1, validating secret"
2010:09:01-12:49:19 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Secret validated!"
2010:09:01-12:49:19 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Validating MTU size"
2010:09:01-12:49:20 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Initdead time over"
2010:09:01-12:49:20 host ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="--- Node is enabled ---"
2010:09:01-12:49:20 host ha_daemon[4630]: id="38B7" severity="info" sys="System" sub="ha" name="Slave is dead, taking over!"
2010:09:01-12:49:20 host ha_daemon[4630]: id="38B1" severity="info" sys="System" sub="ha" name="Switching to Slave mode"
2010:09:01-12:49:20 host ha_daemon[4630]: id="38B1" severity="info" sys="System" sub="ha" name="Switching to Slave mode"
2010:09:01-12:49:22 host slon_control[5172]: Dropping privileges to user postgres with user ID 999 and group ID 999
2010:09:01-12:49:22 host slon_control[5177]: Starting ASG slon control on Node 2
2010:09:01-12:49:22 host slon_control[5177]: Slonik error, process exited with value 255
2010:09:01-12:49:22 host slon_control[5177]: Slonik error, process exited with value 255
2010:09:01-12:49:22 host slon_control[5177]: Set mode to SLAVE
2010:09:01-12:49:23 host ctsyncd: pidfile: No such file or directory
2010:09:01-12:49:23 host ctsyncd: ctsyncd started
2010:09:01-12:49:23 host ctsyncd: pidfile: No such file or directory
2010:09:01-12:49:23 host ctsyncd: tuple-cache initialized (hashsize 512)
2010:09:01-12:49:23 host ctsyncd: seq-cache initialized (hashsize 64)
2010:09:01-12:49:23 host ctsyncd: joined mcast group 224.0.0.82:1999
2010:09:01-12:49:23 host ctsyncd: new session 'default' (ip=198.19.250.2 grp=224.0.0.82:1999)
2010:09:01-12:49:23 host ctsyncd: now running in INIT mode
2010:09:01-12:49:23 host ctsyncd: Now running in SLAVE mode, syncing...
2010:09:01-12:49:24 host slon_control[5177]: Slonik error, process exited with value 255
2010:09:01-12:49:24 host slon_control[5177]: Starting replication from Node 1 to 2
2010:09:01-12:49:24 host slon_control[5177]: Found following tables in reporting: webanon_clients webanon_users mailanon_domains mailanon_addrs accounting accounting_archive auth ipscount ips ips_archive imp2p imp2p_archive mailsec_archive pfilter pfilter_archive mailsec websec_aud_archive websec_au_archive websec_bud_archive websec_aud websec_au websec_bud
2010:09:01-12:49:24 host slon_control[5177]: Found following sequences in reporting: webanon_clients__rowno_seq webanon_users__rowno_seq mailanon_domains__rowno_seq mailanon_addrs__rowno_seq accounting__rowno_seq accounting_archive__rowno_seq auth__rowno_seq ipscount__rowno_seq ips__rowno_seq ips_archive__rowno_seq imp2p__rowno_seq imp2p_archive__rowno_seq mailsec__rowno_seq mailsec_archive__rowno_seq pfilter__rowno_seq pfilter_archive__rowno_seq websec_aud__rowno_seq websec_aud_archive__rowno_seq websec_au__rowno_seq websec_au_archive__rowno_seq websec_bud__rowno_seq websec_bud_archive__rowno_seq
2010:09:01-12:49:29 host-2 slon_control[5177]: Slonik error, process exited with value 255
2010:09:01-12:49:30 host-2 slon_control[5177]: Starting replication from Node 1 to 2
2010:09:01-12:49:30 host-2 slon_control[5177]: Found following tables in pop3: confd_accounts confd_whitelist quarantine serveraddress confd_users accounts messages servers
2010:09:01-12:49:30 host-2 slon_control[5177]: Found following sequences in pop3: accounts_account_id_seq messages_message_id_seq servers_server_id_seq
2010:09:01-12:49:32 host-2 ctsyncd: Got SIGUSR1, set status to SLAVE
2010:09:01-12:49:32 host-2 slon_control[5177]: Started slon process 5992 for reporting
2010:09:01-12:49:32 host-2 slon_control[5177]: Started slon process 5993 for pop3
2010:09:01-12:49:32 host-2 slon[5992]: [1-1] CONFIG main: slon version 1.2.20 starting up
2010:09:01-12:49:32 host-2 slon[5993]: [1-1] CONFIG main: slon version 1.2.20 starting up
2010:09:01-12:49:32 host-2 slon[5994]: [2-1] CONFIG main: local node id = 2
2010:09:01-12:49:32 host-2 slon[5994]: [3-1] CONFIG main: launching sched_start_mainloop
2010:09:01-12:49:32 host-2 slon[5994]: [4-1] CONFIG main: loading current cluster configuration
2010:09:01-12:49:32 host-2 slon[5994]: [5-1] CONFIG storeNode: no_id=1 no_comment='Node 1'
2010:09:01-12:49:32 host-2 slon[5994]: [6-1] CONFIG storePath: pa_server=1 pa_client=2 pa_conninfo="dbname=reporting host=198.19.250.1 user=ha_sync password=slony"
2010:09:01-12:49:32 host-2 slon[5994]: [6-2]  pa_connretry=10
2010:09:01-12:49:32 host-2 slon[5994]: [7-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:32 host-2 slon[5994]: [8-1] CONFIG main: configuration complete - starting threads
2010:09:01-12:49:32 host-2 slon[5996]: [2-1] CONFIG main: local node id = 2
2010:09:01-12:49:32 host-2 slon[5996]: [3-1] CONFIG main: launching sched_start_mainloop
2010:09:01-12:49:32 host-2 slon[5994]: [9-1] CONFIG enableNode: no_id=1
2010:09:01-12:49:32 host-2 slon[5996]: [4-1] CONFIG main: loading current cluster configuration
2010:09:01-12:49:32 host-2 slon[5996]: [5-1] CONFIG storeNode: no_id=1 no_comment='Node 1'
2010:09:01-12:49:32 host-2 slon[5996]: [6-1] CONFIG storePath: pa_server=1 pa_client=2 pa_conninfo="dbname=pop3 host=198.19.250.1 user=ha_sync password=slony"
2010:09:01-12:49:32 host-2 slon[5996]: [6-2]  pa_connretry=10
2010:09:01-12:49:32 host-2 slon[5996]: [7-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:32 host-2 slon[5996]: [8-1] CONFIG main: configuration complete - starting threads
2010:09:01-12:49:32 host-2 slon[5996]: [9-1] CONFIG enableNode: no_id=1
2010:09:01-12:49:32 host-2 slon[5996]: [10-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:32 host-2 slon[5996]: [11-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:32 host-2 slon[5996]: [12-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:32 host-2 slon[5996]: [13-1] CONFIG storeSet: set_id=1 set_origin=1 set_comment='pop3 tables'
2010:09:01-12:49:33 host-2 slon[5996]: [14-1] CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward='t'
2010:09:01-12:49:33 host-2 slon[5996]: [15-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:33 host-2 slon[5996]: [16-1] CONFIG enableSubscription: sub_set=1
2010:09:01-12:49:33 host-2 slon[5996]: [17-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:33 host-2 slon[5994]: [10-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:33 host-2 slon[5994]: [11-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:33 host-2 slon[5994]: [12-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:33 host-2 slon[5994]: [13-1] CONFIG storeSet: set_id=1 set_origin=1 set_comment='reporting tables'
2010:09:01-12:49:33 host-2 slon[5994]: [14-1] CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward='t'
2010:09:01-12:49:33 host-2 slon[5994]: [15-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:49:45 host-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Autojoin enabled!"
2010:09:01-12:49:45 host-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Reading cluster configuration"
2010:09:01-12:50:01 host-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Monitoring interfaces for link beat: eth0 eth1 eth2 "
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Monitoring interfaces for link beat: eth0 eth1 eth2 "
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth0!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth0!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth2!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth2!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Found link beat on eth0 again!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth1!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth1!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Lost link beat on eth1!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Found link beat on eth2 again!"
2010:09:01-12:50:09 host-2 ha_daemon[4630]: id="38A3" severity="debug" sys="System" sub="ha" name="Netlink: Found link beat on eth1 again!"
2010:09:01-12:52:14 astaro-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Monitoring interfaces for link beat: eth0 eth1 eth2 "
2010:09:01-12:55:47 astaro-2 slon[5994]: [16-1] CONFIG enableSubscription: sub_set=1
2010:09:01-12:55:48 astaro-2 slon[5994]: [17-1] CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2010:09:01-12:59:26 astaro-1 slon_control[3559]: Initial synchronization for node 2 finished!
2010:09:01-12:59:26 astaro-1 ha_daemon[3259]: id="38A0" severity="info" sys="System" sub="ha" name="Deactivating sync process for database on node 2"
2010:09:01-12:59:42 astaro-2 ha_daemon[4630]: id="38A0" severity="info" sys="System" sub="ha" name="Initial synchronization finished!"
2010:09:01-12:59:43 astaro-1 ha_daemon[3259]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: SYNCING -> ACTIVE"