This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

HA confd sync daemon not running - restarted [8.002]

I keep getting messages that HA confd sync daemon is not running and that it's being restarted... I didn't know keeping a HA setup alive would be such a pain [:(]

When I look in the selfmonitoring logs I see the following repeating over and over again:


2010:10:02-02:06:51 firewall-2 selfmonng[4604]: W triggerAction: 'cmd'

2010:10:02-02:06:51 firewall-2 selfmonng[4604]: W actionCmd(-): '/var/mdw/scripts/confd-sync start'

2010:10:02-02:06:56 firewall-2 selfmonng[4604]: I check Failed increment confd-sync_running counter 1 - 3

2010:10:02-02:07:01 firewall-2 selfmonng[4604]: I check Failed increment confd-sync_running counter 2 - 3

2010:10:02-02:07:06 firewall-2 selfmonng[4604]: W check Failed increment confd-sync_running counter 3 - 3

2010:10:02-02:07:06 firewall-2 selfmonng[4604]: W NOTIFYEVENT Name=confd-sync_running Level=INFO Id=170 suppressed

This thread was automatically locked due to age.

Parents

0 Fury over 15 years ago

Around 2 in the morning the error messages stopped, it looks like things are working now:


2010:10:03-02:05:48 firewall-1 slon[8015]: [4-1] CONFIG slon: child terminated status: 0; pid: 22627, current worker pid: 22627
2010:10:03-02:05:48 firewall-1 slon[8015]: [5-1] CONFIG slon: restart of worker
2010:10:03-02:05:48 firewall-1 slon[8015]: [1-1] CONFIG main: slon version 2.0.3 starting up
2010:10:03-02:05:48 firewall-1 slon[8015]: [2-1] CONFIG slon: watchdog ready - pid = 8015
2010:10:03-02:05:48 firewall-1 slon[8015]: [3-1] CONFIG slon: worker process created - pid = 27000
2010:10:03-02:05:48 firewall-1 slon[27000]: [3-1] CONFIG main: Integer option vac_frequency = 3
2010:10:03-02:05:48 firewall-1 slon[27000]: [4-1] CONFIG main: Integer option log_level = -1
2010:10:03-02:05:48 firewall-1 slon[27000]: [5-1] CONFIG main: Integer option sync_interval = 2000
2010:10:03-02:05:48 firewall-1 slon[27000]: [6-1] CONFIG main: Integer option sync_interval_timeout = 10000
2010:10:03-02:05:48 firewall-1 slon[27000]: [7-1] CONFIG main: Integer option sync_group_maxsize = 20
2010:10:03-02:05:48 firewall-1 slon[27000]: [8-1] CONFIG main: Integer option desired_sync_time = 60000
2010:10:03-02:05:48 firewall-1 slon[27000]: [9-1] CONFIG main: Integer option syslog = 2
2010:10:03-02:05:48 firewall-1 slon[27000]: [10-1] CONFIG main: Integer option quit_sync_provider = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [11-1] CONFIG main: Integer option quit_sync_finalsync = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [12-1] CONFIG main: Integer option sync_max_rowsize = 8192
2010:10:03-02:05:48 firewall-1 slon[27000]: [13-1] CONFIG main: Integer option sync_max_largemem = 5242880
2010:10:03-02:05:48 firewall-1 slon[27000]: [14-1] CONFIG main: Integer option remote_listen_timeout = 300
2010:10:03-02:05:48 firewall-1 slon[27000]: [15-1] CONFIG main: Boolean option log_pid = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [16-1] CONFIG main: Boolean option log_timestamp = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [17-1] CONFIG main: Boolean option cleanup_deletelogs = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [18-1] CONFIG main: Real option real_placeholder = 0.000000
2010:10:03-02:05:48 firewall-1 slon[27000]: [19-1] CONFIG main: String option cluster_name = asg_cluster
2010:10:03-02:05:48 firewall-1 slon[27000]: [20-1] CONFIG main: String option conn_info = dbname=pop3 user=ha_sync
2010:10:03-02:05:48 firewall-1 slon[27000]: [21-1] CONFIG main: String option pid_file = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [22-1] CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2010:10:03-02:05:48 firewall-1 slon[27000]: [23-1] CONFIG main: String option archive_dir = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [24-1] CONFIG main: String option sql_on_connection = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [25-1] CONFIG main: String option lag_interval = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [26-1] CONFIG main: String option command_on_logarchive = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [27-1] CONFIG main: String option syslog_facility = LOG_DAEMON
2010:10:03-02:05:48 firewall-1 slon[27000]: [28-1] CONFIG main: String option syslog_ident = slon
2010:10:03-02:05:48 firewall-1 slon[27000]: [29-1] CONFIG main: String option cleanup_interval = 10 minutes
2010:10:03-02:05:49 firewall-1 slon[27000]: [30-1] CONFIG main: local node id = 1
2010:10:03-02:05:49 firewall-1 slon[27000]: [31-1] CONFIG main: launching sched_start_mainloop
2010:10:03-02:05:49 firewall-1 slon[27000]: [32-1] CONFIG main: loading current cluster configuration
2010:10:03-02:05:49 firewall-1 slon[27000]: [33-1] CONFIG storeNode: no_id=2 no_comment='Node 2'
2010:10:03-02:05:49 firewall-1 slon[27000]: [34-1] CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo="dbname=pop3 host=198.19.250.2 user=ha_sync password=slony"
2010:10:03-02:05:49 firewall-1 slon[27000]: [34-2]  pa_connretry=10
2010:10:03-02:05:49 firewall-1 slon[27000]: [35-1] CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [36-1] CONFIG storeSet: set_id=1 set_origin=1 set_comment='pop3 tables'
2010:10:03-02:05:49 firewall-1 slon[27000]: [37-1] CONFIG main: last local event sequence = 5000000021
2010:10:03-02:05:49 firewall-1 slon[27000]: [38-1] CONFIG main: configuration complete - starting threads
2010:10:03-02:05:49 firewall-1 slon[27000]: [39-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [40-1] CONFIG enableNode: no_id=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [41-1] CONFIG cleanupThread: thread starts
2010:10:03-02:05:49 firewall-1 slon[27000]: [42-1] CONFIG cleanupThread: bias = 35383
2010:10:03-02:05:49 firewall-1 slon[27000]: [43-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [44-1] CONFIG version for "dbname=pop3 host=198.19.250.2 user=ha_sync password=slony" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [45-1] CONFIG remoteWorkerThread_2: update provider configuration
2010:10:03-02:05:49 firewall-1 slon[27000]: [46-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [47-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [48-1] CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [49-1] CONFIG remoteWorkerThread_2: update provider configuration
2010:10:03-02:10:27 firewall-1 slon_control[13436]: Initial synchronization for node 2 finished!
2010:10:03-02:10:27 firewall-1 ha_daemon[12829]: id="38A0" severity="info" sys="System" sub="ha" name="Deactivating sync process for database on node 2"
2010:10:03-02:10:41 firewall-2 ha_daemon[4399]: id="38A0" severity="info" sys="System" sub="ha" name="Initial synchronization finished!"
2010:10:03-02:10:42 firewall-1 ha_daemon[12829]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: SYNCING -> ACTIVE"

Why it takes 12+ hours and 5Mb of HA logs to finally restart some stuff and reinstall a node I don't get... but it's finally working [:)]

Reply

0 Fury over 15 years ago

Around 2 in the morning the error messages stopped, it looks like things are working now:


2010:10:03-02:05:48 firewall-1 slon[8015]: [4-1] CONFIG slon: child terminated status: 0; pid: 22627, current worker pid: 22627
2010:10:03-02:05:48 firewall-1 slon[8015]: [5-1] CONFIG slon: restart of worker
2010:10:03-02:05:48 firewall-1 slon[8015]: [1-1] CONFIG main: slon version 2.0.3 starting up
2010:10:03-02:05:48 firewall-1 slon[8015]: [2-1] CONFIG slon: watchdog ready - pid = 8015
2010:10:03-02:05:48 firewall-1 slon[8015]: [3-1] CONFIG slon: worker process created - pid = 27000
2010:10:03-02:05:48 firewall-1 slon[27000]: [3-1] CONFIG main: Integer option vac_frequency = 3
2010:10:03-02:05:48 firewall-1 slon[27000]: [4-1] CONFIG main: Integer option log_level = -1
2010:10:03-02:05:48 firewall-1 slon[27000]: [5-1] CONFIG main: Integer option sync_interval = 2000
2010:10:03-02:05:48 firewall-1 slon[27000]: [6-1] CONFIG main: Integer option sync_interval_timeout = 10000
2010:10:03-02:05:48 firewall-1 slon[27000]: [7-1] CONFIG main: Integer option sync_group_maxsize = 20
2010:10:03-02:05:48 firewall-1 slon[27000]: [8-1] CONFIG main: Integer option desired_sync_time = 60000
2010:10:03-02:05:48 firewall-1 slon[27000]: [9-1] CONFIG main: Integer option syslog = 2
2010:10:03-02:05:48 firewall-1 slon[27000]: [10-1] CONFIG main: Integer option quit_sync_provider = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [11-1] CONFIG main: Integer option quit_sync_finalsync = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [12-1] CONFIG main: Integer option sync_max_rowsize = 8192
2010:10:03-02:05:48 firewall-1 slon[27000]: [13-1] CONFIG main: Integer option sync_max_largemem = 5242880
2010:10:03-02:05:48 firewall-1 slon[27000]: [14-1] CONFIG main: Integer option remote_listen_timeout = 300
2010:10:03-02:05:48 firewall-1 slon[27000]: [15-1] CONFIG main: Boolean option log_pid = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [16-1] CONFIG main: Boolean option log_timestamp = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [17-1] CONFIG main: Boolean option cleanup_deletelogs = 0
2010:10:03-02:05:48 firewall-1 slon[27000]: [18-1] CONFIG main: Real option real_placeholder = 0.000000
2010:10:03-02:05:48 firewall-1 slon[27000]: [19-1] CONFIG main: String option cluster_name = asg_cluster
2010:10:03-02:05:48 firewall-1 slon[27000]: [20-1] CONFIG main: String option conn_info = dbname=pop3 user=ha_sync
2010:10:03-02:05:48 firewall-1 slon[27000]: [21-1] CONFIG main: String option pid_file = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [22-1] CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2010:10:03-02:05:48 firewall-1 slon[27000]: [23-1] CONFIG main: String option archive_dir = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [24-1] CONFIG main: String option sql_on_connection = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [25-1] CONFIG main: String option lag_interval = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [26-1] CONFIG main: String option command_on_logarchive = [NULL]
2010:10:03-02:05:48 firewall-1 slon[27000]: [27-1] CONFIG main: String option syslog_facility = LOG_DAEMON
2010:10:03-02:05:48 firewall-1 slon[27000]: [28-1] CONFIG main: String option syslog_ident = slon
2010:10:03-02:05:48 firewall-1 slon[27000]: [29-1] CONFIG main: String option cleanup_interval = 10 minutes
2010:10:03-02:05:49 firewall-1 slon[27000]: [30-1] CONFIG main: local node id = 1
2010:10:03-02:05:49 firewall-1 slon[27000]: [31-1] CONFIG main: launching sched_start_mainloop
2010:10:03-02:05:49 firewall-1 slon[27000]: [32-1] CONFIG main: loading current cluster configuration
2010:10:03-02:05:49 firewall-1 slon[27000]: [33-1] CONFIG storeNode: no_id=2 no_comment='Node 2'
2010:10:03-02:05:49 firewall-1 slon[27000]: [34-1] CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo="dbname=pop3 host=198.19.250.2 user=ha_sync password=slony"
2010:10:03-02:05:49 firewall-1 slon[27000]: [34-2]  pa_connretry=10
2010:10:03-02:05:49 firewall-1 slon[27000]: [35-1] CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [36-1] CONFIG storeSet: set_id=1 set_origin=1 set_comment='pop3 tables'
2010:10:03-02:05:49 firewall-1 slon[27000]: [37-1] CONFIG main: last local event sequence = 5000000021
2010:10:03-02:05:49 firewall-1 slon[27000]: [38-1] CONFIG main: configuration complete - starting threads
2010:10:03-02:05:49 firewall-1 slon[27000]: [39-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [40-1] CONFIG enableNode: no_id=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [41-1] CONFIG cleanupThread: thread starts
2010:10:03-02:05:49 firewall-1 slon[27000]: [42-1] CONFIG cleanupThread: bias = 35383
2010:10:03-02:05:49 firewall-1 slon[27000]: [43-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [44-1] CONFIG version for "dbname=pop3 host=198.19.250.2 user=ha_sync password=slony" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [45-1] CONFIG remoteWorkerThread_2: update provider configuration
2010:10:03-02:05:49 firewall-1 slon[27000]: [46-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [47-1] CONFIG version for "dbname=pop3 user=ha_sync" is 80404
2010:10:03-02:05:49 firewall-1 slon[27000]: [48-1] CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2010:10:03-02:05:49 firewall-1 slon[27000]: [49-1] CONFIG remoteWorkerThread_2: update provider configuration
2010:10:03-02:10:27 firewall-1 slon_control[13436]: Initial synchronization for node 2 finished!
2010:10:03-02:10:27 firewall-1 ha_daemon[12829]: id="38A0" severity="info" sys="System" sub="ha" name="Deactivating sync process for database on node 2"
2010:10:03-02:10:41 firewall-2 ha_daemon[4399]: id="38A0" severity="info" sys="System" sub="ha" name="Initial synchronization finished!"
2010:10:03-02:10:42 firewall-1 ha_daemon[12829]: id="38A0" severity="info" sys="System" sub="ha" name="Node 2 changed state: SYNCING -> ACTIVE"

Why it takes 12+ hours and 5Mb of HA logs to finally restart some stuff and reinstall a node I don't get... but it's finally working [:)]

Children

No Data