Since moving the platform to an active/active cluster config I'm noticing an ongoing daily increase in CPU utilisation and an interesting memory utilisation pattern.
Interesting update - the master server within the cluster just fell over and rebooted.
Due to the bug in reporting (master and nodes all show the same performance stats) I now can see the utilisation stats from what was the worker node. Looks near 'normal' compared to the master.
Erm, the new master just failed and rebooted causing another failover. Will grab logs. Possibly this is now occurring as the platform is starting to come under some load.
Next time this is happening could you see what process is consuming the memory and cpu?
Bit difficult to check processes during the failure as the first I know about it is when I lose services (this is not a smooth handover!) and see an outage for 1-2 mins before the cluster node (worker) becomes the master.
It got to the point yesterday where the cluster became too unstable (failing back and forth) and I had to shut down one of the nodes. I did notice once the "master" had finally stabilised as a single node that it had taken one of the ethernet ports - eth1 (the default internet route out) and had renamed it eth7. This meant that the systems were mismatched as the second node did not have an eth7 and I'm guessing therefore no default route out. Both servers have been successfully functioning as stand-alone nodes in the past with no hardware changes since joining them as a cluster.
I've bought both nodes back online this morning and the cluster seems to be behaving itself, but it had only been 5 minutes so far ;-)
