You can use the Streaming Replication Analysis dashboard to monitor Streaming Replication and Failover Manager high-availability scenarios. To view the Streaming Replication Analysis dashboard, you must enable probes that monitor replication-related activity; to change a probe configuration, highlight the server name in the PEM client tree control, and select Manage Probes… from the Management dashboard.To view the Streaming Replication Analysis dashboard for the master node of a replication scenario, enable the following probes:To view the Streaming Replication Analysis dashboard for the standby node of a replication scenario, you must enable the following probe:Then, to open the Streaming Replication Analysis dashboard, navigate to the Monitoring tab, and:
1. Select the name of the agent that monitors the node from the Agents drop-down menu.
2. Select the name of the monitored server from the Servers drop-down menu.
3. Select Streaming Replication Analysis from the Dashboards drop-down menu.
4. Then, to open the Streaming Replication dashboard (see Figure 14.11), right click on the name of the master or standby node of the replication scenario in the Object Browser tree control, and select Streaming Replication Analysis from the Dashboards context menu.14.1.1 Monitoring Failover ManagerIf you are using EDB Failover Manager to monitor your replication scenario, you must manually install and configure Failover Manager. For detailed information about installing Failover Manager, visit the EnterpriseDB website at:To monitor the status of a Failover Manager cluster on the Streaming Replication dashboard, you must provide the following information on the Advanced tab of the server Properties dialog for each node of the cluster:
• Use the EFM Cluster Name field to specify the name of the Failover Manager cluster. The cluster name is the prefix of the name of the cluster properties file. For example, if your cluster properties file is named efm.properties, your cluster name is efm.
• Use the EFM Installation Path field to specify the location of the Failover Manager binary file. By default, the Failover Manager binary file is installed in /usr/efm-2.1/bin.After registering your servers, the Streaming Replication Analysis dashboard will display status information about your EFM cluster near the bottom of the dashboard (see Figure 14.12).The Failover Manager Cluster Status section of the Streaming Replication Analysis dashboard displays information about the monitored cluster:The Failover Manager Cluster Information table provides information about the Failover Manager cluster:
• The Properties column displays the name of the cluster property.
• The Values column displays the current value of the property.The Failover Manager Node Status table displays information about each node of the Failover Manager cluster:
• The Agent Type column displays the type of agent that resides on the node; the possible values are Master, Standby, Witness, Idle, and Promoting.
• The Address column displays the IP address of the node.
• The Agent column displays the status of the agent that resides on the node.
• The DB column displays the status of the database that resides on the node.
• The XLog Location column displays the transaction log location of the database.
• The Status Information column displays any error-related information about the node.
• The XLog Information column displays any error-related information about the transaction log.14.1.2 Replacing a Master NodeYou can use the PEM client to replace the Master node of a Failover Manager cluster with a standby node. To initiate the failover process, select Replace Cluster Master from the Management menu. A dialog opens, asking you to confirm that you wish to replace the current master node (see Figure 14.13).Select Yes to remove the current master node from the Failover Manager cluster and promote a standby node to the role of read/write master node within a Failover Manager cluster. The node with the highest promotion priority (defined in Failover Manager) will become the new master node. PEM will display a dialog, reporting the job status (see Figure 14.14).When the job completes and the Streaming Replication Analysis dashboard refreshes, you can review the Failover Manager Node Status table to confirm that a standby node has been promoted to the role of master within the Failover Manager cluster.Replication ensures that data written to the Master node of a cluster is preserved on a Standby node; if a problem occurs on the Master node (such as hardware failure), a Standby node can easily be promoted to replace the failed node. The behavior of a replication cluster can be described as:active/active – In an active/active cluster, the master node or nodes manages write transactions while the standby nodes are available for read requests. Streaming Replication with hot standby or EDB Postgres Multi-Master Replication manage active/active clusters. If you are using an active/active replication scenario, you probably have a PEM agent on each node of the cluster, and are actively monitoring each node with PEM.active/passive – In an active/passive cluster, only the master node is used for read and write transactions. Standby nodes ensure that in the event of a failure of the master node, a complete backup is available to replace the master node. RHCS (Red Hat Cluster Suite), Veritas Replicator, or Streaming Replication without hot standby manage active/passive clusters. If you are using an active/passive replication scenario, you are probably only actively monitoring the master node of your replication scenario with PEM.If PEM is configured to monitor the master and standby nodes, and a standby is promoted, the PEM agents that are currently monitoring the nodes will continue to monitor the newly promoted master unless the node on which the agent resides fails.If PEM is configured to only monitor the master node and the master node fails, the new master node will be unmonitored unless you configure a standby PEM agent to takeover monitoring of the new master. To create an agent hierarchy that allows an agent to assume monitoring a newly promoted Master, you should install a PEM agent on the Master node, and on any Standby node that might be promoted to master; each agent should be bound to the PEM server.To configure PEM to promote an agent on a Standby node to monitor the newly promoted Master, you must:
• To enable takeover of a server, right-click on the name of a server in the PEM client tree control, and select Properties from the context menu. When the Properties dialog opens, check the box next to Allow takeover? on the PEM Agent tab of the Server Properties dialog. Each server that will potentially be promoted to the role of Master should be configured to allow takeover by another agent.After allowing takeover of the server, add configuration steps to your failover script that instruct the PEM agent on a promoted Standby node (the new Master node) to assume monitoring the database server.To instruct the agent to takeover the monitoring of a server, the failover process must create a file in a special flag directory which will instruct the agent to take responsibility for the specified server. By default, the flag directory used by the agent is:Where $TMPDIR is the temporary directory for the user account under which the agent runs.The user account is usually root on a Linux system or Administrator on Windows. You can override the directory path by specifying a value for the AgentFlagDir configuration option in the registry on Windows, or in the agent_flag_dir parameter in the agent configuration file on Linux.touch /tmp/pem/agent-agent_id/takeover-server-server_idagent_id is the numeric identifier of the agent that should takeover the monitoring of the serverserver_id is the numeric identifier of the server that will be taken over.To find the agent_id and server_id, log into the PEM client, and highlight the name of the agent or server; the numeric identifier will be displayed in the ID row on the Properties pane of the PEM client.