Skip to content
Announcing BigAnimal: Fully managed PostgreSQL in the Cloud
Get startedContact usCareersDocsPlans

How to Promote a Cloud Native PostgreSQL Replica

Jonathan Battiato4/2/2021
PostgreSQLKubernetes

In case you missed it, in a previous post we provided an overview of the Cloud Native PostgreSQL (CNP) plugin for kubectl, including installation instructions. In this blog post, we will show how to use the promote subcommand of such plugin to perform a switchover in a Cloud Native PostgreSQL cluster.

This useful command allows you to specify the replica pod to be promoted as the new primary. The magic embedded in the operator will take care of the Kubernetes services as well, routing read/write connections to the new promoted instance.

Basic usage


kubectl cnp promote --help

Output:

Promote a certain server as a primary

Usage:
  kubectl cnp promote [cluster] [server] [flags]

Flags:
  -h, --help   help for promote
[...]


It is enough to specify the cluster name and the Pod name, which runs the target replica that we want to promote as the new primary. The CNP operator will act accordingly.

Use case: Maintenance of a Kubernetes node

As Kubernetes Admin/SysAdmin, we are often required to perform some maintenance tasks on the physical machines that are hosting the worker nodes of our Kubernetes cluster. The CNP operator has been designed in such a way to offer true high availability of the PostgreSQL cluster, spreading CNP instances on different K8S worker nodes.

With the promote command, you can switch over to a specific instance of the CNP cluster, routing your application load as well to point to the new primary. As a result, the maintenance tasks can be performed on the worker node where the former primary was running.

Consider the following example:

We have three K8S worker nodes and three CNP instances, one on each physical worker node. We must perform a kernel upgrade on the node where the primary instance is running.

We start checking the current situation:

kubectl get pods -w -o wide

Take note of the following IPs as we will need them later:

cluster-example-custom-1 IP: 10.244.1.4
cluster-example-custom-2 IP: 10.244.3.5

As we described in the previous article, we can use the status command as well:

kubectl cnp status cluster-example-custom

Now that we have a clear overview of the cluster conditions, we promote the second replica with:

kubectl cnp promote cluster-example-custom cluster-example-custom-2

Once this command returns the prompt, we will monitor the ongoing procedure by executing the kubectl cnp status cluster-example-custom command a few times in a row. If executed frequently for a short time interval, the cluster status will change before our eyes, and show some different stages.

First, the requested replica is promoted, while the old primary is stopped.

Second, the old primary is restarted and configured as a new replica.

And … what about our applications?

Application connections are routed to the primary through the cluster-example-custom-rw service, whose endpoint is configured according to the current situation.

As expected, the cluster-example-custom-rw service is now pointing to the new primary IP address. The CNP operator handles the service as well, changing the endpoint from:

to:

Use the following command to get the endpoints of the read/write service:

kubectl get endpoints cluster-example-custom-rw

Now we can upgrade the kernel on the Kubernetes worker node, where the former primary instance is running, since it is now just a replica. Keep in mind that if you use local storage for PostgreSQL, you are advised to take advantage of the “nodeMaintenanceWindow” setting as described in the documentation to prevent the standard self-healing procedure to kick in.

Conclusions

The CNP plugin is more and more emerging as a great tool for managing Cloud Native PostgreSQL clusters in Kubernetes. Any DBA appreciates the ability to perform a PostgreSQL switchover using a single command.

It is also worth mentioning that endpoints to the actual pods will be replaced with the proper working ones in the CNP services. Thanks to the operator, applications are lifted from this responsibility and can continue to work with no changes to their configurations.
 

Jonathan Battiato is a CKA, Linux SysAdmin at EDB. His core areas of expertise include Kubernetes administration, Linux administration, PostgreSQL backup, and replication architecture. Prior to working at EDB, Jonathan worked as an Linux Administrator, PostgreSQL consultant, PostgreSQL and Linux tea ...