Switchover and Switchback of CloudNativePG Replica Clusters in a Distributed Topology (K8s) - Part 2

This runbook details the operational procedures for performing a controlled Switchover and Switchback between two EDB Postgres Advanced 18 clusters managed by CloudNativePG (CNPG). This is Part 2 of the series. If you haven't set up the distributed topology yet, start with Part 1: Deployment of PostgreSQL Replica Cluster via Barman Cloud Plugin on CloudNativePG.


Objective

The goal is to safely rotate the Primary role between a Primary Cluster (cluster-primary) and a Replica Cluster (cluster-replica) within a Distributed Topology. This process ensures zero data loss by utilizing the CNPG native promotion and demotion workflow.


Environment Details:

  • Operator Version: 1.27.0
  • Database: EDB Postgres Advanced 18
  • Backup: Barman-Plugin
  • Primary Cluster: cluster-primary (Namespace: primary)
  • Replica Cluster: cluster-replica (Namespace: replica)

Phase 1: Initial Switchover (Primary → Replica)

Step 1: Verify the initial status of both clusters

Check the health and LSN of the primary and replica clusters before starting the operation.

Primary:

user% kubectl cnp status cluster-primary -n primary
Cluster Summary
Name                     primary/cluster-primary
System ID:                7611448666720448538
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Primary instance:        cluster-primary-1
Primary promotion time:  2026-02-27 07:48:28 +0000 UTC (310h54m54s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    185M
Current Write LSN:       0/9000060 (Timeline: 1 - WAL File: 000000010000000000000009)

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-primary
First Point of Recoverability:  2026-02-27 13:20:08 IST
Last Successful Backup:          2026-02-27 13:20:08 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000010000000000000008   @   2026-02-27T07:54:14.417091Z
Last Failed WAL:                -

Streaming Replication status
Replication Slots Enabled
Name                Sent LSN   Write LSN  Flush LSN  Replay LSN  Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
----                --------   ---------  ---------  ----------  ---------  ---------  ----------  -----      ----------  -------------  ----------------
cluster-primary-2  0/9000060  0/9000060  0/9000060  0/9000060   00:00:00   00:00:00   00:00:00    streaming  async       0              active
cluster-primary-3  0/9000060  0/9000060  0/9000060  0/9000060   00:00:00   00:00:00   00:00:00    streaming  async       0              active

Instances status
Name                Current LSN  Replication role  Status  QoS          Manager Version  Node
----                -----------  ----------------  ------  ---          ---------------  ----
cluster-primary-1  0/9000060    Primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-2  0/9000060    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-3  0/9000060    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Replica:

user% kubectl cnp status cluster-replica -n replica
Replica Cluster Summary
Name                     replica/cluster-replica
System ID:                7611448666720448538
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Designated primary:      cluster-replica-1
Source cluster:          cluster-primary
Primary promotion time:  2026-02-27 07:52:58 +0000 UTC (310h50m34s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    104M

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-replica
First Point of Recoverability:  -
Last Successful Backup:          -
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000010000000000000008   @   2026-02-27T07:54:22.112507Z
Last Failed WAL:                -

Instances status
Name                Current LSN  Replication role              Status  QoS          Manager Version  Node
----                -----------  ----------------              ------  ---          ---------------  ----
cluster-replica-1  0/9000000    Designated primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-2  0/9000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-3  0/9000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Step 2: Demote the Primary Cluster

Change the primary's replica section and obtain the demotion token.

Primary Cluster Configuration Change:

From:

  replica:
    primary: cluster-primary
    source: cluster-primary

To:

  replica:
    primary: cluster-replica
    source: cluster-replica

Obtain Token and Check Status:

user% kubectl get cluster cluster-primary -n primary \
  -o jsonpath='{.status.demotionToken}'
eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjEiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwQSIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTE0NDg2NjY3MjA0NDg1MzgiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC9BMDAwMDI4IiwidGltZU9mTGF0ZXN0Q2hlY2twb2ludCI6IlRodSBNYXIgMTIgMDY6NTA6NTEgMjAyNiIsIm9wZXJhdG9yVmVyc2lvbiI6IjEuMjcuMCJ9%

user% kubectl cnp status cluster-primary -n primary
Replica Cluster Summary
Name                     primary/cluster-primary
System ID:                7611448666720448538
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Designated primary:      cluster-primary-1
Source cluster:          cluster-replica
Primary promotion time:  2026-03-12 06:50:52 +0000 UTC (1m25s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    200M

Demotion token
Token                               eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjEiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwQSIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTE0NDg2NjY3MjA0NDg1MzgiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC9BMDAwMDI4IiwidGltZU9mTGF0ZXN0Q2hlY2twb2ludCI6IlRodSBNYXIgMTIgMDY6NTA6NTEgMjAyNiIsIm9wZXJhdG9yVmVyc2lvbiI6IjEuMjcuMCJ9
Validity                            valid
Latest checkpoint's TimeLineID     1
Latest checkpoint's REDO WAL file  00000001000000000000000A
Latest checkpoint's REDO location  0/A000028
Database system identifier          7611448666720448538 (ok)
Time of latest checkpoint          Thu Mar 12 06:50:51 2026
Version of the operator            1.27.0

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-primary
First Point of Recoverability:  2026-02-27 13:20:08 IST
Last Successful Backup:          2026-02-27 13:20:08 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              00000001000000000000000A   @   2026-03-12T06:52:15.544285Z
Last Failed WAL:                -

Instances status
Name                Current LSN  Replication role              Status  QoS          Manager Version  Node
----                -----------  ----------------              ------  ---          ---------------  ----
cluster-primary-1  0/A0000A0    Designated primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-2  0/A0000A0    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-3  0/A0000A0    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Step 3: Promote the Replica Cluster

Apply the promotionToken to the Replica Cluster configuration.

Replica Cluster Configuration Change:

From:

  replica:
    primary: cluster-primary
    source: cluster-primary

To:

  replica:
    primary: cluster-replica
    promotionToken: eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjEiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwQSIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTE0NDg2NjY3MjA0NDg1MzgiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC9BMDAwMDI4IiwidGltZU9mTGF0ZXN0Q2hlY2twb2ludCI6IlRodSBNYXIgMTIgMDY6NTA6NTEgMjAyNiIsIm9wZXJhdG9yVmVyc2lvbiI6IjEuMjcuMCJ9
    source: cluster-primary

Step 4: Verify status after first switchover/promotion

Ensure the roles have swapped correctly.

A] Replica cluster successfully promoted as primary:

user% kubectl cnp status cluster-replica -n replica
Cluster Summary
Name                     replica/cluster-replica
System ID:                7611448666720448538
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Primary instance:        cluster-replica-1
Primary promotion time:  2026-02-27 07:52:58 +0000 UTC (311h6m25s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    169M
Current Write LSN:       0/B006E90 (Timeline: 2 - WAL File: 00000002000000000000000B)

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-replica
First Point of Recoverability:  -
Last Successful Backup:          -
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              00000002000000000000000A   @   2026-03-12T06:57:33.229487Z
Last Failed WAL:                -

Streaming Replication status
Replication Slots Enabled
Name                Sent LSN   Write LSN  Flush LSN  Replay LSN  Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
----                --------   ---------  ---------  ----------  ---------  ---------  ----------  -----      ----------  -------------  ----------------
cluster-replica-2  0/B006E90  0/B006E90  0/B006E90  0/B006E90   00:00:00   00:00:00   00:00:00    streaming  async       0              active
cluster-replica-3  0/B006E90  0/B006E90  0/B006E90  0/B006E90   00:00:00   00:00:00   00:00:00    streaming  async       0              active

Instances status
Name                Current LSN  Replication role  Status  QoS          Manager Version  Node
----                -----------  ----------------  ------  ---          ---------------  ----
cluster-replica-1  0/B006E90    Primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-2  0/B006E90    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-3  0/B006E90    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

B] Primary cluster successfully demoted as replica:

user% kubectl cnp status cluster-primary -n primary
Replica Cluster Summary
Name                     primary/cluster-primary
System ID:                7611448666720448538
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Designated primary:      cluster-primary-1
Source cluster:          cluster-replica
Primary promotion time:  2026-03-12 06:50:52 +0000 UTC (9m32s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    216M

Demotion token
Token                               eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjEiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwQSIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTE0NDg2NjY3MjA0NDg1MzgiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC9BMDAwMDI4IiwidGltZU9mTGF0ZXN0Q2hlY2twb2ludCI6IlRodSBNYXIgMTIgMDY6NTA6NTEgMjAyNiIsIm9wZXJhdG9yVmVyc2lvbiI6IjEuMjcuMCJ9
Validity                            valid
Latest checkpoint's TimeLineID     1
Latest checkpoint's REDO WAL file  00000001000000000000000A
Latest checkpoint's REDO location  0/A000028
Database system identifier          7611448666720448538 (ok)
Time of latest checkpoint          Thu Mar 12 06:50:51 2026
Version of the operator            1.27.0

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-primary
First Point of Recoverability:  2026-02-27 13:20:08 IST
Last Successful Backup:          2026-02-27 13:20:08 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              00000002000000000000000A   @   2026-03-12T06:57:39.660756Z
Last Failed WAL:                -

Instances status
Name                Current LSN  Replication role              Status  QoS          Manager Version  Node
----                -----------  ----------------              ------  ---          ---------------  ----
cluster-primary-1  0/B000000    Designated primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-2  0/B000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-3  0/B000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Step 5: Check replication

Create a table on the new primary and verify on the replica.

Promoted Primary (cluster-replica):

user% kubectl cnp psql cluster-replica -n replica
psql (18.1.0)
Type "help" for help.

postgres=# \dt
Did not find any tables.
postgres=# create table test (id int);
CREATE TABLE
postgres=# insert into test values (1);
INSERT 0 1
postgres=# select * from test;
 id
----
  1
(1 row)

Demoted Replica (cluster-primary):

user% kubectl cnp psql cluster-primary -n primary
psql (18.1.0)
Type "help" for help.

postgres=# \dt
Did not find any tables.

Step 6: Take a fresh backup of the current primary

user% kubectl cnp backup cluster-replica -n replica --method=plugin --plugin-name=barman-cloud.cloudnative-pg.io
backup/cluster-replica-20260312133305 created

swapnilsuryawanshi@MAC-CR9L20YFN6 plugin % kubectl get backup -n replica
NAME                               AGE   CLUSTER           METHOD   PHASE       ERROR
cluster-replica-20260312133305    58s   cluster-replica   plugin   completed  

Step 7: Verify replication after backup

user% kubectl cnp psql cluster-primary -n primary
psql (18.1.0)
Type "help" for help.

postgres=# \dt
          List of tables
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | test | table | postgres
(1 row)

postgres=# select * from test;
 id
----
  1
(1 row)

Phase 2: Switchback (Replica → Primary)

Step 8: Demote the current primary (cluster-replica) to a replica

Change the config and retrieve the new demotion token.

Replica Cluster Configuration Change:

From:

  replica:
    primary: cluster-replica
    promotionToken: eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjEiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwQSIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTYyNzEyOTAzMDUyODIwNzMiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC9BMDAwMDI4IiwidGltZU9mTGF0ZXN0Q2hlY2twb2ludCI6IlRodSBNYXIgMTIgMDc6NTE6MjMgMjAyNiIsIm9wZXJhdG9yVmVyc2lvbiI6IjEuMjcuMCJ9
    source: cluster-primary

To:

  replica:
    primary: cluster-primary
    source: cluster-primary

Token Retrieval and Status:

user% kubectl get cluster cluster-replica -n replica -o jsonpath='{.status.demotionToken}'
eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjIiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAyMDAwMDAwMDAwMDAwMDAxMCIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTYyNzEyOTAzMDUyODIwNzMiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC8xMDAwMDAyOCIsInRpbWVPZkxhdGVzdENoZWNrcG9pbnQiOiJUaHUgTWFyIDEyIDA4OjEwOjA4IDIwMjYiLCJvcGVyYXRvclZlcnNpb24iOiIxLjI3LjAifQ==%

user% kubectl cnp status cluster-replica -n replica
Replica Cluster Summary
Name                     replica/cluster-replica
System ID:                7616271290305282073
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Designated primary:      cluster-replica-1
Source cluster:          cluster-primary
Primary promotion time:  2026-03-12 08:10:09 +0000 UTC (14s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    248M

Demotion token
Token                               eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjIiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAyMDAwMDAwMDAwMDAwMDAxMCIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTYyNzEyOTAzMDUyODIwNzMiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC8xMDAwMDAyOCIsInRpbWVPZkxhdGVzdENoZWNrcG9pbnQiOiJUaHUgTWFyIDEyIDA4OjEwOjA4IDIwMjYiLCJvcGVyYXRvclZlcnNpb24iOiIxLjI3LjAifQ==
Validity                            valid
Latest checkpoint's TimeLineID     2
Latest checkpoint's REDO WAL file  000000020000000000000010
Latest checkpoint's REDO location  0/10000028
Database system identifier          7616271290305282073 (ok)
Time of latest checkpoint          Thu Mar 12 08:10:08 2026
Version of the operator            1.27.0

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-replica
First Point of Recoverability:  2026-03-12 13:33:10 IST
Last Successful Backup:          2026-03-12 13:33:10 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              00000002.history   @   2026-03-12T08:10:15.149343Z
Last Failed WAL:                -

Instances status
Name                Current LSN  Replication role              Status  QoS          Manager Version  Node
----                -----------  ----------------              ------  ---          ---------------  ----
cluster-replica-1  0/100000A0    Designated primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-2  0/100000A0    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-3  0/100000A0    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Step 9: Promote the current replica (cluster-primary) to a primary

Apply the promotion token back to the original Primary.

Primary Cluster Configuration Change:

From:

  replica:
    primary: cluster-replica
    source: cluster-replica

To:

  replica:
    primary: cluster-primary
    promotionToken: eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjIiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAyMDAwMDAwMDAwMDAwMDAxMCIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTYyNzEyOTAzMDUyODIwNzMiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC8xMDAwMDAyOCIsInRpbWVPZkxhdGVzdENoZWNrcG9pbnQiOiJUaHUgTWFyIDEyIDA4OjEwOjA4IDIwMjYiLCJvcGVyYXRvclZlcnNpb24iOiIxLjI3LjAifQ==
    source: cluster-replica

Step 10: Check the status of both clusters

Verify the final switchback state.

A] cluster-primary successfully promoted back as primary:

user% kubectl cnp status cluster-primary -n primary
Cluster Summary
Name                     primary/cluster-primary
System ID:                7616271290305282073
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Primary instance:        cluster-primary-1
Primary promotion time:  2026-03-12 07:51:23 +0000 UTC (25m20s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    344M
Current Write LSN:       0/11001F18 (Timeline: 3 - WAL File: 000000030000000000000011)

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-primary
First Point of Recoverability:  2026-03-12 13:15:19 IST
Last Successful Backup:          2026-03-12 13:15:19 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000030000000000000010   @   2026-03-12T08:16:05.880203Z
Last Failed WAL:                -

Streaming Replication status
Replication Slots Enabled
Name                Sent LSN     Write LSN    Flush LSN    Replay LSN  Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
----                --------     ---------    ---------    ----------  ---------  ---------  ----------  -----      ----------  -------------  ----------------
cluster-primary-2  0/11001F18  0/11001F18  0/11001F18  0/11001F18  00:00:00   00:00:00   00:00:00    streaming  async       0              active
cluster-primary-3  0/11001F18  0/11001F18  0/11001F18  0/11001F18  00:00:00   00:00:00   00:00:00    streaming  async       0              active

Instances status
Name                Current LSN  Replication role  Status  QoS          Manager Version  Node
----                -----------  ----------------  ------  ---          ---------------  ----
cluster-primary-1  0/11001F18    Primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-2  0/11001F18    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-primary-3  0/11001F18    Standby (async)    OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

B] cluster-replica successfully demoted back as a replica:

user% kubectl cnp status cluster-replica -n replica
Replica Cluster Summary
Name                     replica/cluster-replica
System ID:                7616271290305282073
PostgreSQL Image:        docker.enterprisedb.com/k8s/edb-postgres-advanced:18-standard-ubi9
Designated primary:      cluster-replica-1
Source cluster:          cluster-primary
Primary promotion time:  2026-03-12 08:10:09 +0000 UTC (8m7s)
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Size:                    264M

Demotion token
Token                               eyJsYXRlc3RDaGVja3BvaW50VGltZWxpbmVJRCI6IjIiLCJyZWRvV2FsRmlsZSI6IjAwMDAwMDAyMDAwMDAwMDAwMDAwMDAxMCIsImRhdGFiYXNlU3lzdGVtSWRlbnRpZmllciI6Ijc2MTYyNzEyOTAzMDUyODIwNzMiLCJsYXRlc3RDaGVja3BvaW50UkVET0xvY2F0aW9uIjoiMC8xMDAwMDAyOCIsInRpbWVPZkxhdGVzdENoZWNrcG9pbnQiOiJUaHUgTWFyIDEyIDA4OjEwOjA4IDIwMjYiLCJvcGVyYXRvclZlcnNpb24iOiIxLjI3LjAifQ==
Validity                            valid
Latest checkpoint's TimeLineID     2
Latest checkpoint's REDO WAL file  000000020000000000000010
Latest checkpoint's REDO location  0/10000028
Database system identifier          7616271290305282073 (ok)
Time of latest checkpoint          Thu Mar 12 08:10:08 2026
Version of the operator            1.27.0

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      s3-store/cluster-replica
First Point of Recoverability:  2026-03-12 13:33:10 IST
Last Successful Backup:          2026-03-12 13:33:10 IST
Last Failed Backup:              -
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000030000000000000010   @   2026-03-12T08:16:08.35668Z
Last Failed WAL:                -

Instances status
Name                Current LSN  Replication role              Status  QoS          Manager Version  Node
----                -----------  ----------------              ------  ---          ---------------  ----
cluster-replica-1  0/11000000    Designated primary            OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-2  0/11000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane
cluster-replica-3  0/11000000    Standby (in Replica Cluster)  OK      BestEffort  1.27.0           cnp-1.27.0-control-plane

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.11.0   N/A     Reconciler Hooks, Lifecycle Service

Step 11: Final Replication Status Verification

Ensure data consistency and test write/archive features on the original Primary.

Primary (cluster-primary):

user% kubectl cnp psql cluster-primary -n primary
psql (18.1.0)
Type "help" for help.

postgres=# \dt
          List of tables
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | test | table | postgres
(1 row)

postgres=# select * from test;
 id
----
  1
  2
(2 rows)

postgres=# insert into test values (3);
INSERT 0 1

postgres=# checkpoint;
CHECKPOINT

postgres=# select * from pg_switch_wal();
 pg_switch_wal
---------------
 0/11002180
(1 row)

Replica (cluster-replica):

user% kubectl cnp psql cluster-replica -n replica
psql (18.1.0)
Type "help" for help.

postgres=# \dt
          List of tables
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | test | table | postgres
(1 row)

postgres=# select * from test;
 id
----
  1
  2
  3
(3 rows)

Conclusion

This runbook demonstrated a complete, zero-data-loss Switchover and Switchback cycle between two CloudNativePG clusters using CNPG's native demotion token and promotion token workflow.

By following the structured two-phase approach, first rotating the Primary role to cluster-replica, then rotating it back to cluster-primary, we confirmed that:

  • The demotion token safely fences the outgoing primary before promotion begins.
  • The promotion token ensures the incoming primary resumes exactly where the previous primary left off, maintaining WAL continuity across timeline switches.
  • All data written during each phase was correctly replicated and verified on the standby side.

This pattern is fully compatible with the Barman Cloud Plugin for WAL archiving and backup, making it suitable for production distributed topologies on Kubernetes.

Share this