EDB Docs - EDB Postgres Distributed (PGD) v6.5.0 - Upgrading PGD 4 and Postgres to PGD 6

Upgrade a PGD 4 cluster to PGD 6 and Postgres to a new major version at the same time, moving from HARP Proxy to Connection Manager, using one of two approaches: full downtime or a rolling upgrade in place. If you're not also upgrading Postgres, see Upgrading PGD 4 to PGD 6 instead.

Note

This procedure assumes HARP Proxy isn't co-located with a PGD node.

Before you begin

Confirm every node is running the supported starting version. See Supported PGD upgrade paths, then check the versions in your cluster.

To list all nodes in the BDR group along with their names, IDs, and statuses:

SELECT node_name, node_id, node_group_name, node_state 
FROM bdr.node_summary;

Upgrading with full downtime

Choose this approach when a maintenance window is acceptable and finishing quickly matters more than staying online. See Choosing between a rolling and full-downtime upgrade for the trade-offs against the rolling upgrade described below.

Disconnect applications from the cluster, and stop HARP Proxy on every node running it:
```
sudo systemctl stop harp-proxy
```
On each node, one at a time or all at once, install the combined PGD 6 and new Postgres major version package (for example, edb-pgd6-expanded-pg17), alongside the existing PGD 4 and Postgres packages:
```
dnf install edb-pgd6-expanded-pg<postgres_version> -y
```
Package managers install each Postgres major version to its own path, so the running node and its current binaries are untouched.
Create a fresh, empty data directory for the new Postgres version on each node using its initdb, matching the old cluster's checksum and locale settings. Leave this new cluster shut down:
```
<new_bindir>/initdb -D <new_datadir> --data-checksums
```
Add --locale, --encoding, or other initdb options as needed to match the old cluster's settings. Use a different <new_datadir> path if you don't want to use the default data directory location.
Copy each node's old cluster's postgresql.conf and postgresql.auto.conf, pg_hba.conf, and conf.d directory (if present) into the new data directory, adjusting for any configuration parameters that changed between Postgres major versions.
Stop Postgres on every node, so systemd doesn't automatically restart it while pgd node upgrade manages the process directly:
```
sudo systemctl stop postgres
```
pgd node upgrade doesn't require the old instance to be either running or stopped beforehand, since it stops and starts both the old and new instances itself several times during the run.
On each node, run pgd node upgrade with the --check option first, to perform a dry run that validates the upgrade without making changes:
```
pgd node <node_name> upgrade --check \
  --old-bindir <old_bindir> \
  --new-bindir <new_bindir> \
  --old-datadir <old_datadir> \
  --new-datadir <new_datadir> \
  --database <database_name> \
  --username <install_user>
```
- --old-bindir and --new-bindir are the old and new Postgres installation's bin directories.
- --old-datadir and --new-datadir are the old cluster's data directory and the empty new data directory you created earlier.
- --database is the PGD-enabled database name, and --username is the cluster's install user.
- --check runs the PGD-specific validation and pg_upgrade --check, without modifying either cluster. See Modes of operation for what running with --check validates.
Run pgd node upgrade again without --check to migrate the node's data into the new data directory, using the same options:
```
pgd node <node_name> upgrade \
  --old-bindir <old_bindir> \
  --new-bindir <new_bindir> \
  --old-datadir <old_datadir> \
  --new-datadir <new_datadir> \
  --database <database_name> \
  --username <install_user>
```
Add the --link option to use hard links instead of copying files, if the old and new data directories are on the same filesystem. See pg_upgrade in the PostgreSQL documentation for more information.
PGD itself upgrades to PGD 6 once you start the new data directory in the next step, with Connection Manager enabled by default, no extra step needed.
Stop HARP manager on every node:
```
sudo systemctl stop harp-manager
```
Monitor progress with sudo journalctl -u harp-manager -f.
Point Postgres at the new binaries and data directory on each node. For example, update the Environment=PGDATA and ExecStart lines in the postgres.service file:
```
Environment=PGDATA=<new_datadir>
ExecStart=<new_bindir>/postgres -D ${PGDATA}
```
Then reload the service definition and start it:
```
sudo systemctl daemon-reload
sudo systemctl start postgres
```
Each node comes up on the new Postgres version and PGD 6, with Connection Manager already enabled. You don't need to start HARP manager again.
Confirm every node has rejoined the cluster and is healthy:
```
pgd nodes list
```
Every node should show ACTIVE under Join State and Up under Node Status. Then continue to Moving to Connection Manager below.

Upgrading with a rolling upgrade, in place

HARP Proxy continues routing on all nodes until the entire cluster is upgraded to PGD 6 or higher. It keeps using its own leader-election mechanism throughout, since a 4.x cluster doesn't have a write leader. Start the upgrade on a node that isn't the current HARP Proxy leader, where <location> is the HARP location name (a data center or site defined in the HARP configuration, not a node name):

harpctl get leader <location>

Use the command-line utility pgd node upgrade to migrate each node's Postgres version and move it to Connection Manager in the same pass. It wraps the standard pg_upgrade, adding PGD-specific steps around it so replication slots and origins carry over correctly.

Upgrading CAMO pairs

If your cluster uses CAMO, upgrade each CAMO pair as a unit. BDR 4.x configures CAMO through the bdr.camo_pairs catalog, which doesn't carry over to PGD 6. You must deconfigure each pair before upgrading either of its two nodes, then recreate CAMO protection on the new commit-scope-based configuration once both nodes are upgraded. See Commit At Most Once for background on how CAMO pairs protect transactions.

Ensure bdr.enable_camo remains off for transactions on either of the two nodes, or redirect clients away from the two nodes. Removing the CAMO pairing while attempting to use CAMO leads to errors and prevents further transactions.
```
SHOW bdr.enable_camo;
```
Deconfigure CAMO by using bdr.remove_camo_pair to uncouple the pair:
```
SELECT bdr.remove_camo_pair('<node_group>', '<node1>', '<node2>');
```
Upgrade both nodes in the pair using the steps below, then return here to reconfigure CAMO.
Create a dedicated node group for the two nodes and move them into that node group:
```
SELECT bdr.create_node_group('<node_group>', parent_group_name := '<parent_group>');
```
Then, on each of the two nodes:
```
SELECT bdr.switch_node_group('<node_group>');
```

Create a commit scope for this node group, and thus the pair of nodes, to use CAMO:

SELECT bdr.create_commit_scope(
  commit_scope_name := '<scope_name>',
  origin_node_group := '<node_group>',
  rule := 'CAMO'
);

Reactivate CAMO protection, either by setting a default_commit_scope for the group:
```
SELECT bdr.alter_node_group_option(
  node_group_name := '<node_group>',
  config_key := 'default_commit_scope',
  config_value := '<scope_name>'
);
```
or by changing the clients to explicitly set bdr.commit_scope instead of bdr.enable_camo for their sessions or transactions:
```
SET bdr.commit_scope = '<scope_name>';
```
Allow clients to connect to the CAMO-protected nodes again, if necessary.

For each node, one at a time, starting with a node that isn't the current HARP Proxy leader:

Fence the node from HARP, then confirm it's fenced, so it doesn't become the leader partway through the upgrade:
```
harpctl fence <node_name>
harpctl get node <node_name>
```
Install the combined PGD 6 and new Postgres major version package (for example, edb-pgd6-expanded-pg17), alongside the existing PGD 4 and Postgres packages:
```
dnf install edb-pgd6-expanded-pg<postgres_version> -y
```
Package managers install each Postgres major version to its own path, so the running node and its current binaries are untouched.
Create a fresh, empty data directory for the new Postgres version using its initdb, matching the old cluster's checksum and locale settings. Leave this new cluster shut down:
```
<new_bindir>/initdb -D <new_datadir> --data-checksums
```
Add --locale, --encoding, or other initdb options as needed to match the old cluster's settings. Use a different <new_datadir> path if you don't want to use the default data directory location.
Copy the old cluster's postgresql.conf and postgresql.auto.conf, pg_hba.conf, and conf.d directory (if present) into the new data directory, adjusting for any configuration parameters that changed between Postgres major versions.
Stop Postgres on the node, so systemd doesn't automatically restart it while pgd node upgrade manages the process directly:
```
sudo systemctl stop postgres
```
pgd node upgrade doesn't require the old instance to be either running or stopped beforehand, since it stops and starts both the old and new instances itself several times during the run.
Run pgd node upgrade with the --check option first, to perform a dry run that validates the upgrade without making changes:
```
pgd node <node_name> upgrade --check \
  --old-bindir <old_bindir> \
  --new-bindir <new_bindir> \
  --old-datadir <old_datadir> \
  --new-datadir <new_datadir> \
  --database <database_name> \
  --username <install_user>
```
- --old-bindir and --new-bindir are the old and new Postgres installation's bin directories.
- --old-datadir and --new-datadir are the old cluster's data directory and the empty new data directory you created earlier.
- --database is the PGD-enabled database name, and --username is the cluster's install user.
- --check runs the PGD-specific validation and pg_upgrade --check, without modifying either cluster. See Modes of operation for what running with --check validates.
Run pgd node upgrade again without --check to migrate the node's data into the new data directory, using the same options:
```
pgd node <node_name> upgrade \
  --old-bindir <old_bindir> \
  --new-bindir <new_bindir> \
  --old-datadir <old_datadir> \
  --new-datadir <new_datadir> \
  --database <database_name> \
  --username <install_user>
```
Add the --link option to use hard links instead of copying files, if the old and new data directories are on the same filesystem. See pg_upgrade in the PostgreSQL documentation for more information.
PGD itself upgrades to PGD 6 once you start the new data directory in the next step, with Connection Manager enabled by default, no extra step needed.
Stop HARP manager on the node:
```
sudo systemctl stop harp-manager
```
Monitor progress with sudo journalctl -u harp-manager -f.

Point Postgres at the new binaries and data directory. For example, update the Environment=PGDATA and ExecStart lines in the postgres.service file:
```
Environment=PGDATA=<new_datadir>
ExecStart=<new_bindir>/postgres -D ${PGDATA}
```
Then reload the service definition and start it:
```
sudo systemctl daemon-reload
sudo systemctl start postgres
```
The node comes up on the new Postgres version and PGD 6, with Connection Manager enabled. You don't need to start HARP manager again.
Unfence the node from HARP:
```
harpctl unfence <node_name>
```
Confirm the node is healthy and running the new Postgres version:
```
pgd nodes list --versions
```
Repeat for each remaining node, one at a time, until every node is running PGD 6 and the new Postgres version. Then continue to Moving to Connection Manager below.

For the full set of options and flags, see the pgd node upgrade command reference.

Moving to Connection Manager

Once every node is running PGD 6 and the new Postgres version, regardless of which upgrade path you took, finish moving the cluster off HARP Proxy and onto Connection Manager.

Confirm every node is running the new version:
```
pgd nodes list --versions
```
Confirm every node reports the same protocol_version in bdr.group_raft_details, confirming the whole cluster has moved to the new Raft protocol version, not just that individual nodes report success:
```
SELECT node_name, protocol_version FROM bdr.group_raft_details;
```

Run the following query from any of the upgraded nodes to ensure SCRAM hashes are the same across all nodes for each user. This query is required before applications switch to Connection Manager:

DO $$
DECLARE
    rec RECORD;
    command TEXT;
    password TEXT;
BEGIN
    FOR rec IN SELECT rolname,rolpassword FROM pg_authid WHERE rolcanlogin = true AND rolpassword like 'SCRAM-SHA%'
    LOOP
        password := rec.rolpassword;
        command := 'ALTER ROLE ' || quote_ident(rec.rolname) || ' WITH ENCRYPTED PASSWORD ' || quote_literal(password);
        EXECUTE command;
    END LOOP;
END;
$$;
SELECT bdr.wait_slot_confirm_lsn(NULL,NULL);

Enable node group routing as appropriate for your global or local routing requirement. For local routing, enable it on subgroups. For global routing, enable it on the top group:
```
SELECT bdr.alter_node_group_option(node_group_name := '<group_name>', config_key := 'enable_routing', config_value := true::TEXT);
```
Switch your application to Connection Manager.
Stop any HARP manager and HARP Proxy services still running:
```
sudo systemctl stop harp-manager
sudo systemctl stop harp-proxy
```
Monitor progress with sudo journalctl -u harp-manager -f.
Confirm the cluster is healthy:
```
pgd cluster show --health
```

Upgrading PGD 4 and Postgres to PGD 6 v6.5.0

Note

Before you begin

Upgrading with full downtime

Upgrading with a rolling upgrade, in place

Upgrading CAMO pairs

Moving to Connection Manager

← Prev

↑ Up

Next →