Operator-driven disaster recovery for Hybrid Manager Innovation Release

From HM 2026.11, the HM operator can recover a Hybrid Manager (HM) installation for you. Instead of installing an empty HM and then restoring the control plane into it yourself with Velero, as in Disaster recovery for a single Hybrid Manager installation or Disaster recovery for a multi-DC Hybrid Manager deployment, you install the new HM with one extra field, spec.recoveryFrom, in its HybridControlPlane resource. The operator installs it against the object storage the old HM left behind and restores the control plane from it.

What the operator recovers

The operator restores the HM control plane and the Postgres management resources from the old installation's backups. It doesn't restore your Postgres clusters. The following table shows what the operator recovers and what you still restore yourself.

The internal databases are what make operator-driven recovery different from the manual procedure: they bootstrap from the old backups and come up holding the old data, instead of being initialized empty and restored into afterwards.

Prerequisites

• The same HM version on both sides. Install the new HM at the version the old one was running. The operator compares spec.version with the version recorded in the old backup folder and stops the installation if they differ: a recovery restores a version, it doesn't upgrade one.

• HM 2026.11 or later. Earlier versions don't label the Postgres management resources the operator restores, so a backup taken by an earlier HM brings none of them back.

• HM operator 2.3.0 or later. Earlier operators skip the stages that run after the installation, so a recovery can reach Completed without restoring the Location resources or the resources of your Postgres clusters. See the operator compatibility matrix.

• Both kinds of backup in the old object storage. See Checking the old object storage.

• Read access to the old bucket from the new environment. If object storage is reached through workload identity, the identity of the new environment needs access to the old bucket as well as its own.

• The image pull secret on the new cluster, created as for any installation. See Preparing your environment. It's environment setup, not restored state, so the recovery never restores it.

• edbctl version 1.11.0 or later. See EDB Hybrid Manager CLI.

Gathering the values of the old installation

Collect four values that describe the installation you're recovering:

export OLD_BUCKET=<old_bucket>
export OLD_BACKUP_ID=<old_backup_folder>
export OLD_LOCATION_ID=<old_location_id>
export NEW_BACKUP_ID=<new_backup_folder>

Where:

• OLD_BUCKET is the object storage bucket the old HM wrote its backups to.

• OLD_BACKUP_ID is the spec.globalParameters.internal_backup_folder of the old HM.

• OLD_LOCATION_ID is the spec.componentsParameters.upm-beacon.beacon_location_id of the old HM. Take it from the old HybridControlPlane resource, for example from your GitOps repository: nothing in the bucket records it, and if the old cluster is gone you can't look it up. If you no longer have it, use mode: full, where locationID is optional. full disables streaming on every restored location, so re-enable it by hand on any location that survived, as in Restoring the Postgres clusters. See mode.

• NEW_BACKUP_ID is where the recovered HM writes its own backups. Use a new, unused folder name of 12 lowercase letters and digits. It must differ from OLD_BACKUP_ID. See Backup folders.

Checking the old object storage

Confirm that the old object storage holds what the recovery needs. A recovery reads one folder of one bucket, edb-internal-backups/${OLD_BACKUP_ID}/:

edb-internal-backups/<folder>/
├── databases/<db>/          internal database backups: base/ and wals/
├── velero/backups/<name>/   Velero backup of the Kubernetes state
└── hm-metadata.json         the HM version that wrote this folder

Both databases/ and velero/backups/ must be populated: Velero holds the Kubernetes state, and the internal database backups hold the control-plane data. At least one internal database needs a base backup in databases/<db>/base/. The recovery stops without one, and warns, naming them, about any database that has none: the installation brings those up empty.

To check that the internal databases have base backups, list their backups. This example uses the AWS CLI; use the equivalent for your object storage:

aws s3 ls --recursive s3://${OLD_BUCKET}/edb-internal-backups/${OLD_BACKUP_ID}/databases/

If the old HM is still reachable, take a fresh set of backups first. Take the list of internal databases to back up from the ScheduledBackup resources, not from the list of Postgres clusters: an internal cluster without backups configured is deliberately never backed up.

kubectl get scheduledbackup -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.spec.cluster.name}{"\n"}{end}'

Writing the recovery resource

Take the HybridControlPlane resource of the HM you're recovering, give it a new internal_backup_folder, and add spec.recoveryFrom. Edit the existing resource and substitute the values from Gathering the values of the old installation:

apiVersion: edbpgai.edb.com/v1alpha1
kind: HybridControlPlane
metadata:
  name: edbpgai
spec:
  version: <the version of the old HM>
  globalParameters:
    internal_backup_folder: <NEW_BACKUP_ID>
  recoveryFrom:
    mode: full            # or primary-only: see mode
    target:
      objectStoreSecretRef:
        name: edb-object-storage
        namespace: edb-system
      internalBackupFolder: <OLD_BACKUP_ID>
      locationID: <OLD_LOCATION_ID>
    reuse:
      objectStore: true
    preflight:
      enforce: true
    claim: false          # true only for a real DR: see claim

Backup folders

The backup folder of the recovered HM must differ from the old one. The recovered HM reads the old folder and writes its own backups to its own folder, so a target.internalBackupFolder equal to globalParameters.internal_backup_folder is rejected.

Reusing the old folder is the most common mistake when you build the resource from the old one, because the old resource already carries the old folder in globalParameters. Replace that value with the new folder before you apply the resource.

mode

Set primary-only when only the location that hosted the control plane is gone and the other locations survive and keep running their Postgres clusters, and full when other locations are gone too. primary-only requires target.locationID.

The mode decides which restored Location resources come back with streaming disabled:

primary-only leaves the surviving locations streaming, which is what lets the recovered control plane take back the Postgres clusters still running there. With full, re-enable streaming by hand on the locations that survived, as in Restoring the Postgres clusters. A wrong mode is recoverable but not silent: primary-only where other locations were lost too leaves the recovered control plane streaming to locations that are gone, until you disable it on them as in Disable resource streaming on an affected location.

target

Identify the old installation in target: where its backups are stored, which folder holds them, and which location the old control plane ran in. It has three fields:

• objectStoreSecretRef: the secret that describes the old object storage, its provider, bucket and credentials, in the same format as the edb-object-storage secret of a running HM. The bucket is read in place and read-only, so read access is enough. The name and the namespace are both required.

• internalBackupFolder: the internal_backup_folder of the old HM, OLD_BACKUP_ID.

• locationID: the location the old control plane ran in, OLD_LOCATION_ID. It must be a valid DNS label. It's required for primary-only and optional for full, but worth setting either way when several HM installations share a bucket.

reuse.objectStore

Set it to true. It decides whether the object storage credentials come back from the backup, and its default, false, isn't what a recovery into the same bucket needs.

Only the first case is supported. See Known limitations.

With true, the operator restores the object storage credentials from the backup. These are the edb-object-storage secret, the per-project edb-object-storage-prj-* secrets, and the slo-kv-store config map that records which object storage each project is bound to. They come back together on purpose. If the recovery used the old bucket but bound the projects again from scratch, the projects would end up bound to the default object storage, permanently.

Regardless of this setting, the operator keeps a renamed copy of every object storage credential in the hm-recovery namespace, because restoring your Postgres clusters later reads their backups from the old bucket.

The image pull secrets are never restored. They must exist before the installation can start, so they're always the new environment's own.

preflight.enforce

Defaults to true, which makes a failed preflight check stop the installation rather than letting it build a control plane on data that can't be restored. Set it to false only to turn the failures into warnings, for example when the old HM ran an operator earlier than 2.2.0, whose backup folder has no hm-metadata.json to check the version against. Confirm the versions match yourself first. The failures it lets through are recorded in status.recovery.ignoredPreflightFailures.

claim

Defaults to false. Leave it off to rehearse a recovery, and set it to true only for a real disaster recovery, when the old HM is gone.

With claim: true, the recovery records in the old backup folder that this HM has taken it over. If the old HM comes back, it reads that record and fences itself: it pauses its backups, fences its internal databases and stops the server that manages its Postgres clusters. The claim can't be taken back.

Once the control plane is recovered, the claim also removes the old HM from the federation of this one. If you rebuild it later, it can only rejoin through a new federation join.

If another HM has already claimed the same folder, the recovery fails and stays failed, and you must rebuild the environment.

Installing the new Hybrid Manager

Apply the resource and let the operator install the new HM. Follow Install Hybrid Manager on Kubernetes, except for the step that creates the install secrets.

Monitoring the recovery progress

Read the current phase, stage and message from the status of the HybridControlPlane resource:

kubectl get hcp <name> -o jsonpath='{.status.recovery.phase}/{.status.recovery.stage}{"\n"}'
kubectl get hcp <name> -o jsonpath='{.status.recovery.message}{"\n"}'

The recovery runs in stages, before and after the installation:

Failed isn't final. A failed stage is run again a minute later, and the installation waits meanwhile. The cause is usually outside the resource, such as object storage that isn't reachable yet, so a recovery can show Failed and still complete once the cause clears. Read status.recovery.message before treating it as the outcome.

kubectl delete restore recovery-0-restorelocations -n velero

If the installed HM can't support RestoreLocations or RestorePGManagement, the stage is skipped rather than run, and the recovery goes on without it. The skipped stages and the reason are listed in status.recovery.skippedStages, and the Completed message names them. Restore what a skipped stage left out by hand, as in Restore resources. Velero can already read the old backups through the recovery-source backup storage location.

Troubleshooting preflight check failures

If a check fails, status.recovery.message names it. If the old bucket can't be read, the checks after that one are skipped.

Validating the recovery

Confirm that the recovery succeeded by checking what it restored, as well as its phase. Start with the phase of the HybridControlPlane resource and of the recovery:

kubectl get hcp <name> -o jsonpath='phase={.status.phase} recovery={.status.recovery.phase}{"\n"}'
# phase=deployed recovery=Completed

Both backup storage locations should be Available, which means Velero reached the object storage through both folders:

kubectl get backupstoragelocation -n velero \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,ACCESS:.spec.accessMode,PREFIX:.spec.objectStorage.prefix'
# default          Available  ReadWrite  edb-internal-backups/<new>/velero
# recovery-source  Available  ReadOnly   edb-internal-backups/<old>/velero

If the recovery reports that it kept the existing edb-object-storage secret, that's the expected outcome, because that secret already describes the object storage the new HM can reach.

The Location resources of the old HM should be back. List every location and check the Streaming column:

edbctl location list

Streaming is disabled on the locations that mode selects: every restored location with full, or only target.locationID with primary-only. That follows the mode you chose, not which locations were actually lost, so it may not match them yet. Before you remove any Postgres resources, make sure streaming is disabled on every affected location and on no other, as in Restoring the Postgres clusters.

The Postgres management resources should be back too, in the same namespace as the Location resources:

kubectl get clusterwrappers,remotepgs,backupwrappers -A

Once the recovery is Completed, the internal databases look like new ones in their specification, with bootstrap set to initdb. That's expected, because their recovered bootstrap is only read when they're created, and the operator then puts back their normal specification. The data is the recovered one.

Restoring the Postgres clusters

Restore the Postgres clusters yourself, because the operator brings back the Postgres management resources but not the clusters themselves. Whether a location streams decides what HM does with the clusters recorded there.

Go through the locations in this order:

1. Re-enable streaming on every location that survived, if the recovery disabled it, as full does. The clusters, replicas and groups still running there need it, to be taken back and repaired in the next steps.

   To re-enable streaming on a location, run:

   → WrapCopy

   edbctl dr set-location-streaming --id <location-id> --enable

   Where <location-id> is the ID of the location, as shown in the Location ID column of edbctl location list. Then check the Streaming column of the same output to confirm it took effect.

2. Before you remove anything, confirm that streaming is disabled on every affected location and on no other. An affected location is one that was lost, or the one the recovered HM runs in, when it reuses the old location ID. If a lost location still streams, for example because primary-only was used when other locations were lost too, disable it as in Disable resource streaming on an affected location. If a surviving location is left disabled, the clusters with replicas or groups there can't be repaired in the next step, because a forced failover or dropping a lost group needs streaming.

3. Remove the Postgres clusters, replicas and groups that ran in the lost locations, as in Remove Postgres clusters, replicas and groups from an affected location. Delete each cluster that ran only in lost locations, and repair in place each one that still has replicas or groups in a surviving location.

4. Only then re-enable streaming on the location the recovered HM runs in, with the command in step 1, if it reuses the old location ID. If you re-enable it before the removal, HM creates the clusters that ran there again from scratch, rather than from their backups.

5. Restore each deleted cluster from the HM console, from its most recent backup, as in Restore Postgres clusters.

Known limitations

• Your Postgres clusters are restored by hand, one at a time, from the HM console. See Restoring the Postgres clusters.

• Custom managed storage locations aren't restored. If the old HM had any, restore them as in the first step of Restore resources before you restore the Postgres clusters that back up to them.

• The recovered HM keeps using the bucket of the old one. Recovering into an environment with its own bucket isn't supported: the restored resources still point at the old bucket, and the backups of your Postgres clusters stay there. Use the manual procedure instead.

• Identity provider (IdP) settings aren't restored. Configure your IdP on the recovered HM again.