Using the S3 storage plugin

Use the S3 storage plugin to write WarehousePG (WHPG) backups directly to Amazon Simple Storage Service (S3) or any S3-compatible storage server, and restore them without storing large dump files on local segment disks. Run the gpbackup and gprestore commands with the --plugin-config option and a YAML configuration file, using the same configuration file for both backup and restore.

Configuring the plugin

The gpbackup_s3_plugin binary ships with whpg-backup, installed at $GPHOME/bin/gpbackup_s3_plugin on every host in your cluster. Point to it in a YAML configuration file located on the coordinator host. The file supports the following options:

executablepath: <absolute-path-to-plugin>
options: 
  region: <aws-region>
  endpoint: <s3-endpoint>
  aws_access_key_id: <access-key>
  aws_secret_access_key: <secret-key>
  bucket: <s3-bucket-name>
  folder: <s3-subfolder-path>
  encryption: [on|off]
  http_proxy: <proxy-url>
  backup_max_concurrent_requests: [int]
  backup_multipart_chunksize: [string]
  restore_max_concurrent_requests: [int]
  restore_multipart_chunksize: [string]

Where:

executablepath: Absolute path to the plugin, for example $GPHOME/bin/gpbackup_s3_plugin.
region: AWS region (ignored if endpoint is specified).
endpoint: Custom S3-compatible endpoint.
aws_access_key_id: Your AWS/S3 access key.
aws_secret_access_key: Your AWS/S3 secret key.
bucket: The target S3 bucket (must already exist).
folder: The backup sub-location. Created automatically if missing.
encryption: Enables SSL for the S3 connection. Default is on.
http_proxy: (Optional) If your WarehousePG segments do not have direct internet access, provide the URL of your corporate proxy server (e.g., http://proxy.example.com:8080).
backup_max_concurrent_requests: (Optional) Number of threads used to upload parts of a file simultaneously. Use this parameter in conjuction with the gpbackup --jobs option to increase your overall backup concurrency.
backup_multipart_chunksize: (Optional) The size of each data chunk uploaded to the S3 bucket. Use this parameter along with the gpbackup --jobs option and the backup_max_concurrent_requests parameter to fine tune your backups.
restore_max_concurrent_requests: (Optional) Number of threads used to download parts of a file during recovery.
restore_multipart_chunksize: (Optional) The size of each data chunk for each individual part of a file during a multipart transfer from S3 to the segments. Use this parameter along with restore_max_concurrent_requests to fine tune your restores.

Note

The S3 user you configure to run the backups via aws_access_key_id must have upload/delete permissions on that bucket for backups and open/download/view permissions for restores.
All segment hosts must have network access to the S3 endpoint, or the proxy configured via http_proxy.

Performing a backup

To perform a backup, use the gpbackup command with the --plugin-config option and specify the path to your YAML file:

gpbackup --dbname <database-name> --plugin-config /<path-to-config>/s3-config.yaml

The plugin organizes backups files using the following directory structure:

bucket/folder/backups/YYYYMMDD/YYYYMMDDHHMMSS/

Performing a restore

To restore a backup created with the S3 plugin, you must use the same --plugin-config file. You will also need the timestamp of the backup you wish to restore.

gprestore --timestamp <YYYYMMDDHHMMSS> --plugin-config /<path-to-config>/s3-config.yaml

Example

Create a configuration file named s3-test-config.yaml:

executablepath: $GPHOME/bin/gpbackup_s3_plugin
options: 
  region: us-west-2
  aws_access_key_id: test-s3-user
  aws_secret_access_key: asdf1234asdf
  bucket: gpdb-backup
  folder: test/backup3

On the coordinator, run the backup:

gpbackup --dbname demo --single-data-file --plugin-config /home/gpadmin/s3-test-config.yaml

Execute the restore (using the timestamp generated by the backup above):

gprestore --timestamp 20260318120000 --plugin-config /home/gpadmin/s3-test-config.yaml