Barman Cloud - Part 2: Cloud Backup

In the first part of this blog, Jonathan explained how the barman-wal-archive command works. Now, assuming you followed those instructions, you have a properly configured PostgreSQL instance up and running. In this second part, I will show you how the barman-cloud-backup command works.

As you can guess from the command name itself, the barman-cloud-backup command allows you to execute backups directly from a PostgreSQL server and use an S3 compatible object store in the Cloud as destination.

postgres@vm:~ $ barman-cloud-backup --help
usage: barman-cloud-backup [-V] [--help] [-v | -q] [-P PROFILE] [-z | -j]
                           [-e {AES256,aws:kms}] [-t] [-h HOST] [-p PORT]
                           [-U USER] [--immediate-checkpoint] [-J JOBS]
                           [-S MAX_ARCHIVE_SIZE] [--endpoint-url ENDPOINT_URL]
                           destination_url server_name

This script can be used to perform a backup of a local PostgreSQL instance and
ship the resulting tarball(s) to the Cloud. Currently only AWS S3 is supported.

positional arguments:
  destination_url       URL of the cloud destination, such as a bucket in AWS
                        S3. For example: `s3://bucket/path/to/folder`.
  server_name           the name of the server as configured in Barman.

optional arguments:
  -V, --version         show program's version number and exit
  --help                show this help message and exit
  -v, --verbose         increase output verbosity (e.g., -vv is more than -v)
  -q, --quiet           decrease output verbosity (e.g., -qq is less than -q)
  -P PROFILE, --profile PROFILE
                        profile name (e.g. INI section in AWS credentials
                        file)
  -z, --gzip            gzip-compress the WAL while uploading to the cloud
  -j, --bzip2           bzip2-compress the WAL while uploading to the cloud
  -e {AES256,aws:kms}, --encryption {AES256,aws:kms}
                        Enable server-side encryption for the transfer.
                        Allowed values: 'AES256'|'aws:kms'.
  -t, --test            Test cloud connectivity and exit
  -h HOST, --host HOST  host or Unix socket for PostgreSQL connection
                        (default: libpq settings)
  -p PORT, --port PORT  port for PostgreSQL connection (default: libpq
                        settings)
  -U USER, --user USER  user name for PostgreSQL connection (default: libpq
                        settings)
  --immediate-checkpoint
                        forces the initial checkpoint to be done as quickly as
                        possible
  -J JOBS, --jobs JOBS  number of subprocesses to upload data to S3 (default:
                        2)
  -S MAX_ARCHIVE_SIZE, --max-archive-size MAX_ARCHIVE_SIZE
                        maximum size of an archive when uploading to S3
                        (default: 100GB)
  --endpoint-url ENDPOINT_URL
                        Override default S3 endpoint URL with the given one

Now that we have a clearer idea of the command and its options, we are ready to execute our first cloud backup:

postgres@vm:~ $ barman-cloud-backup -P barman-cloud \
  -e AES256 -j --immediate-checkpoint -J 4 \
  s3://barman-s3-test/ pg12

Once the backup has been completed successfully, the base directory containing the backup will be in your S3 bucket. Let’s check it, building the destination path with the server name and the base directory:

postgres@vm:~ $ aws s3 --profile barman-cloud ls s3://barman-s3-test/pg12/base/
                           PRE 20200713T120856/

Alternatively, you can use barman-cloud-backup-list, but in this article I would like to focus on the mechanics behind it.

The 20200711T092548 directory contains all files related to the backup that we have just executed. Let’s look at its contents:

postgres@vm:~ $ aws s3 --profile barman-cloud ls s3://barman-s3-test/pg12/base/20200713T120856/
2020-07-13 12:09:08       1138 backup.info
2020-07-13 12:09:07    9263096 data.tar.bz2

As we can see, there is a compressed and encrypted file containing our backup (data.tar.bz2) and a file called backup.info containing the information related to the backup. We can restore the backup by copying and unzipping the data.tar.bz2 file on our local server as shown below:

postgres@vm:~ $ aws s3 --profile barman-cloud cp s3://barman-s3-test/pg12/base/20200713T120856/data.tar.bz2 restore-dir
download: s3://barman-s3-test/pg12/base/20200713T120856/data.tar.bz2 to restore-dir/data.tar.bz2

postgres@vm:~ $ cd restore-dir

postgres@vm:~/restore-dir $ tar xjvf data.tar.bz2

postgres@vm:~/restore-dir $ ls
PG_VERSION    conf.d        pg_commit_ts  pg_ident.conf  pg_notify    pg_snapshots  pg_subtrans  pg_wal                postgresql.conf
backup_label  data.tar.bz2  pg_dynshmem   pg_logical     pg_replslot  pg_stat       pg_tblspc    pg_xact
base          global        pg_hba.conf   pg_multixact   pg_serial    pg_stat_tmp   pg_twophase  postgresql.auto.conf

Great! As we can see, all the files in the DATADIR of the PostgreSQL instance that we backed up, including the configuration files, are listed correctly here.

Conclusions

With the barman-cloud-backup command Barman introduces an important feature that allows you to execute and directly ship base backups from a local PostgreSQL server to cloud object storage services that are compatible with AWS S3 in just a few simple steps. It supports encryption, parallel upload, compression, and allows to save disk space on your local server and transfer safely the backups to the cloud.

With the release of Barman 2.11, which happened a few days ago, important new features have been introduced, including the barman-cloud-wal-restore and the barman-cloud-restore commands, which allow to retrieve the backup and WAL files from an object store such as AWS S3. In that regard, we will publish a new blog article soon explaining how to prepare the recovery of a PostgreSQL instance using these commands. Stay tuned.