Automating Barman with Puppet: it2ndq/barman (part two)

In the first part of this article we configured Vagrant to execute two Ubuntu 14.04 Trusty Tahr virtual machines, respectively called pg and backup. In this second part we will look at how to use Puppet to set up and configure a PostgreSQL server on pg and back it up via Barman from the backup box.

Puppet: configuration

After defining the machines as per the previous article, we need to specify the required Puppet modules that librarian-puppet will manage for us.

Two modules are required:

1. puppetlabs/postgresql (http://github.com/puppetlabs/puppetlabs-postgresql/) to install PostgreSQL on the pg VM
2. it2ndq/barman (http://github.com/2ndquadrant-it/puppet-barman) to install Barman on backup

Both modules will be installed from Puppet Forge. For the puppetlabs/postgresql module, we’ll have to use version 4.2.0 at most at the moment, as the latest version (4.3.0) is breaking the postgres_password parameter we’ll be using later (see this pull request). Let’s create a file called Puppetfile containing this content in the project directory:

forge "http://forgeapi.puppetlabs.com"
mod "puppetlabs/postgresql", "<4.3.0"
mod "it2ndq/barman"

We can now install the Puppet modules and their dependencies by running:

$ librarian-puppet install --verbose

Although not essential, it’s preferable to use the option --verbose every time librarian-puppet is used. Without it the command is very quiet and it’s useful to have details about what it’s doing in advance. For example, without using --verbose, you may find out that you’ve wasted precious time waiting for a dependency conflict to be resolved, only to see an error many minutes later.

Upon successful completion of the command, a modules directory containing the barman and postgresql modules and their dependencies (apt, concat, stdlib) will be created in our working directory. In addition, librarian-puppet will create the Puppetfile.lock file to identify dependencies and versions of the installed modules, pinning them to prevent future updates. This way, subsequent librarian-puppet install runs will always install the same version of the modules instead of possible upgrades (in case an upgrade is required, librarian-puppet update will do the trick).

Now we can tell Vagrant we are using a Puppet manifest to provision the servers. We alter the Vagrantfile as follows:

Vagrant.configure("2") do |config|
  {
    :pg => {
      :ip      => '192.168.56.221',
      :box     => 'ubuntu/trusty64'
    },
    :backup => {
      :ip      => '192.168.56.222',
      :box     => 'ubuntu/trusty64'
    }
  }.each do |name,cfg|
    config.vm.define name do |local|
      local.vm.box = cfg[:box]
      local.vm.hostname = name.to_s + '.local.lan'
      local.vm.network :private_network, ip: cfg[:ip]
      family = 'ubuntu'
      bootstrap_url = 'http://raw.github.com/hashicorp/puppet-bootstrap/master/' + family + '.sh'

      # Run puppet-bootstrap only once
      local.vm.provision :shell, :inline => <<-eos
        if [ ! -e /tmp/.bash.provision.done ]; then
          curl -L #{bootstrap_url} | bash
          touch /tmp/.bash.provision.done
        fi
      eos

      # Provision with Puppet
      local.vm.provision :puppet do |puppet|
        puppet.manifests_path = "manifests"
        puppet.module_path = [".", "modules"]
        puppet.manifest_file = "site.pp"
        puppet.options = [
         '--verbose',
        ]
      end
    end
  end
end

With the lines we’ve just added, we’ve given Vagrant the instructions to provision the VMs using manifests/site.pp as the main manifest and the modules included in the modules directory. This is the final version of our Vagrantfile.

We now have to create the manifests directory:

$ mkdir manifests

and write in it a first version of site.pp. We’ll start with a very basic setup:

node backup {
  class { 'barman':
    manage_package_repo => true,
  }
}
node pg {}

We can now start the machines and see that on backup there is a Barman server with a default configuration (and no PostgreSQL on pg yet). Let’s log into backup:

$ vagrant ssh backup

and take a look at /etc/barman.conf:

# Main configuration file for Barman (Backup and Recovery Manager for PostgreSQL)
# Further information on the Barman project at www.pgbarman.org
# IMPORTANT: Please do not edit this file as it is managed by Puppet!
# Global options

[barman]
barman_home = /var/lib/barman
barman_user = barman
log_file = /var/log/barman/barman.log
compression = gzip
backup_options = exclusive_backup
minimum_redundancy = 0
retention_policy =
retention_policy_mode = auto
wal_retention_policy = main
configuration_files_directory = /etc/barman.conf.d

The next step is running a PostgreSQL instance on pg. We must be aware of the parameters required by Barman on the PostgreSQL server, so we need to set:

• wal_level at least at archive level
• archive_mode to on
• archive_command so that the WALs can be copied on backup
• a rule in pg_hba.conf for access from backup

All of these parameters can be easily set through the puppetlabs/postgresql module. In addition, on the Barman server, we need:

• a PostgreSQL connection string
• a .pgpass file for authentication
• a SSH command
• to perform the SSH key exchange

it2ndq/barman generates a private/public keypair in ~barman/.ssh. However, automatically exchanging the keys between the servers requires the presence of a Puppet Master which is beyond the objectives of this tutorial (it will be part of the next instalment, which will focus on the setup of a Puppet Master and the barman::autoconfigure class) – therefore this last step will be performed manually.

We edit the site.pp file as follows:

node backup {
  class { 'barman':
    manage_package_repo => true,
  }
  barman::server {'test-server':
    conninfo     => 'user=postgres host=192.168.56.221',
    ssh_command  => 'ssh postgres@192.168.56.221',
  }
  file { '/var/lib/barman/.pgpass':
    ensure  => 'present',
    owner   => 'barman',
    group   => 'barman',
    mode    => 0600,
    content => '192.168.56.221:5432:*:postgres:insecure_password',
  }
}

node pg {
  class { 'postgresql::server':
    listen_addresses     => '*',
    postgres_password    => 'insecure_password',
    pg_hba_conf_defaults => false,
  }
  postgresql::server::pg_hba_rule {'Local access':
    type        => 'local',
    database    => 'all',
    user        => 'all',
    auth_method => 'peer',
  }
  postgresql::server::pg_hba_rule {'Barman access':
    type        => 'host',
    database    => 'all',
    user        => 'postgres',
    address     => '192.168.56.222/32',
    auth_method => 'md5',
  }
  postgresql::server::config_entry {
    'wal_level'       : value => 'archive';
    'archive_mode'    : value => 'on';
    'archive_command' : value => 'rsync -a %p barman@192.168.56.222:/var/lib/barman/test-server/incoming/%f';
  }
  class { 'postgresql::server::contrib':
    package_ensure => 'present',
  }
}

Having changed the manifest, the provision has to be rerun:

$ vagrant provision

With the machines running, we can proceed with the key exchanges. We log into pg:

$ vagrant ssh pg

and we create the keypair for the postgres user, using ssh-keygen, leaving every field empty when prompted (so always pressing enter):

vagrant@pg:~$ sudo -iu postgres
postgres@pg:~$ ssh-keygen
postgres@pg:~$ cat .ssh/id_rsa.pub

The last command outputs a long alphanumeric string that has to be appended to the ~barman/.ssh/authorized_keys file on backup.

$ vagrant ssh backup
vagrant@backup:~$ sudo -iu barman
barman@backup:~$ echo "ssh-rsa ..." >> .ssh/authorized_keys

Similarly, we copy the public key of the barman user into the authorized_keys file of the postgres user on pg:

barman@backup:~$ cat .ssh/id_rsa.pub
ssh-rsa ...
barman@backup:~$ logout
vagrant@backup:~$ logout
$ vagrant ssh pg
vagrant@pg:~$ sudo -iu postgres
postgres@pg:~$ echo "ssh-rsa ..." >> .ssh/authorized_keys

At this point, we make a first connection in both directions between the two servers:

postgres@pg:$ ssh barman@192.168.56.222
barman@backup:$ ssh postgres@192.168.56.221

We can run barman check to verify that Barman is working correctly:

barman@backup:~$ barman check all
Server test-server:
        ssh: OK
        PostgreSQL: OK
        archive_mode: OK
        archive_command: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        compression settings: OK
        minimum redundancy requirements: OK (have 0 backups, expected at least 0)

Every line should read “OK”. Now, to perform a backup, simply run:

barman@backup:$ barman backup test-server

A realistic configuration

The Barman configuration used so far is very simple, but you can easily add a few parameters to site.pp and take advantage of all the features of Barman, such as the retention policies and the new incremental backup available in Barman 1.4.0.

We conclude this tutorial with a realistic use case, with the following requirements:

• a backup every night at 1:00am
• the possibility of performing a Point In Time Recovery to any moment of the last week
• always having at least one backup available
• reporting an error via barman check in case the newest backup is older than a week
• enabling incremental backup to save disk space

We use the Puppet file resource to create a .pgpass file with the connection parameters and a cron resource to generate the job to run every night. Finally, we edit the barman::server to add the required Barman parameters.

The end result is:

node backup {
  class { 'barman':
    manage_package_repo => true,
  }
  barman::server {'test-server':
    conninfo                => 'user=postgres host=192.168.56.221',
    ssh_command             => 'ssh postgres@192.168.56.221',
    retention_policy        => 'RECOVERY WINDOW OF 1 WEEK',
    minimum_redundancy      => 1,
    last_backup_maximum_age => '1 WEEK',
    reuse_backup            => 'link',
  }
  file { '/var/lib/barman/.pgpass':
    ensure  => 'present',
    owner   => 'barman',
    group   => 'barman',
    mode    => 0600,
    content => '192.168.56.221:5432:*:postgres:insecure_password',
  }
  cron { 'barman backup test-server':
    command => '/usr/bin/barman backup test-server',
    user    => 'barman',
    hour    => 1,
    minute  => 0,
  }
}
node pg {
  class { 'postgresql::server':
    listen_addresses  => '*',
    postgres_password => 'insecure_password',
    pg_hba_conf_defaults => false,
  }
  postgresql::server::pg_hba_rule {'Local access':
    type        => 'local',
    database    => 'all',
    user        => 'all',
    auth_method => 'peer',
  }
  postgresql::server::pg_hba_rule {'Barman access':
    type        => 'host',
    database    => 'all',
    user        => 'postgres',
    address     => '192.168.56.222/32',
    auth_method => 'md5',
  }
  postgresql::server::config_entry {
    'wal_level'       : value => 'archive';
    'archive_mode'    : value => 'on';
    'archive_command' : value => 'rsync -a %p barman@192.168.56.222:/var/lib/barman/test-server/incoming/%f';
  }
}

Conclusion

With 51 lines of Puppet manifest we managed to configure a pair of PostgreSQL/Barman servers with settings similar to those we might want on a production server. We have combined the advantages of having a Barman server to handle backups with those of having an infrastructure managed by Puppet, reusable and versionable.

In the next and final post in this series of articles we will look at how to use a Puppet Master to export resource between different machines, thus allowing the VMs to exchange the parameters required for correct functioning via the barman::autoconfigure class making the whole setup process easier.