This section describes the basic concepts of the block-level incremental backup referred to simply as an incremental backup.As previously described, the full base backup uses the pg_basebackup utility program to accomplish the task. An incremental backup is a unique, utility for BART.The amount of time required to produce an incremental backup is generally less than a full base backup as modified relation blocks are saved instead of all, full relation files of the database cluster.This also results in less disk space than a full base backup taken in plain text format. Note that if the full base backup is taken in tar format, this saves disk space as well.Generally, all BART features such as retention policy management apply to incremental backups as well as full base backups. See Section 5.2.5 for information on retention policy management as applied to incremental backups.2.2.1 RequirementsNote: Incremental backups cannot be taken from standby database servers. Only full base backups can be taken from standby database servers.
• Create or select an operating system account to be used as the BART user account. See Section 4.1 for information on choosing and setting up the BART user account.
• In the BART configuration file, the cluster_owner parameter must be set to the Linux operating system user account that owns the directory of the database cluster from which incremental backups are to be taken. The allow_incremental_backups parameter must be enabled. See Section 4.2.5 for information.
• A password-less SSH/SCP connection must be established between the BART user account on the BART host and the cluster_owner user account on the database server. Note: This password-less SSH/SCP connection must be established even if BART and the database server are running on the same host and the BART user account and the cluster_owner user account are the same account. See Section 4.2.1 for information.
• In addition to the BART host where the BART backup catalog resides, the BART package must also be installed on every remote database server on which incremental backups are to be restored. In order to restore an incremental backup, the bart program must be executable on the remote host by the remote user specified by the remote_host parameter in the BART configuration file or by the -r option when using the RESTORE subcommand to restore the incremental backup. See Section 220.127.116.11 for information on restoring incremental backups on remote hosts.
• For restoring incremental backups on a remote database server, a password-less SSH/SCP connection must be established between the BART user account on the BART host and the remote user on the remote host that is specified by the remote_host parameter in the BART configuration file or by the -r option when using the RESTORE subcommand to restore the incremental backup. See Section 18.104.22.168 for information on restoring incremental backups on remote hosts.
• Compression of archived WAL files in the BART backup catalog is not permitted for database servers on which incremental backups are to be taken. The wal_compression setting in the BART configuration file must not be enabled for those database servers. Disabled is the default setting unless the parameter is altered in the global section or the server section. See sections 4.1 and 4.2.5 for information on the wal_compression parameter.
• 2.2.2 Concept Overview
3. An initial full base backup must be taken with the BACKUP subcommand. This full base backup establishes the parent of the first incremental backup.
5. Incremental backups are taken using the BACKUP subcommand with the --parent option to specify the backup identifier or name of a previous, full base backup or an incremental backup. Any previous backup may be chosen as the parent as long as all backups belong to the same timeline.
6. The incremental backup process identifies which WAL files may contain changes from when the parent backup was taken to the starting point of the incremental backup. The corresponding MBM files are used to locate and copy the modified blocks to the incremental backup directory along with other database cluster directories and files. Instead of backing up all, full relation files, only the modified blocks are copied and saved. In addition, the relevant MBM files are condensed into one consolidated block map (CBM) file that is stored with the incremental backup. Note: Multiple block copier threads can be used to copy the modified blocks to the incremental backup directory. See Section 4.2.5 for setting the thread_count parameter in the BART configuration file.
7. The restore process for an incremental backup is invoked using the RESTORE subcommand in the same manner as restoring a full base backup. The -i option specifies the backup identifier or name of the incremental backup to restore. The restore process begins by going back through the chain of past, parent incremental backups until the initial full base backup starting the chain is identified. This full base backup provides the initial set of directories and files to be restored to the location specified with the -p option.
8. In order to determine which blocks have changed since the parent backup, the WAL files created from the time of the parent backup up to the start of the incremental backup are scanned by the WAL scanner program bart-scanner.The WAL scanner determines from the WAL files which blocks have been modified and records the information in a file called the modified block map (MBM) file. One MBM file is created for each WAL file.The MBM file is stored in the archive path directory backup_path/server_name/archived_wals where backup_path is the BART backup catalog parent directory specified in the global section of the BART configuration file and server_name is the lowercase conversion of the database server name specified for this database server in the server section of the BART configuration file. This is the same directory where the archived WAL files are stored in the BART backup catalog.The following is the content of the archive path showing the MBM files created for the WAL files. (The user name and group name of the files have been removed from the example to list the WAL files and MBM files in a more comparable manner.)In preparation for any incremental backup, the WAL files should be scanned as soon as they are copied to the BART backup catalog. Thus, the WAL scanner should be running as soon as the WAL files from the database cluster are archived to the BART backup catalog.If the BART backup catalog contains WAL files that have not yet been scanned, starting the WAL scanner begins scanning these files.Should a WAL file failed to be scanned resulting in a missing MBM file, the WAL scanner can be used to specify an individual WAL file to be scanned.The WAL files produced at the time of the parent backup up to the start of the incremental backup contain the information of which blocks were modified during that time interval. That information has been consolidated into an MBM file for each WAL file by the WAL scanner.The MBM files for the relevant WAL files are read, and the information is used to copy the modified blocks from the database cluster to the BART backup catalog. When compared to a full, plain backup, the number and sizes of relation files can be significantly less for the incremental backup.For comparison, the following is an abbreviated list of the files copied to the archived base subdirectory of a full base backup for one database:In contrast, the following is the content of the archived base subdirectory of the same database from a subsequent incremental backup:The information from the MBM files are consolidated into one file called a consolidated block map (CBM) file. During the restore operation for the incremental backup, the CBM file is used to identify the modified blocks to be restored for that backup.In addition, the incremental backup also stores other required subdirectories and files from the database cluster as is done for full base backups.Restoring an incremental backup may require additional setup requirements depending upon the host on which the incremental backup is to be restored:
• BART Host. If an incremental backup is to be restored onto the same host where BART has been installed, the restore process is outlined in Section .
• The bart program must be available on the remote host because the RESTORE subcommand invocation for an incremental backup results in the execution of the bart program on the remote host to copy the modified blocks from the BART backup catalog on the BART host to the restore directory on the remote host.The RESTORE subcommand is used to restore an incremental backup by specifying its backup identifier or name with the -i option. All RESTORE options are used in the same manner as when restoring a full base backup.For each incremental backup, the CBM file is used to identify and copy blocks from the incremental backup.If there are new relations or databases identified in the CBM file, then relevant relation files are copied. If consolidated block map information is found indicating the drop of a relation or a database, then the relevant files are removed from the restore directory. Similarly if there is any indication of a table truncation, then the related files are truncated.See Section 5.4.8 for information on using the RESTORE subcommand for restoring an incremental backup.If an incremental backup is to be restored on a remote host on which BART has not been installed, then the following steps must be implemented.Step 1: Install BART on the remote host to which an incremental backup is to be restored. Use the instructions in Section 3.2 to install BART on this remote host.Note: No editing is needed in the BART configuration file bart.cfg installed on the remote host.Step 2: Determine the Linux operating system user account on the remote host to be used as the remote user specified by the remote_host parameter in the BART configuration file or by the -r option when using the RESTORE subcommand to restore the incremental backup. This remote user must also be the owner of the directory where the incremental backup is to be restored on the remote host. For example, the user account is typically enterprisedb for Advanced Server or postgres for PostgreSQLStep 3: Make sure a password-less SSH/SCP connection has been established from the BART user on the BART host to the remote user on the remote host. See Section 4.2.1 for information on setting up a password-less SSH/SCP connection.Step 4: When the remote user connects to the remote host, the remote user’s PATH environment variable must include the BART bin directory. For example, modify the user’s ~/.bashrc or ~/.bash_profile file to set the PATH environment variable such as in the following example:2.2.6 Creating a Backup ChainA backup chain is the set of backups consisting of a full base backup and all of its successive incremental backups. Tracing back on the parent backups of all incremental backups in the chain eventually leads back to that single, full base backup.It is possible to have a “multi-forked” backup chain, which is two or more successive lines of incremental backups, all of which begin with the same, full base backup. Thus, within the chain there is a backup that serves as the parent of more than one incremental backup.Since restoration of an incremental backup is dependent upon first restoring the full base backup, then all successive incremental backups up to, and including the incremental backup specified by the RESTORE subcommand, it is crucial to note the following:
• The actions of retention policy management are applied to the full base backup and all of its successive incremental backups within the chain in an identical manner as if they were one backup. Thus, use of retention policy management does not result in the breakup of a backup chain. See Section 5.2.5 for information on retention policy management of incremental backups.The allow_incremental_backups parameter is set to enabled in the BART configuration file to permit incremental backups on the listed database server:After the database server has been started with WAL archiving enabled to the BART backup catalog, the WAL scanner is started:A series of incremental backups are taken. The first incremental backup specifies the full base backup as the parent. Each successive incremental backup then uses the preceding incremental backup as its parent. See Section 5.4.3 for information on the BACKUP subcommand.The following output of the SHOW-BACKUPS subcommand lists the backup chain, which are backups full_1, incr_1-a, incr_1-b, and incr_1-c.Note that for the full base backup full_1, the BACKUP PARENT field contains none. For each incremental backup, the BACKUP PARENT field contains the backup identifier or name of its parent backup.A second backup chain is created in the same manner with the BACKUP subcommand. The following shows the addition of the resulting, second backup chain consisting of full base backup full_2 and incremental backups incr_2-a and incr_2-b.The following additional incremental backups starting with incr_1-b-1, which designates incr_1-b as the parent, results in the forking from that backup into a second line of backups in the chain consisting of full_1, incr_1-a, incr_1-b, incr_1-b-1, incr_1-b-2, and incr_1-b-3 as shown in the following list:Restoring an incremental backup is done with the RESTORE subcommand and its options in the same manner as for restoring a full base backup. Specify the backup identifier or backup name of the incremental backup to be restored as shown by the following:Restoring incremental backup incr_1-b as shown by the preceding example results in the restoration of full base backup full_1, then incremental backups incr_1-a and finally, incr_1-b.