Table of Contents Previous Next


5 Operation : 5.2 Managing Backups Using a Retention Policy

1.
Determine and set a retention policy in the BART configuration file. A retention policy is a rule that determines when a backup is considered obsolete. The retention policy can be applied globally to all servers (see Section 4.1), but each server can override the global retention policy with its own (see Section 4.2.5).
2.
Use the MANAGE subcommand to categorize and manage backups according to the retention policy. Such functionality includes determining which active backups should be considered obsolete at this current point in time, selecting backups to keep indefinitely, and physically deleting obsolete backups. When an obsolete backup is deleted, its backup taken with the BACKUP subcommand along with that backup’s archived WAL files are deleted.
3.
Once the retention policies have been determined and verified, you can create a cron job to periodically run the MANAGE subcommand to evaluate the backups and then list and/or delete the obsolete backups.
See Section 5.2.5 on how retention policy management applied to full backups affects incremental backups.
Section 5.2.1 provides an overview of the terminology and types of retention policies.
Section 5.2.2 describes the concept of marking the status of backups according to the retention policy.
Section 5.2.3 describes setting the different types of retention policies.
Section 5.2.4 describes the process of managing the backups such as marking the backup status, keeping selected backups indefinitely, listing obsolete backups, and deleting obsolete backups.
Note: The examples shown in the previously listed sections were generated with BART version 1.1. The retention policy management process is the same for the current BART version, however the displayed output of the SHOW-BACKUPS and SHOW-SERVERS subcommands now include a few additional fields that do not influence the retention policy.
5.2.1 Overview
Active. The backup satisfies the retention policy applicable to its server. Such backups would be considered necessary to ensure the recovery safety for the server and thus should be retained.
Obsolete. The backup does not satisfy the retention policy applicable to its server. The backup is no longer considered necessary for the recovery safety of the server and thus can be deleted.
Keep. The backup is to be retained regardless of the retention policy applicable to its server. The backup is considered vital to the recovery safety for the server and thus should not be deleted for an indefinite period of time.
Redundancy Retention Policy. The redundancy retention policy relies on a specified, maximum number of most recent backups to retain for a given server. When the number of backups exceeds that maximum number, the oldest backups are considered obsolete (except for backups marked as keep). Section 5.2.3.1 describes this type of retention policy.
Recovery Window Retention Policy. The recovery window retention policy relies on a time frame (the recovery window) for when a backup should be considered active. The boundaries defining the recovery window are the current date/time (the ending boundary of the recovery window) and the date/time going back in the past for a specified length of time (the starting boundary of the recovery window). If the date/time the backup was taken is within the recovery window (that is, the backup date/time is on or after the starting date/time of the recovery window), then the backup is considered active, otherwise it is considered obsolete (except for backups marked as keep). Section 5.2.3.2 describes this type of retention policy.
Thus, for the recovery window retention policy, the recovery window time frame dynamically shifts so the end of the recovery window is always the current date/time when the MANAGE subcommand is run. As you run the MANAGE subcommand at future points in time, the starting boundary of the recovery window moves forward in time.
You can see the starting boundary of the recovery window at any point in time by running the SHOW-SERVERS subcommand. The RETENTION POLICY field of the SHOW-SERVERS subcommand displays the starting boundary of the recovery window.
When a backup is initially created with the BACKUP subcommand, it is always recorded with active status.
It is only by using the MANAGE subcommand at a particular point in time that active backups are evaluated to determine if their status should be changed to obsolete in accordance with the retention policy.
In addition, it is only when the MANAGE subcommand is invoked either with no options or with only the -s option (in order to specify the database server) that active backups are evaluated and also marked (that is, internally recorded by BART) as obsolete. See Section 5.4.7 for information on the MANAGE subcommand usage with its options.
Using the MANAGE subcommand, specify the -c option along with the backup identifier or name with the -i option. If you wish to keep this particular backup indefinitely, use -c keep, otherwise use -c nokeep.
If you used the -c nokeep option, the backup status is changed back to active. The next time the MANAGE subcommand is used, the backup is re-evaluated to determine if its status needs to be changed back to obsolete based upon the current retention policy in the BART configuration file.
Note that if the retention_policy parameter is set in a certain manner, you run the MANAGE subcommand to mark the backups according to that retention_policy setting, and then you change or disable the retention_policy parameter by commenting it out, the current, marked status of the backups are probably inconsistent with the current retention_policy setting.
If you wish to modify the backup status to be consistent with the current retention_policy setting, then perform the following:
If there are backups currently marked as obsolete that would no longer be considered obsolete under a new retention policy, run the MANAGE subcommand with the -c nokeep option for such backups to change the obsolete status to active status. You can also specify the -i all option, which would change all backups back to active status, including those currently marked as keep.
Run the MANAGE subcommand either with no options or with only the -s option to reset the marked status based upon the new retention_policy setting in the BART configuration file.
Deletion of backups with the MANAGE -d option relies on the last occasion when the backups have been marked.
The retention policy is determined by the retention_policy parameter in the BART configuration file. To set it globally for all servers, see Section 4.1. To set it by database server, see Section 4.2.5.
To use the redundancy retention policy, set retention_policy = max_number BACKUPS where max_number is a positive integer designating the maximum number of most recent backups.
The keyword BACKUPS must always be specified in plural form (for example, 1 BACKUPS).
BART will accept a maximum integer value of 2,147,483,647 for max_number, however, a realistic, practical value based on your system environment must always be used.
The redundancy retention policy is the default type of retention policy if all keywords BACKUPS, DAYS, WEEKS, and MONTHS following the max_number integer are omitted as shown by the following example:
The SHOW-SERVERS subcommand displays the 3 backup redundancy retention policy in the RETENTION POLICY field:
To use the recovery window retention policy, set the retention_policy parameter to the desired length of time for the recovery window in one of the following ways:
Set to max_number DAYS to define the start date/time recovery window boundary as the number of days specified by max_number going back in time from the current date/time.
Set to max_number WEEKS to define the start date/time recovery window boundary as the number of weeks specified by max_number going back in time from the current date/time.
Set to max_number MONTHS to define the start date/time recovery window boundary as the number of months specified by max_number going back in time from the current date/time.
The keywords DAYS, WEEKS, and MONTHS must always be specified in plural form (for example, 1 DAYS, 1 WEEKS, or 1 MONTHS).
BART will accept a maximum integer value of 2,147,483,647 for max_number, however, a realistic, practical value based on your system environment must always be used.
Invoking BART in debug mode (with the -d option) using the MANAGE subcommand with the -n option displays the calculated time length of the recovery window based on the retention_policy setting and the current date/time.
For example, using the following retention_policy settings:
If the MANAGE subcommand is invoked in debug mode along with the -n option on 2015-04-17, the following results are displayed:
For server acctg, 72 hours translates to a recovery window of 3 days.
For server dev, 504 hours translates to a recovery window of 21 days (3 weeks).
For server hr, 2160 hours translates to a recovery window of 90 days (3 months).
Note: For a setting of max_number MONTHS, the calculated, total number of days for the recovery window is dependent upon the actual number of days in the preceding months from the current date/time. Thus, max_number MONTHS is not always exactly equivalent to max_number x 30 DAYS. (For example, if the current date/time is in the month of March, a 1-month recovery window would be equivalent to only 28 days because the preceding month is February. Thus, for a current date of March 31, a 1-month recovery window would start on March 3.) However, the typical result is that the day of the month of the starting recovery window boundary will be the same day of the month of when the MANAGE subcommand is invoked.
Using the SHOW-SERVERS subcommand, the RETENTION POLICY field displays the start of the recovery window.
The start of the 3-day recovery window displayed in the RETENTION POLICY field is 2015-04-07 14:57:36 EDT when the SHOW-SERVERS subcommand is invoked on 2015-04-10.
At this current point in time, backups taken on or after 2015-04-07 14:57:36 EDT would be considered active. Backups taken prior to 2015-04-07 14:57:36 EDT would be considered obsolete except for backups marked as keep.
The start of the 3-week recovery window displayed in the RETENTION POLICY field is 2015-03-20 14:59:42 EDT when the SHOW-SERVERS subcommand is invoked on 2015-04-10.
At this current point in time, backups taken on or after 2015-03-20 14:59:42 EDT would be considered active. Backups taken prior to 2015-03-20 14:59:42 EDT would be considered obsolete except for backups marked as keep.
The start of the 3-month recovery window displayed in the RETENTION POLICY field is 2015-01-10 14:04:23 EST when the SHOW-SERVERS subcommand is invoked on 2015-04-10.
At this current point in time, backups taken on or after 2015-01-10 14:04:23 EST would be considered active. Backups taken prior to 2015-01-10 14:04:23 EST would be considered obsolete except for backups marked as keep.
The MANAGE subcommand is used to evaluate and categorize backups according to the retention policy set in the BART configuration file. When a backup is first created with the BACKUP subcommand, it always marked as active. Upon usage of the MANAGE subcommand, an active backup may be marked as obsolete. Obsolete backups can then be deleted.
Section 5.2.4.1 discusses the rules for deleting backups dependent upon the backup status and the subcommand used.
Section 5.2.4.2 shows how to retain a backup indefinitely by marking it as keep as well as resetting backups marked as obsolete and keep back to active status.
Section 5.2.4.3 demonstrates the general process for evaluating, marking, then deleting obsolete backups.
The MANAGE subcommand deletion relies solely upon how a backup status is currently marked (that is, internally recorded by BART). The current setting of the retention_policy parameter in the BART configuration file is ignored.
The DELETE subcommand relies solely upon the current setting of the retention_policy parameter in the BART configuration file. The current active, obsolete, or keep status of a backup is ignored.
Thus, the deletion behavior of the MANAGE subcommand and the DELETE subcommand are based on different aspects of the retention policy.
The specific deletion rules for the MANAGE and DELETE subcommands are as follows:
The MANAGE subcommand with the -d option can only delete backups marked as obsolete. This deletion occurs regardless of the current retention_policy setting in the BART configuration file.
Under a redundancy retention policy currently set with the retention_policy parameter in the BART configuration file, any backup regardless of its marked status, can be deleted with the DELETE subcommand when the backup identifier or name is specified with the -i option, and if the current total number of backups for the specified database server is greater than the maximum number of redundancy backups currently specified with the retention_policy parameter. If the total number of backups is less than or equal to the specified, maximum number of redundancy backups, then no additional backups can be deleted using DELETE with the -i backup option.
Under a recovery window retention policy currently set with the retention_policy parameter in the BART configuration file, any backup regardless of its marked status, can be deleted with the DELETE subcommand when the backup identifier or name is specified with the -i option, and if the backup date/time is not within the recovery window currently specified with the retention_policy parameter. If the backup date/time is within the recovery window, then it cannot be deleted using DELETE with the -i backup option.
Invoking the DELETE subcommand with the -i all option results in the deletion of all backups regardless of the retention policy and regardless of whether the status is marked as active, obsolete, or keep.
Note 1: Deletion occurs only if the total number of backups for the specified database server is greater than the specified, maximum number of redundancy backups currently set with the redundancy_policy parameter in the BART configuration file.
Note 2: Deletion occurs only if the backup is not within the recovery window currently set with the redundancy_policy parameter in the BART configuration file.
Use the MANAGE subcommand with the -c keep option to retain such backups indefinitely as shown by the following example:
The next time the MANAGE subcommand is invoked with either no options or with only the -s option, the active backup may be marked as obsolete according to the current retention_policy setting.
Based upon the current number of backups for the database server for a redundancy retention policy or the current date/time for a recovery window retention policy, when the MANAGE subcommand is invoked it evaluates active backups for the database server specified by the -s option or for all database servers if -s all is specified or the -s option is omitted.
Note: The status of backups currently marked as obsolete or keep are not changed. To re-evaluate such backups and then classify them, their status must first be reset back to active with the MANAGE -c nokeep option. See Section 5.2.2 for information.
Invoke the MANAGE subcommand with the -n option to perform a dry run to observe which active backups would be changed to obsolete according to the retention policy:
The dry run shows that backups 1428502049836 and 1428422324880 would be marked as obsolete.
Invoke the MANAGE subcommand omitting the -n option to change and mark the status of the backups as obsolete:
The obsolete backups can be observed in a number of ways. Use the MANAGE subcommand with the -l option to list the obsolete backups:
The STATUS field of the SHOW-BACKUPS subcommand displays the current status:
The details of an individual backup can be displayed using the SHOW-BACKUPS subcommand with the -t option. Note the status in the BACKUP STATUS field.
Use the MANAGE subcommand with the -d option to physically delete the obsolete backups including the unneeded WAL files.
The SHOW-BACKUPS subcommand now displays the remaining backups marked as active or keep:
Invoke the MANAGE subcommand with the -n option to perform a dry run to observe which active backups would be changed to obsolete according to the retention policy.
The dry run shows that backups 1428684544008 and 1428590536488 would be marked as obsolete.
Invoke the MANAGE subcommand omitting the -n option to change and mark the status of the backups as obsolete:
The obsolete backups can be observed in a number of ways. Use the MANAGE subcommand with the -l option to list the obsolete backups:
The STATUS field of the SHOW-BACKUPS subcommand displays the current status:
The details of an individual backup can be displayed using the SHOW-BACKUPS subcommand with the -t option. Note the status in the BACKUP STATUS field.
Use the MANAGE subcommand with the -d option to physically delete the obsolete backups including the unneeded WAL files.
The SHOW-BACKUPS subcommand now displays the remaining backups marked as active or keep:
The retention policy rules are applied to full backups. Determining when a backup is obsolete by the redundancy retention policy is based solely by the number of full backups. All incremental backups are excluded from the comparison count against the retention_policy setting for the maximum number of backups. Determining when a backup is obsolete by the recovery window retention policy is based solely by the backup date/time of the full backup. The backup date/times of any successive incremental backups in the chain are ignored when comparing with the recovery window.
The actions applied by the MANAGE and DELETE subcommands on a full backup are applied to all incremental backups in the chain in the same manner.
The following are some specific points regarding the MANAGE and DELETE subcommands on incremental backups:
When the MANAGE subcommand is invoked, the status applied to the full backup is also applied to all successive incremental backups in the chain.
The MANAGE subcommand with the -c { keep | nokeep } option cannot specify the backup identifier or backup name of an incremental backup with -i backup option. The -i backup option can only specify the backup identifier or backup name of a full backup. The option -i all is allowed to be used. When the MANAGE subcommand with the -c { keep | nokeep } option is applied to a full backup, the same status change is made to all incremental backups in the chain.
The DELETE subcommand with the -s server -i backup options is allowed to specify the backup identifier or backup name of an incremental backup in which case that incremental backup along with all its successive incremental backups are deleted, thus shortening that backup chain.
When the redundancy retention policy is used and the MANAGE subcommand is invoked, the status of the oldest, active, full backups are changed to obsolete if the number of full backups exceeds the maximum number specified by the retention_policy parameter in the BART configuration file.
See Section 5.2.3.1 for information on the redundancy retention policy.
When determining the number of backups that exceeds the number specified by the retention_policy parameter, only full backups are counted for the comparison. The number of incremental backups is not included in the count for this comparison against the retention_policy parameter setting.
The following examples show usage of the MANAGE and DELETE subcommands when a 3 backup redundancy retention policy is in effect as shown by the following server configuration:
The following is the current set of backups. (In these examples, some columns have been omitted from the SHOW-BACKUPS output in order to display the relevant information in a more observable manner.)
Backup chain: 1481749619582 => 1481749651927 => 1481749673603 => 1481749696905
The MANAGE subcommand is invoked as shown by the following:
Second backup chain: 1481749997807 => 1481750098924
The MANAGE subcommand is invoked, but now with a total of four active full backups.
Invoking the MANAGE subcommand with the -d option deletes the entire obsolete backup chain.
The following section provides an example using the recovery window retention policy. Example usage of the MANAGE subcommand with other options along with usage of the DELETE subcommand are shown in this next section.
When the recovery window retention policy is used and the MANAGE subcommand is invoked, the status of active, full backups are changed to obsolete if the date/time of the full backup is outside of the recovery window (that is, the full backup date/time is prior to the start of the recovery window as defined the by current date/time when the MANAGE subcommand is invoked, and then going back in time by the amount of time set by the retention_policy parameter in the BART configuration file.
See Section 5.2.3.2 for information on the recovery window retention policy.
The following examples show usage of the MANAGE and DELETE subcommands when a 1-day recovery window retention policy is in effect as shown by the following server configuration:
The following is the current set of backups. (In these examples, some columns have been omitted from the SHOW-BACKUPS output in order to display the relevant information in a more observable manner.)
First backup chain: 1481552078404 => 1481553088053 => 1481553914533 => 1481554802918 => 1481559014359
Second backup chain: 1481553651165 => 1481554203288 => 1481559303348
The MANAGE subcommand is invoked when the first full backup 1481552078404 falls out of the recovery window. When the MANAGE subcommand is invoked, it is 2016-12-13 09:20:03 EST, thus making the start of the 1-day recovery window at 2016-12-12 09:20:03 EST exactly one day earlier. This backup was taken at 2016-12-12 09:14:39 EST, which is about 5 ½ minutes before the start of the recovery window, thus making the backup obsolete.
The following example shows how the entire backup chain is changed back to active status by invoking the MANAGE subcommand with the -c nokeep option on the full backup of the chain.
The following example shows usage of the DELETE subcommand on an incremental backup. The specified incremental backup 1481554802918 in the first backup chain as well as its successive incremental backup 1481559014359 are deleted.
The results show that incremental backup 1481554802918 as well as its successive backup 1481559014359 are no longer listed by the SHOW-BACKUPS subcommand.
The MANAGE subcommand is invoked again. This time both backup chains are marked obsolete since the full backups of both chains fall out of the start of the recovery window, which is now 2016-12-12 09:55:03 EST.
The following shows usage of the MANAGE subcommand with the -c keep option to keep a backup chain indefinitely. The MANAGE subcommand with the -c keep option must specify the backup identifier or backup name of the full backup of the chain and not any incremental backup.
The following now displays the full backup 1481553651165 of the backup chain and its successive incremental backups 1481554203288 and 1481559303348 changed to keep status.
Finally, the MANAGE subcommand with the -d option is used to delete the obsolete backup chain.

5 Operation : 5.2 Managing Backups Using a Retention Policy

Table of Contents Previous Next