Commit scopes v5
Commit scopes give applications granular control about durability and consistency of EDB Postgres Distributed.
A commit scope is a named rule that describes the behavior of COMMIT replication. The actual behavior depends on whether a commit scope uses Group Commit, Commit At Most Once, Lag Control, or combination of these.
To use Group Commit, first define a commit scope, which determines the PGD nodes involved in the commit of a transaction. Once a scope is established, you can configure a transaction to use Group Commit as follows:
You must set the commit scope before the transaction writes any data.
For this example, you might previously have defined the commit scope as:
This example assumes a node group named
example_bdr_group exists and includes at least two PGD nodes as members, either directly or in subgroups. Any transaction committed in the
example_scope requires one extra confirmation from a PGD node in the group. Together with the origin node, this accounts for "ANY 2" nodes out of the group, on which the transaction is guaranteed to be visible and durable after the commit.
Rules for commit scopes can depend on the node the transaction is committed on, that is, the node that acts as the origin for the transaction. To make this transparent for the application, PGD allows a commit scope to define different rules depending on where the transaction originates from.
For example, consider an EDB Postgres Distributed cluster with nodes spread across two data centers: a left and a right one. Assume the top-level PGD node group is called
top_group. You can use the following commands to set up
subgroups and create a commit scope requiring all nodes in the local data center to confirm the transaction but only one node from the remote one:
Now, using the
example_scope on any node that's part of
left_dc uses the first scope. Using the same scope on a node that's part of
right_dc uses the second scope. This is an effective way of creating an inverted scope without having to juggle scope names in the application.
Each group can also have a default commit scope specified using the
bdr.alter_node_group_option admin interface.
Making the above scopes the default ones for all transactions originating on nodes in those groups looks like this:
PGD nodes can send confirmations for a transaction at different times. In increasing levels of protection, from the perspective of the confirming node, these are:
received— A remote PGD node confirms the transaction immediately after receiving it, prior to starting the local application.
replicated— Confirms after applying changes of the transaction but before flushing them to disk.
durable— Confirms the transaction after all of its changes are flushed to disk.
visible(default) — Confirms the transaction after all of its changes are flushed to disk and it's visible to concurrent transactions.
In rules for commit scopes, you can append these confirmation levels to the node group definition in parentheses with
ON as follows:
ANY 2 (right_dc) ON replicated
ALL (left_dc) ON visible(default)
ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable
The grammar for commit scopes is composed as follows:
node_group— Name of a node group.
( group_commit_parameter = value )— Options for Group Commit.
transaction_tracking(Boolean) — Specifies whether to track status of transaction.
conflict_resolution(enum) — Specifies how to handle conflicts. Possible values are
async, meaning to resolve conflicts asynchronously during replication using the conflict resolution policy, or
eagermeaning that conflicts are resolved eagerly during COMMIT by aborting one of the conflicting transactions.
commit_decision(enum) — Specifies how the COMMIT decision is made. The value
commit_scope_groupspecification also affects the COMMIT decision, not just durability. The value
partnermeans the partner node decides whether transactions can be committed. This value is allowed only on 2 data node groups. The value
raftmeans the COMMIT decision is done using Raft consensus independent of
ABORT ON ( abort_on_parameter = value )— Allows automatic transaction abort on timeout.
timeout(interval) — Timeout in milliseconds (accepts other units).
DEGRADE ON ( degrade_on_parameter = value )— Allows degrading to asynchronous operation on timeout
timeout(interval) — Timeout in milliseconds (accepts other units) after which operation becomes asynchronous.
require_write_lead(Boolean) — Specifies whether the node must be a write lead to be able to switch to asynchronous mode.
( lag_control_parameter = value )— Options for Lag Control:
max_lag_size(int) — Maximum allowed lag based on WAL bytes.
max_lag_time(interval) — Maximum allowed lag based on wall clock sampling.
max_commit_delay(interval) — Maximum delay that can be injected to commit to try to keep within the lag limits.
CAMO commit scope kind is mostly syntactic sugar for
GROUP COMMIT (transaction_tracking = true, commit_decision = partner) with additional
DEGRADE ON clause. It's expected that
GROUP COMMIT will eventually gain the
DEGRADE ON clause as well, making
CAMO syntax deprecated.
While the grammar for
synchronous_standby_names and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example
synchronous_standby_names = 'ANY 1 (..)' is equivalent to a commit scope of
ANY 2 (...). This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction.
bdr.add_commit_scope creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope.
Alternatively, you can invoke it multiple times with the same
commit_scope_name but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction.
To change a specific rule for a single origin node group in a commit scope, you can use the function
You can use
bdr.remove_commit_scope to drop a single rule in a commit scope. If you define multiple rules for the commit scope, you must invoke this function once per rule to fully remove the entire commit scope.
Removing a commit scope that's still used as default by a node group isn't allowed.