Transparent Data Encryption (TDE) v1.30.1

Transparent Data Encryption, or TDE, is a technology used by several database vendors to encrypt data at rest, i.e. database files on disk. TDE does not however encrypt data in use.

TDE is included in EDB Postgres Advanced Server and EDB Postgres Extended Server from version 15, and is supported by the EDB Postgres® AI for CloudNativePG™ Cluster operator.

With TDE activated, both WAL files and files for tables will be encrypted. Data encryption/decryption is entirely transparent to the user, as it is managed by the database without requiring any application changes or updated client drivers.

EDB Postgres® AI for CloudNativePG™ Cluster provides 3 ways to use TDE:

• using a secret containing the passphrase
• using a secret containing a custom passphrase command
• using a pair of secrets containing custom wrap/unwrap commands

Passphrase secret

The basic approach is to store the passphrase in a Kubernetes secret. Such a passphrase will be used to encrypt the EPAS binary key.

Activating TDE on the operator is simple. In the epas section of the manifest, use the tde stanza to enable TDE, and set the Kubernetes secret that will hold the TDE encryption key.

For example:

  […]
  postgresql:
    epas:
      tde:
        enabled: true
        keyLength: 256
        secretKeyRef:
          name: tde-key
          key: key

You can find an example in cluster-example-tde.yaml.

The key stored in the secret will be used as the pass-phrase to invoke openssl to wrap/unwrap the EPAS encryption key.

The optional keyLength field selects the AES key length used to encrypt the database files. The supported values are 128 (the default) and 256. AES-256 requires EPAS or PG Extended 16 or newer. When restoring a physical backup, set keyLength to the value used by the source cluster. The restored data directory keeps the source key: a different value is not applied and is not validated, it only makes the spec misleading.

How to create the secret containing the passphrase

First choose the passphrase. While it is recommended to use a randomly generated passphrase, in this example we will use PostgresRocks as passphrase, and rely on kubectl to generate for us the secret definition:

kubectl create secret generic -o yaml tde-key \
    --from-literal=key=PostgresRocks

This should return something like this:

apiVersion: v1
data:
  key: UG9zdGdyZXNSb2Nrcw==
kind: Secret
metadata:
  creationTimestamp: "YYYY-MM-DDTHH:MM:SSZ"
  name: tde-key
  namespace: default
  resourceVersion: ....
  uid: ....
type: Opaque

Remember to run kubectl apply or remove the -o yaml option to the create command above to actually create the secret in the cluster.

Custom passphrase command

Instead of the secretKeyRef in the cluster manifest snippet above, it is possible to specify a passphraseCommand stored in a secret. The passphrase command can be run to generate a passphrase to be used with openssl.

  […]
  postgresql:
    epas:
      tde:
        enabled: true
        keyLength: 256
        passphraseCommand:
          name: tde-passphrase
          key: command

The passphrase command should write to standard output. For example, we could simply use echo my-passphrase.

The passphrase generated by the command will be used the same way the secretKeyRef was used, i.e. as a passphrase argument for openssl.

Custom wrap/unwrap commands

It is also possible to specify the wrap and unwrap commands, rather than rely on the default invocation of openssl. This can be done by creating secrets containing the custom commands, and declaring those secrets in the tde stanza.

The snippet below shows a cluster with TDE enabled using custom commands.

  […]
  postgresql:
    epas:
      tde:
        enabled: true
        keyLength: 256
        wrapCommand:
          name: tde-wrap-command
          key: command
        unwrapCommand:
          name: tde-unwrap-command
          key: command

The custom commands need to obey the following conventions:

1. The custom wrap command should accept input from standard input, which EPAS will use to feed it the binary key. It should write to a file via an explicit argument (not shell redirections). Moreover, the file argument should be given the string "%p", which is a placeholder EPAS will use to pass the file path of the new, wrapped encryption key file.

2. The custom unwrap command should write to standard output. It should have an explicit file path argument for input (not shell redirections). Again, the file argument should be given the string "%p", which is the placeholder EPAS will fill in with the wrapped encryption key file path.

For example:

• wrap command: openssl enc -aes-128-cbc -pass pass:temp-pass -e -out %p
• unwrap command: openssl enc -aes-128-cbc -pass pass:temp-pass -d -in %p

Example using HashiCorp Vault

The following example shows how to use HashiCorp Vault to store the encryption key and use it to activate TDE. The vault CLI is used to interact with Vault and is included by default in the EDB Postgres Advanced Server (EPAS) image.

First, wherever you have vault running you must enable the Transit secrets engine and create a key:

vault secrets enable transit
vault write -f transit/keys/pg-tde

Then, create a secret containing the custom wrap/unwrap commands. The wrap and unwrap commands will 'wrap' a binary that is in the EPAS image. The binary will interact with the vault API to encrypt/decrypt the EPAS encryption.

The binary needs 5 flags: --file, --host, --secret, --key and --vault-endpoint. The --host flag is in the format of http://vault-host:vault-port and needs to be provided to reach the Vault. The server--secret flag is the name of the Kubernetes secret that contains the vault token and the --key flag is the key in that secret pointing the vault token. The --vault-endpoint flag is the name of the key that was created inside vault; in the example above it is pg-tde.

If running the Vault operator in Kubernetes the root token can be obtained from the following two commands:

kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
cat cluster-keys.json | jq -r ".root_token"

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host http://vault:8200 --secret vault-token --key token --vault-endpoint pg-tde" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host http://vault:8200 --secret vault-token --key token --vault-endpoint pg-tde" \
    --from-literal=token="hvs.whatever"

You can now create a Cluster that is referencing the secrets:

apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
  name: hashicorp-vault-tde
spec:
  instances: 3
  storage:
    size: 1Gi
  postgresql:
    epas:
      tde:
        enabled: true
        wrapCommand:
          name: vault-token
          key: wrap
        unwrapCommand:
          name: vault-token
          key: unwrap

Kubernetes authentication

Instead of storing a long-lived Vault token in a Kubernetes secret, the wrapper can authenticate to Vault using the Kubernetes auth method, logging in with the pod's own ServiceAccount token. This is required if you want to use a non-default auth mount path or a Vault Enterprise namespace, covered below.

Enable the Kubernetes auth method and configure it to trust the cluster's API server:

vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc" \
    kubernetes_ca_cert=@ca.crt \
    token_reviewer_jwt=@token

Create a policy granting access to the transit key, and a role binding it to the ServiceAccount used by the Cluster (by default, the ServiceAccount name matches the Cluster name, unless spec.serviceAccountName is set):

vault policy write tde-policy - <<EOF
path "transit/encrypt/pg-tde" { capabilities = ["update"] }
path "transit/decrypt/pg-tde" { capabilities = ["update"] }
EOF

vault write auth/kubernetes/role/tde-role \
    bound_service_account_names=hashicorp-vault-tde \
    bound_service_account_namespaces=default \
    policies=tde-policy \
    ttl=1h

Then omit --secret and --key from the wrap/unwrap commands entirely. Without them, the binary authenticates using the ServiceAccount token mounted in the pod and the Vault role passed with --role (default tde-role):

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role"

Non-default Kubernetes authentication mount path

By default the wrapper logs in against the kubernetes auth mount. If your Kubernetes auth backend is enabled at a custom path — common when Vault is provisioned via Terraform, which often creates a dedicated mount per cluster or tenant — the default won't match and Vault returns 403 permission denied. Use --kubernetes-mount-path (default kubernetes) to set it. Pass the mount name only (e.g. cluster-abc), not the full auth/cluster-abc/login path.

Enable the Kubernetes auth method and configure it for a custom cluster-abc path:

vault auth enable -path=cluster-abc kubernetes

vault write auth/cluster-abc/config \
    kubernetes_host="https://kubernetes.default.svc" \
    kubernetes_ca_cert=@ca.crt \
    token_reviewer_jwt=@token

Create a policy granting access to the transit key, and a role binding it to the ServiceAccount used by the Cluster (by default, the ServiceAccount name matches the Cluster name, unless spec.serviceAccountName is set):

vault policy write tde-policy - <<EOF
path "transit/encrypt/pg-tde" { capabilities = ["update"] }
path "transit/decrypt/pg-tde" { capabilities = ["update"] }
EOF

vault write auth/cluster-abc/role/tde-role \
    bound_service_account_names=hashicorp-vault-tde \
    bound_service_account_namespaces=default \
    policies=tde-policy

Create the vault-token secret including the --kubernetes-mount-path cluster-abc option:

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role --kubernetes-mount-path cluster-abc" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role --kubernetes-mount-path cluster-abc"

Vault Enterprise namespaces

Use --namespace to target a Vault Enterprise namespace explicitly. It overrides the ambient VAULT_NAMESPACE environment variable, and applies whether the binary authenticates via --secret/--key or via Kubernetes auth.

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role --namespace admin/my-team" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host http://vault:8200 --vault-endpoint pg-tde --role tde-role --namespace admin/my-team"

Enable TLS

To enable TLS, set --enable-tls to true and make sure --host uses the https:// scheme — TLS is only actually negotiated when the address scheme is https, so --enable-tls has no effect against an http:// host. This applies both when authenticating with --secret/--key and with the Kubernetes authentication flow.

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host https://vault:8200 --secret vault-token --key token --vault-endpoint pg-tde --enable-tls true" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host https://vault:8200 --secret vault-token --key token --vault-endpoint pg-tde --enable-tls true" \
    --from-literal=token="hvs.whatever"

The equivalent using Kubernetes authentication instead:

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host https://vault:8200 --vault-endpoint pg-tde --role tde-role --enable-tls true" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host https://vault:8200 --vault-endpoint pg-tde --role tde-role --enable-tls true"

Verifying the Vault server certificate

By default the client does not verify the Vault server's TLS certificate (--verify-ca defaults to false, applying InsecureSkipVerify). Set --verify-ca to true to enable verification. There is no dedicated flag for the CA bundle itself: the binary picks it up from the VAULT_CACERT environment variable, so this is needed whenever Vault's certificate is signed by a CA that isn't already trusted by the container's system trust store — the case for most self-signed or internal-CA Vault deployments.

kubectl create secret generic -o yaml vault-token \
    --from-literal=wrap="/bin/vault wrap --file %p --host https://vault:8200 --vault-endpoint pg-tde --role tde-role --enable-tls true --verify-ca true" \
    --from-literal=unwrap="/bin/vault unwrap --file %p --host https://vault:8200 --vault-endpoint pg-tde --role tde-role --enable-tls true --verify-ca true"

Then, specify the environment variables in a secret or configMap.

kubectl create secret generic -o yaml env-var-secret \
--from-literal=VAULT_CACERT="/projected/certificate/vault-ca.pem" \

Reference the secret in the Cluster spec envFrom section, so that the binary can use the environment variables, as they will be injected in the pods. If one or more environment variables refers to files, rely on the projectedVolumeTemplate to mount custom files, and on secrets or configmaps for their contents. Following the example, the values of the tls-vault-secret key ca is mounted as file into the path /projected/certificate/vault-ca.pem.

kubectl create secret generic -o yaml tls-vault-secret --from-file=ca=vault-ca.crt

apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
  name: hashicorp-vault-tde
spec:
  envFrom:
  - secretRef:
      name: env-var-secret
  projectedVolumeTemplate:
    sources:
      - secret:
          name: tls-vault-secret
          items:
            - key: ca
              path: certificate/vault-ca.pem
  instances: 3
  storage:
    size: 1Gi
  postgresql:
    epas:
      tde:
        enabled: true
        wrapCommand:
          name: vault-token
          key: wrap
        unwrapCommand:
          name: vault-token
          key: unwrap

Environment variables

Besides its CLI flags, /bin/vault also reads the standard set of VAULT_* environment variables supported by HashiCorp's Go client library (VAULT_ADDR, VAULT_TOKEN, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CACERT, VAULT_CLIENT_CERT/VAULT_CLIENT_KEY, and more) — inject them the same way as VAULT_CACERT above, via envFrom and, for file-based values, projectedVolumeTemplate. See Vault's environment variables reference for the full list and description of each.