Using PGFS with Google cloud storage

PGFS supports Google Cloud Storage (GCS) using:

• Static credentials: The GCS service account keys are used as static credentials in the auth block. For more information, see Static credentials.

• Environment variables: PGFS can be configured to inherit authentication credentials directly from the operating system environment where Postgres is running. This method is particularly effective for managed environments, such as Google Kubernetes Engine (GKE), where credentials can be injected into the container.

  When using this method, PGFS automatically searches for relevant environment keys to authorize the connection, allowing you to simplify your SQL commands by omitting the credentials parameter.

Syntax

PGFS uses the gs:// prefix to identify GCS buckets. Use the following structure to define a location for GCS:

SELECT pgfs.create_storage_location(
  'storage_location_name', 
  'gs://bucket_name'
   credentials => '{}');

The credentials argument is optional. If you choose to pass credentials manually using JSON instead of environment variables, the following options are supported for GCS:

See the Google Cloud documentation for more information on how to manage service account keys.

You can also set up these options via the equivalent environment variables to facilitate authentication in managed environments such as Google Kubernetes Engine.

Examples

These examples shows how to create a storage location for GCS:

Static credentials: Private GCS bucket

Create a storage location for a private GCS bucket using static credentials:

SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images',
                                    credentials => '{"google_service_account_key_file": "/var/run/gcs.json"}'
       );

Environment variables: GKE

PGFS can inherit credentials directly from the operating system environment where Postgres is running. For example:

• Set the variable at the OS level:

  → WrapCopy

  EXPORT GOOGLE_APPLICATION_CREDENTIALS=/var/run/gcs.json

1. Create the storage location and omit the credentials parameter. PGFS will automatically check for the relevant environment keys to authorize the connection:

   → WrapCopy

   SELECT pgfs.create_storage_location(
      'edb_ai_example_images', 
      'gs://my-company-ai-images');