Using PGFS with S3-compatible storage

PGFS provides support for AWS S3 or an on-premises S3-compatible provider using:

• IAM roles: If your Postgres instance is running on cloud infrastructure, use Instance Profiles or IAM Roles. This is the most secure method as it uses temporary, rotating credentials. Attach an IAM policy directly to the underlying virtual machine or container by setting auth.type to iam. PGFS automatically detects the instance metadata and uses these temporary, rotating credentials to sign requests.

• Static credentials: PGFS uses static crendentials method to embed security keys directly within a storage location definition for S3-compatible storage providers.

Syntax

PGFS uses the s3: prefix for S3-compatible storage providers. Use the following structure to define a location for S3-compatible storage providers:

SELECT pgfs.create_storage_location(
    'storage_location_name',
    's3://bucket_name',
    options => '{ }',
    credentials => '{ }'
);

For S3-compatible systems, use the following JSON keys:

• The options argument defines the connection behavior for the storage provider.

  Option	Description
  region	The region of the S3-compatible storage system. If omitted, the client attempts auto-discovery.
  endpoint	The specific endpoint URL for the S3-compatible storage system.
  bucket	Used to explicitly provide the bucket name if it can't be passed in the URL.
  allow_http	Set to true if your endpoint uses plain HTTP instead of HTTPS/TLS.
  skip_signature	Set to true to disable Hash-based Message Authentication Code (HMAC) authentication (this method is used when you're not providing access_key_id/secret_access_key in the credentials argument).

• The credentials argument provides the actual authentication secrets.

  Option	Description
  access_key_id	HMAC credentials (often the username for non-AWS S3 providers).
  secret_access_key	HMAC credentials (often the password for non-AWS S3 providers).
  session_token	A temporary session token that can be used instead of HMAC credentials.

Examples

AWS S3 public bucket

This example uses a public bucket on AWS S3. Public buckets don't require authentication.

SELECT pgfs.create_storage_location('edb_ai_example_images', 's3://public-ai-team',
                                    options => '{"region": "eu-central-1", "skip_signature": "true"}'
       );

AWS S3 private bucket

This example uses a private bucket on AWS S3. Private buckets require authentication. The example uses HMAC credentials.

SELECT pgfs.create_storage_location('internal_ai_project', 's3://my-company-ai-images',
                                    options => '{"region": "eu-central-1"}',
                                    credentials => '{"access_key_id": "secret", "secret_access_key":"secret!"}'
       );

Non-AWS S3 / S3-compatible with HTTPS

This example uses an S3-compatible system like minIO. The endpoint must be provided in this case. You can omit it only when using AWS S3.

SELECT pgfs.create_storage_location('ai_images_local_minio', 's3://my-ai-images',
                                    options => '{"endpoint": "https://minio-api.apps.local"}',
                                    credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}'
       );

Non-AWS S3 / S3-compatible with HTTP

This example uses an S3-compatible system like minIO. The endpoint must be provided in this case. You can omit it only be when using AWS S3.

In this case, the server doesn't use TLS encryption, so the code configures a plain HTTP connection.

SELECT pgfs.create_storage_location('ai_images_local_minio', 
                                    's3://my-ai-images',
                                    options => '{"endpoint": "http://minio-api.apps.local", "allow_http":"true"}',
                                    credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}'
       );

AWS S3 with IAM roles

This example uses an AWS S3 URL to define a new storage location. By setting auth.type to iam, the driver inherits permissions from the host environment eliminating need for access keys.

SELECT pgfs.create_storage_location(
    'production_lake',
    's3://my-analytics-bucket/',
    '{"region": "us-east-1", "auth": {"type": "iam"}}' 
);