Pipelines API reference v7

This page covers the full public API for AI pipelines: the types and enums used across the API, the views for inspecting pipeline state, the core CRUD and execution functions, and the configuration helper functions for pipeline steps and vector indexes. For model configuration helpers, see the Models reference.

Types

aidb.PipelineAutoProcessingMode

Controls how a pipeline automatically processes new or changed data.

CREATE TYPE PipelineAutoProcessingMode AS ENUM (
    'Live',
    'Background',
    'Disabled'
);

aidb.PipelineDataFormat

Specifies the format of source data the pipeline processes.

CREATE TYPE PipelineDataFormat AS ENUM (
    'Text',
    'Image',
    'Pdf'
);

aidb.PipelineSourceType

Indicates the type of data source a pipeline reads from.

CREATE TYPE PipelineSourceType AS ENUM (
    'Table',
    'Volume',
    'Empty'
);

aidb.PipelineDestinationType

Indicates the type of destination a pipeline writes to.

CREATE TYPE PipelineDestinationType AS ENUM (
    'Table',
    'Volume',
    'Empty'
);

aidb.PipelineStepOperation

Defines the operation performed by a pipeline step.

CREATE TYPE PipelineStepOperation AS ENUM (
    'ChunkText',
    'SummarizeText',
    'ParseHtml',
    'ParsePdf',
    'PerformOcr',
    'KnowledgeBase',
    'PdfToImage',
    'SemanticKB'
);

aidb.PipelineStatus

Represents the current processing state of a pipeline.

CREATE TYPE PipelineStatus AS ENUM (
Stale         
Processing    
UpToDate      
NoResults     
Failed        
Unknown       
PartialErrors 
BlockingErrors
);

aidb.DistanceOperator

Specifies the distance metric used for vector similarity search.

CREATE TYPE DistanceOperator AS ENUM (
    'L2',
    'InnerProduct',
    'Cosine',
    'L1',
    'Hamming',
    'Jaccard'
);

Domains

aidb.pipeline_name_50

A TEXT domain enforcing that pipeline names are no longer than 50 characters.

aidb.background_sync_interval

An INTERVAL domain enforcing that background sync intervals are between 1 second and 2 days (inclusive).

Views

aidb.pipelines

Also accessible as aidb.pipes. Lists all registered pipelines and their configuration, including source, destination, processing mode, and step definitions.

Example

SELECT name, source, destination, auto_processing FROM aidb.pipelines;

aidb.pipeline_metrics

Also accessible as aidb.pipem. Shows current processing statistics for each pipeline.

Example

SELECT * FROM aidb.pipeline_metrics;

       pipeline       | auto processing | table: unprocessed rows | volume: scans completed | count(source records) | count(destination records) |  Status  | count(record errors) | count(blocking errors)
----------------------+-----------------+-------------------------+-------------------------+-----------------------+----------------------------+----------+----------------------+------------------------
 pipeline__7471a      | Background      |                       0 |                         |                     5 |                          5 | UpToDate |                    0 |                      0
 pipeline__7471b      | Background      |                       0 |                         |                     5 |                          5 | UpToDate |                    0 |                      0
 animal_facts_kb      | Disabled        |                       0 |                         |                    99 |                        372 | UpToDate |                    0 |                      0
 animal_facts_kb_bert | Disabled        |                       0 |                         |                    99 |                        372 | UpToDate |                    0 |                      0
 mpkb_pipe_int        | Disabled        |                       0 |                         |                     2 |                          4 | UpToDate |                    0 |                      0
 mpkb_pipe_text       | Disabled        |                       0 |                         |                     2 |                          4 | UpToDate |                    0 |                      0
(6 rows)

Functions

aidb.create_pipeline

Creates a new pipeline with a source, up to 10 sequential processing steps, and an optional destination.

Parameters

Returns

Example

-- Single-step pipeline: chunk text from a table into a destination table
SELECT aidb.create_pipeline(
    name                => 'my_chunker',
    source              => 'source_docs',
    source_key_column   => 'id',
    source_data_column  => 'body',
    destination         => 'chunked_docs',
    step_1              => 'ChunkText',
    step_1_options      => aidb.chunk_text_config(200, 250, 25),
    auto_processing     => 'Live'
);

-- Multi-step pipeline: parse PDF, then embed into a knowledge base
SELECT aidb.create_pipeline(
    name                => 'pdf_to_kb',
    source              => 'pdf_volume',
    destination         => 'my_kb',
    step_1              => 'ParsePdf',
    step_1_options      => aidb.pdf_parse_config(),
    step_2              => 'KnowledgeBase',
    step_2_options      => aidb.knowledge_base_config('my_model', 'Text'),
    auto_processing     => 'Background',
    background_sync_interval => '60 seconds'
);

aidb.update_pipeline

Updates the auto-processing settings for an existing pipeline.

Parameters

Example

SELECT aidb.update_pipeline('my_chunker', auto_processing => 'Background', background_sync_interval => '5 minutes');

aidb.delete_pipeline

Deletes a pipeline and its configuration. Doesn't delete the source or destination tables.

Parameters

Example

SELECT aidb.delete_pipeline('my_chunker');

aidb.run_pipeline

Manually triggers a pipeline to execute immediately, regardless of its auto_processing mode.

Parameters

Example

SELECT aidb.run_pipeline('my_chunker');

Pipeline step config helpers

These functions return a JSONB configuration object for use in step_N_options parameters of aidb.create_pipeline.

aidb.chunk_text_config

Configures a ChunkText step to split text into smaller segments.

Parameters

Example

-- Chunk into ~200 character segments, max 250, with 25-character overlap
SELECT aidb.chunk_text_config(200, 250, 25, 'chars');

aidb.summarize_text_config

Configures a SummarizeText step to summarize text using a language model.

Parameters

Example

SELECT aidb.summarize_text_config(
    'my_llm',
    chunk_config => aidb.chunk_text_config(100, 100, 10, 'words'),
    prompt       => 'Summarize concisely',
    strategy     => 'reduce',
    reduction_factor => 4
);

aidb.ocr_config

Configures a PerformOcr step to extract text from images using a model.

Parameters

Example

SELECT aidb.ocr_config('my_ocr_model');

aidb.html_parse_config

Configures a ParseHtml step to extract text from HTML content.

Parameters

Example

SELECT aidb.html_parse_config();

aidb.pdf_parse_config

Configures a ParsePdf step to extract content from PDF documents.

Parameters

Example

SELECT aidb.pdf_parse_config(allow_partial_parsing => true);

aidb.knowledge_base_config

Configures a KnowledgeBase step to compute and store embeddings.

Parameters

Example

SELECT aidb.knowledge_base_config(
    'my_embedding_model',
    'Text',
    distance_operator => 'Cosine',
    vector_index      => aidb.vector_index_hnsw_config(m => 16, ef_construction => 64)
);

aidb.knowledge_base_config_from_kb

Configures a KnowledgeBase step to attach a pipeline to an existing knowledge base rather than creating a new one.

Parameters

Example

SELECT aidb.knowledge_base_config_from_kb('Text');

aidb.inference_config

Builds an inference configuration object for use with language model steps such as SummarizeText. All parameters are optional.

Parameters

Example

SELECT aidb.inference_config(
    system_prompt => 'You are a technical summarizer.',
    temperature   => 0.3,
    max_tokens    => 512
);

Vector index config helpers

These functions return a JSONB configuration for the vector_index parameter of aidb.knowledge_base_config.

aidb.vector_index_hnsw_config

Configures an HNSW index (pgvector).

Parameters

HNSW supports a maximum of 2000 dimensions. For higher-dimensional vectors, use `aidb.vector_index_disabled_config()`.

The following table shows how each distance_operator value maps to a pgvector ops class:

Example

SELECT aidb.vector_index_hnsw_config(m => 16, ef_construction => 64);

aidb.vector_index_ivfflat_config

Configures a pgvector IVFFlat index.

Parameters

Example

SELECT aidb.vector_index_ivfflat_config(lists => 100);

aidb.vector_index_chord_hnsw_config

Configures a VectorChord HNSW index.

Parameters

Example

SELECT aidb.vector_index_chord_hnsw_config(m => 16, ef_construction => 64);

aidb.vector_index_chord_vchordq_config

Configures a VectorChord Vchordq index.

Parameters

Example

SELECT aidb.vector_index_chord_vchordq_config(lists => '100', spherical_centroids => true);

aidb.vector_index_hsphere_optimized_config

Configures an HSphere Optimized index.

Parameters

Example

SELECT aidb.vector_index_hsphere_optimized_config(clusters => 256, precision_val => 0.95);

aidb.vector_index_disabled_config

Disables automatic vector index creation. Use this when your embedding dimensions exceed 2000 or when you want to manage indexes manually.

Example

SELECT aidb.vector_index_disabled_config();

Model config helpers

Model config helpers have moved to the Models reference page.