Import Data Pipeline

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

POST

sdk

import-pipeline

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

Single-shot pipeline creation. Accepts a declarative payload describing pipeline steps, schedule, and metadata — returns the created pipeline ID.

This endpoint uses API key authentication. Pass your key via Authorization: Bearer trvh_sdk_... or x-api-key header.

Request Body

string

required

Pipeline name displayed in the UI.

string

Optional description for the pipeline.

string

Pipeline category (e.g., "Real Estate", "E-Commerce").

array

Tags for the pipeline (e.g., ["adp", "mls", "automated"]).

object

required

Where data comes FROM. This connection is inherited by all source type steps.

Show source properties

string

required

Connection ID (e.g., D2026A00003). Same ID format as BDM/Dashboard import dbId.

string

required

Connection type: api, database, file, s3.

object

Source-level defaults inherited by source steps. For API sources: endpoint, method, paginated, paginationKey, rootKey. For DB sources: schemaName, tableName.

Show config properties (API example)

string

API endpoint path.

string

default:"GET"

HTTP method.

boolean

Whether the API supports pagination.

string

Pagination token key in response.

string

JSON root key containing the data array.

object

required

Where data goes TO. This connection is inherited by load, stage, and reconcile steps.

Show destination properties

string

required

Connection ID (e.g., D2026A00002).

string

default:"database"

Connection type. Currently only database is supported for v1.

string

default:"POSTGRESQL"

Database vendor.

string

Default schema name. Inherited by load and stage steps.

array

required

Ordered array of pipeline step definitions. Steps execute sequentially — each step depends on the previous one.

Show step object

string

required

Step type. One of: source, transform, validate, quality, db_lookup, stage, load, reconcile.

string

Display name for the step. Defaults to the step type label if omitted.

object

required

Type-specific configuration. See Step Type Configs below.

object

Optional schedule for automatic execution.

Show schedule properties

boolean

default:false

Whether the schedule is active.

string

Execution frequency: once, hourly, daily, weekly, monthly, cron.

string

Cron expression (when frequency is cron).

string

Schedule start date (ISO format).

string

Daily start time (e.g., "02:00").

Step Type Configs

Source

Extract data from a database, API, file, or S3 bucket. Inherits connectionId and connection config from the top-level source object. You typically don’t need a source step in the steps array — one is auto-generated from the top-level source definition. Only add explicit source steps when you need multiple sources (e.g., a second API call mid-pipeline for enrichment).

string

Override connection ID (for additional sources beyond the primary). If omitted, inherits from top-level source.connectionId.

string

Override API endpoint (for API sources).

string

Table name to extract from (for DB sources).

Transform

Field mapping and data transformation rules.

array

Array of field mapping objects.

Show fieldMapping object

string

required

Source field name.

string

required

Target field name.

array

Array of transformation rules.

Show rule object

string

required

Field to transform.

string

required

Rule type: to_integer, to_string, to_decimal, to_date, pad_trim, uppercase, lowercase, trim.

object

Rule-specific parameters (e.g., {"length": 5, "padChar": "0"} for pad_trim).

Validate

Data validation rules.

array

Show rule object

string

required

Field to validate.

string

required

Rule type: not_null, regex, range, min_length, max_length, email.

object

Rule-specific parameters (e.g., {"pattern": "^\\d{5}$"} for regex).

Quality

Data quality checks.

array

Show rule object

string

required

Field to check.

string

required

Rule type: deduplicate, outlier_detection, completeness_check.

object

Rule-specific parameters. For deduplicate: {"keyFields": ["id"]}. For outlier_detection: {"method": "iqr", "threshold": 3}.

DB Lookup

Read from another database for reference/enrichment.

string

required

Connection ID for lookup database.

string

Schema name.

string

required

Table to look up from.

string

required

Field from pipeline data to match on.

string

required

Field in lookup table to join on.

string

Comma-separated list of fields to pull from lookup table.

Stage

Write to a staging/temporary table for intermediate processing.

string

Staging schema name.

string

required

Staging table/view name.

Load

Write to the final destination database (Postgres for v1). Inherits connectionId, vendor, and schemaName from the top-level destination object.

string

required

Destination table name.

string

Override connection ID. If omitted, inherits from top-level destination.connectionId.

string

Override schema. If omitted, inherits from top-level destination.schemaName.

Reconcile

Post-load data reconciliation — compares source vs destination.

string

default:"default"

"default" uses the built-in ADP screen map. "custom" expects a mapping file at execution time.

boolean

default:true

Compare source and destination row counts.

boolean

default:true

Check non-null completeness per field.

boolean

default:true

Run ADP screen-level validation.

Response

boolean

Whether the import succeeded.

object

The created pipeline document.

Show pipeline object

string

Pipeline ID (e.g., DP-1717000000000).

string

Pipeline name.

array

Array of normalized step objects with generated IDs and status.

string

Pipeline status (draft).

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

SDK API

BI Administration API

Import Data Pipeline

Request Body

Step Type Configs

Source

Transform

Validate

Quality

DB Lookup

Stage

Load

Reconcile

Response

​Request Body

​Step Type Configs

​Source

​Transform

​Validate

​Quality

​DB Lookup

​Stage

​Load

​Reconcile

​Response

Request Body

Step Type Configs

Source

Transform

Validate

Quality

DB Lookup

Stage

Load

Reconcile

Response