Import Data Pipeline

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

POST

sdk

import-pipeline

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

Single-shot pipeline creation. Accepts a declarative payload describing pipeline steps, schedule, and metadata — returns the created pipeline ID.

This endpoint uses API key authentication. Pass your key via Authorization: Bearer trvh_sdk_... or x-api-key header.

Request Body

name

string

required

Pipeline name displayed in the UI.

description

string

Optional description for the pipeline.

Step Type Configs

Source

Extract data from a database, API, file, or S3 bucket. Inherits connectionId and connection config from the top-level source object. You typically don’t need a source step in the steps array — one is auto-generated from the top-level source definition. Only add explicit source steps when you need multiple sources (e.g., a second API call mid-pipeline for enrichment).

config.connectionId

string

Override connection ID (for additional sources beyond the primary). If omitted, inherits from top-level source.connectionId.

config.endpoint

string

Override API endpoint (for API sources).

config.tableName

string

Table name to extract from (for DB sources).

Transform

Field mapping and data transformation rules.

config.fieldMappings

array

Array of field mapping objects.

Show fieldMapping object

source

string

required

Source field name.

target

string

required

Target field name.

config.rules

array

Array of transformation rules.

Show rule object

field

string

required

Field to transform.

rule

string

required

Rule type: to_integer, to_string, to_decimal, to_date, pad_trim, uppercase, lowercase, trim.

params

object

Rule-specific parameters (e.g., {"length": 5, "padChar": "0"} for pad_trim).

Validate

Data validation rules.

config.rules

array

Show rule object

field

string

required

Field to validate.

rule

string

required

Rule type: not_null, regex, range, min_length, max_length, email.

params

object

Rule-specific parameters (e.g., {"pattern": "^\\d{5}$"} for regex).

Quality

Data quality checks.

config.rules

array

Show rule object

field

string

required

Field to check.

rule

string

required

Rule type: deduplicate, outlier_detection, completeness_check.

params

object

Rule-specific parameters. For deduplicate: {"keyFields": ["id"]}. For outlier_detection: {"method": "iqr", "threshold": 3}.

DB Lookup

Read from another database for reference/enrichment.

config.connectionId

string

required

Connection ID for lookup database.

config.schemaName

string

Schema name.

config.tableName

string

required

Table to look up from.

config.lookupKey

string

required

Field from pipeline data to match on.

config.joinField

string

required

Field in lookup table to join on.

config.selectFields

string

Comma-separated list of fields to pull from lookup table.

Stage

Write to a staging/temporary table for intermediate processing.

config.schemaName

string

Staging schema name.

config.tableName

string

required

Staging table/view name.

Load

Write to the final destination database (Postgres for v1). Inherits connectionId, vendor, and schemaName from the top-level destination object.

config.tableName

string

required

Destination table name.

config.connectionId

string

Override connection ID. If omitted, inherits from top-level destination.connectionId.

config.schemaName

string

Override schema. If omitted, inherits from top-level destination.schemaName.

Reconcile

Post-load data reconciliation — compares source vs destination.

config.mappingSource

string

default:"default"

"default" uses the built-in ADP screen map. "custom" expects a mapping file at execution time.

config.compareRowCounts

boolean

default:true

Compare source and destination row counts.

config.checkFieldCompleteness

boolean

default:true

Check non-null completeness per field.

config.screenMapEnabled

boolean

default:true

Run ADP screen-level validation.

Response

success

boolean

Whether the import succeeded.

data

object

The created pipeline document.

Show pipeline object

pipelineId

string

Pipeline ID (e.g., DP-1717000000000).

name

string

Pipeline name.

steps

array

Array of normalized step objects with generated IDs and status.

status

string

Pipeline status (draft).

curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "source": {
      "connectionId": "D2026A00003",
      "type": "api",
      "config": {
        "endpoint": "/v2/property/snapshot",
        "paginated": true,
        "paginationKey": "pagetoken",
        "rootKey": "property"
      }
    },
    "destination": {
      "connectionId": "D2026A00002",
      "vendor": "POSTGRESQL",
      "schemaName": "mls_data"
    },
    "steps": [
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": { "tableName": "mls_properties_with_address" }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}

{
  "success": false,
  "message": "Pipeline JSON must include a name."
}

{
  "error": "API key required"
}

Import Dashboard Client Onboard

SDK API

BI Administration API

Import Data Pipeline

Request Body

Step Type Configs

Source

Transform

Validate

Quality

DB Lookup

Stage

Load

Reconcile

Response

​Request Body

​Step Type Configs

​Source

​Transform

​Validate

​Quality

​DB Lookup

​Stage

​Load

​Reconcile

​Response

Request Body

Step Type Configs

Source

Transform

Validate

Quality

DB Lookup

Stage

Load

Reconcile

Response