Skip to main content
POST
/
v1
/
sdk
/
import-pipeline
curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "domain": "MLS",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "steps": [
      {
        "type": "source",
        "name": "Extract MLS API",
        "config": {
          "connType": "API",
          "endpoint": "/v2/property/snapshot",
          "paginated": true,
          "paginationKey": "pagetoken"
        }
      },
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": {
          "vendor": "POSTGRESQL",
          "schemaName": "mls_data",
          "tableName": "mls_properties_with_address"
        }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "domain": "MLS",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}
Single-shot pipeline creation. Accepts a declarative payload describing pipeline steps, schedule, and metadata — returns the created pipeline ID.
This endpoint uses API key authentication. Pass your key via Authorization: Bearer trvh_sdk_... or x-api-key header.

​
Request Body

​
name
string
required
Pipeline name displayed in the UI.
​
description
string
Optional description for the pipeline.
​
domain
string
Domain identifier (e.g., "MLS", "PRODUCT_CATALOG").
​
category
string
Pipeline category (e.g., "Real Estate", "E-Commerce").
​
tags
array
Tags for the pipeline (e.g., ["adp", "mls", "automated"]).
​
steps
array
required
Ordered array of pipeline step definitions. Steps execute sequentially — each step depends on the previous one.
​
schedule
object
Optional schedule for automatic execution.

​
Step Type Configs

​
Source

Extract data from a database, API, file, or S3 bucket.
​
config.connectionId
string
Connection ID for the data source.
​
config.connType
string
Connection type: Database, API, File, S3.
​
config.endpoint
string
API endpoint path (for API sources).
​
config.paginated
boolean
Whether the API supports pagination.
​
config.paginationKey
string
Pagination token key in the API response.
​
config.schemaName
string
Database schema name (for DB sources).
​
config.tableName
string
Database table name (for DB sources).

​
Transform

Field mapping and data transformation rules.
​
config.fieldMappings
array
Array of field mapping objects.
​
config.rules
array
Array of transformation rules.

​
Validate

Data validation rules.
​
config.rules
array

​
Quality

Data quality checks.
​
config.rules
array

​
DB Lookup

Read from another database for reference/enrichment.
​
config.connectionId
string
required
Connection ID for lookup database.
​
config.schemaName
string
Schema name.
​
config.tableName
string
required
Table to look up from.
​
config.lookupKey
string
required
Field from pipeline data to match on.
​
config.joinField
string
required
Field in lookup table to join on.
​
config.selectFields
string
Comma-separated list of fields to pull from lookup table.

​
Stage

Write to a staging/temporary table for intermediate processing.
​
config.schemaName
string
Staging schema name.
​
config.tableName
string
required
Staging table/view name.

​
Load

Write to the final destination database (Postgres for v1).
​
config.connectionId
string
Connection ID for destination. Can be omitted if using the same connection as source.
​
config.vendor
string
default:"POSTGRESQL"
Database vendor.
​
config.schemaName
string
Destination schema name.
​
config.tableName
string
required
Destination table name.

​
Reconcile

Post-load data reconciliation — compares source vs destination.
​
config.mappingSource
string
default:"default"
"default" uses the built-in ADP screen map. "custom" expects a mapping file at execution time.
​
config.compareRowCounts
boolean
default:true
Compare source and destination row counts.
​
config.checkFieldCompleteness
boolean
default:true
Check non-null completeness per field.
​
config.screenMapEnabled
boolean
default:true
Run ADP screen-level validation.

​
Response

​
success
boolean
Whether the import succeeded.
​
data
object
The created pipeline document.
curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
  -H "Authorization: Bearer trvh_sdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ADP - MLS Property Ingestion",
    "domain": "MLS",
    "category": "Real Estate",
    "tags": ["adp", "mls"],
    "steps": [
      {
        "type": "source",
        "name": "Extract MLS API",
        "config": {
          "connType": "API",
          "endpoint": "/v2/property/snapshot",
          "paginated": true,
          "paginationKey": "pagetoken"
        }
      },
      {
        "type": "transform",
        "name": "Map & Coerce Fields",
        "config": {
          "fieldMappings": [
            {"source": "address.line1", "target": "address_line1"},
            {"source": "summary.yearBuilt", "target": "year_built"}
          ],
          "rules": [
            {"field": "year_built", "rule": "to_integer"}
          ]
        }
      },
      {
        "type": "validate",
        "name": "Validate Required Fields",
        "config": {
          "rules": [
            {"field": "attom_id", "rule": "not_null"},
            {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
          ]
        }
      },
      {
        "type": "load",
        "name": "Load to Postgres",
        "config": {
          "vendor": "POSTGRESQL",
          "schemaName": "mls_data",
          "tableName": "mls_properties_with_address"
        }
      },
      {
        "type": "reconcile",
        "name": "ADP Reconciliation",
        "config": {
          "mappingSource": "default",
          "compareRowCounts": true,
          "checkFieldCompleteness": true,
          "screenMapEnabled": true
        }
      }
    ],
    "schedule": {
      "isEnabled": true,
      "frequency": "daily",
      "startTime": "02:00"
    }
  }'

{
  "success": true,
  "message": "Pipeline imported successfully.",
  "data": {
    "pipelineId": "DP-1717000000000",
    "name": "ADP - MLS Property Ingestion",
    "domain": "MLS",
    "status": "draft",
    "steps": [
      {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
      {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
      {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
      {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
      {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
    ]
  }
}