> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tarvah.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Import Data Pipeline

> Create a complete data pipeline from a JSON definition (API key auth)

Single-shot pipeline creation. Accepts a declarative payload describing pipeline steps, schedule, and metadata — returns the created pipeline ID.

<Note>
  This endpoint uses **API key authentication**. Pass your key via `Authorization: Bearer trvh_sdk_...` or `x-api-key` header.
</Note>

## Request Body

<ParamField body="name" type="string" required>
  Pipeline name displayed in the UI.
</ParamField>

<ParamField body="description" type="string">
  Optional description for the pipeline.
</ParamField>

<ParamField body="category" type="string">
  Pipeline category (e.g., `"Real Estate"`, `"E-Commerce"`).
</ParamField>

<ParamField body="tags" type="array">
  Tags for the pipeline (e.g., `["adp", "mls", "automated"]`).
</ParamField>

<ParamField body="source" type="object" required>
  Where data comes FROM. This connection is inherited by all `source` type steps.

  <Expandable title="source properties">
    <ParamField body="connectionId" type="string" required>
      Connection ID (e.g., `D2026A00003`). Same ID format as BDM/Dashboard import `dbId`.
    </ParamField>

    <ParamField body="type" type="string" required>
      Connection type: `api`, `database`, `file`, `s3`.
    </ParamField>

    <ParamField body="config" type="object">
      Source-level defaults inherited by `source` steps. For API sources: `endpoint`, `method`, `paginated`, `paginationKey`, `rootKey`. For DB sources: `schemaName`, `tableName`.

      <Expandable title="config properties (API example)">
        <ParamField body="endpoint" type="string">API endpoint path.</ParamField>
        <ParamField body="method" type="string" default="GET">HTTP method.</ParamField>
        <ParamField body="paginated" type="boolean">Whether the API supports pagination.</ParamField>
        <ParamField body="paginationKey" type="string">Pagination token key in response.</ParamField>
        <ParamField body="rootKey" type="string">JSON root key containing the data array.</ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="destination" type="object" required>
  Where data goes TO. This connection is inherited by `load`, `stage`, and `reconcile` steps.

  <Expandable title="destination properties">
    <ParamField body="connectionId" type="string" required>
      Connection ID (e.g., `D2026A00002`).
    </ParamField>

    <ParamField body="type" type="string" default="database">
      Connection type. Currently only `database` is supported for v1.
    </ParamField>

    <ParamField body="vendor" type="string" default="POSTGRESQL">
      Database vendor.
    </ParamField>

    <ParamField body="schemaName" type="string">
      Default schema name. Inherited by `load` and `stage` steps.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="steps" type="array" required>
  Ordered array of pipeline step definitions. Steps execute sequentially — each step depends on the previous one.

  <Expandable title="step object">
    <ParamField body="type" type="string" required>
      Step type. One of: `source`, `transform`, `validate`, `quality`, `db_lookup`, `stage`, `load`, `reconcile`.
    </ParamField>

    <ParamField body="name" type="string">
      Display name for the step. Defaults to the step type label if omitted.
    </ParamField>

    <ParamField body="config" type="object" required>
      Type-specific configuration. See **Step Type Configs** below.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="schedule" type="object">
  Optional schedule for automatic execution.

  <Expandable title="schedule properties">
    <ParamField body="isEnabled" type="boolean" default={false}>
      Whether the schedule is active.
    </ParamField>

    <ParamField body="frequency" type="string">
      Execution frequency: `once`, `hourly`, `daily`, `weekly`, `monthly`, `cron`.
    </ParamField>

    <ParamField body="cronExpression" type="string">
      Cron expression (when `frequency` is `cron`).
    </ParamField>

    <ParamField body="startDate" type="string">
      Schedule start date (ISO format).
    </ParamField>

    <ParamField body="startTime" type="string">
      Daily start time (e.g., `"02:00"`).
    </ParamField>
  </Expandable>
</ParamField>

## Step Type Configs

### Source

Extract data from a database, API, file, or S3 bucket. Inherits `connectionId` and connection config from the top-level `source` object. You typically don't need a `source` step in the `steps` array — one is auto-generated from the top-level `source` definition. Only add explicit `source` steps when you need **multiple** sources (e.g., a second API call mid-pipeline for enrichment).

<ParamField body="config.connectionId" type="string">
  Override connection ID (for additional sources beyond the primary). If omitted, inherits from top-level `source.connectionId`.
</ParamField>

<ParamField body="config.endpoint" type="string">
  Override API endpoint (for API sources).
</ParamField>

<ParamField body="config.tableName" type="string">
  Table name to extract from (for DB sources).
</ParamField>

### Transform

Field mapping and data transformation rules.

<ParamField body="config.fieldMappings" type="array">
  Array of field mapping objects.

  <Expandable title="fieldMapping object">
    <ParamField body="source" type="string" required>Source field name.</ParamField>
    <ParamField body="target" type="string" required>Target field name.</ParamField>
  </Expandable>
</ParamField>

<ParamField body="config.rules" type="array">
  Array of transformation rules.

  <Expandable title="rule object">
    <ParamField body="field" type="string" required>Field to transform.</ParamField>

    <ParamField body="rule" type="string" required>
      Rule type: `to_integer`, `to_string`, `to_decimal`, `to_date`, `pad_trim`, `uppercase`, `lowercase`, `trim`.
    </ParamField>

    <ParamField body="params" type="object">Rule-specific parameters (e.g., `{"length": 5, "padChar": "0"}` for `pad_trim`).</ParamField>
  </Expandable>
</ParamField>

### Validate

Data validation rules.

<ParamField body="config.rules" type="array">
  <Expandable title="rule object">
    <ParamField body="field" type="string" required>Field to validate.</ParamField>

    <ParamField body="rule" type="string" required>
      Rule type: `not_null`, `regex`, `range`, `min_length`, `max_length`, `email`.
    </ParamField>

    <ParamField body="params" type="object">Rule-specific parameters (e.g., `{"pattern": "^\\d{5}$"}` for `regex`).</ParamField>
  </Expandable>
</ParamField>

### Quality

Data quality checks.

<ParamField body="config.rules" type="array">
  <Expandable title="rule object">
    <ParamField body="field" type="string" required>Field to check.</ParamField>

    <ParamField body="rule" type="string" required>
      Rule type: `deduplicate`, `outlier_detection`, `completeness_check`.
    </ParamField>

    <ParamField body="params" type="object">
      Rule-specific parameters. For `deduplicate`: `{"keyFields": ["id"]}`. For `outlier_detection`: `{"method": "iqr", "threshold": 3}`.
    </ParamField>
  </Expandable>
</ParamField>

### DB Lookup

Read from another database for reference/enrichment.

<ParamField body="config.connectionId" type="string" required>Connection ID for lookup database.</ParamField>
<ParamField body="config.schemaName" type="string">Schema name.</ParamField>
<ParamField body="config.tableName" type="string" required>Table to look up from.</ParamField>
<ParamField body="config.lookupKey" type="string" required>Field from pipeline data to match on.</ParamField>
<ParamField body="config.joinField" type="string" required>Field in lookup table to join on.</ParamField>
<ParamField body="config.selectFields" type="string">Comma-separated list of fields to pull from lookup table.</ParamField>

### Stage

Write to a staging/temporary table for intermediate processing.

<ParamField body="config.schemaName" type="string">Staging schema name.</ParamField>
<ParamField body="config.tableName" type="string" required>Staging table/view name.</ParamField>

### Load

Write to the final destination database (Postgres for v1). Inherits `connectionId`, `vendor`, and `schemaName` from the top-level `destination` object.

<ParamField body="config.tableName" type="string" required>Destination table name.</ParamField>
<ParamField body="config.connectionId" type="string">Override connection ID. If omitted, inherits from top-level `destination.connectionId`.</ParamField>
<ParamField body="config.schemaName" type="string">Override schema. If omitted, inherits from top-level `destination.schemaName`.</ParamField>

### Reconcile

Post-load data reconciliation — compares source vs destination.

<ParamField body="config.mappingSource" type="string" default="default">
  `"default"` uses the built-in ADP screen map. `"custom"` expects a mapping file at execution time.
</ParamField>

<ParamField body="config.compareRowCounts" type="boolean" default={true}>Compare source and destination row counts.</ParamField>
<ParamField body="config.checkFieldCompleteness" type="boolean" default={true}>Check non-null completeness per field.</ParamField>
<ParamField body="config.screenMapEnabled" type="boolean" default={true}>Run ADP screen-level validation.</ParamField>

## Response

<ResponseField name="success" type="boolean">
  Whether the import succeeded.
</ResponseField>

<ResponseField name="data" type="object">
  The created pipeline document.

  <Expandable title="pipeline object">
    <ResponseField name="pipelineId" type="string">
      Pipeline ID (e.g., `DP-1717000000000`).
    </ResponseField>

    <ResponseField name="name" type="string">Pipeline name.</ResponseField>
    <ResponseField name="steps" type="array">Array of normalized step objects with generated IDs and status.</ResponseField>
    <ResponseField name="status" type="string">Pipeline status (`draft`).</ResponseField>
  </Expandable>
</ResponseField>

<RequestExample>
  ```bash ADP MLS Pipeline theme={null}
  curl -X POST https://<your-instance>/v1/sdk/import-pipeline \
    -H "Authorization: Bearer trvh_sdk_your_key_here" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "ADP - MLS Property Ingestion",
      "category": "Real Estate",
      "tags": ["adp", "mls"],
      "source": {
        "connectionId": "D2026A00003",
        "type": "api",
        "config": {
          "endpoint": "/v2/property/snapshot",
          "paginated": true,
          "paginationKey": "pagetoken",
          "rootKey": "property"
        }
      },
      "destination": {
        "connectionId": "D2026A00002",
        "vendor": "POSTGRESQL",
        "schemaName": "mls_data"
      },
      "steps": [
        {
          "type": "transform",
          "name": "Map & Coerce Fields",
          "config": {
            "fieldMappings": [
              {"source": "address.line1", "target": "address_line1"},
              {"source": "summary.yearBuilt", "target": "year_built"}
            ],
            "rules": [
              {"field": "year_built", "rule": "to_integer"}
            ]
          }
        },
        {
          "type": "validate",
          "name": "Validate Required Fields",
          "config": {
            "rules": [
              {"field": "attom_id", "rule": "not_null"},
              {"field": "zip", "rule": "regex", "params": {"pattern": "^\\d{5}$"}}
            ]
          }
        },
        {
          "type": "load",
          "name": "Load to Postgres",
          "config": { "tableName": "mls_properties_with_address" }
        },
        {
          "type": "reconcile",
          "name": "ADP Reconciliation",
          "config": {
            "mappingSource": "default",
            "compareRowCounts": true,
            "checkFieldCompleteness": true,
            "screenMapEnabled": true
          }
        }
      ],
      "schedule": {
        "isEnabled": true,
        "frequency": "daily",
        "startTime": "02:00"
      }
    }'
  ```
</RequestExample>

<ResponseExample>
  ```json 201 Created theme={null}
  {
    "success": true,
    "message": "Pipeline imported successfully.",
    "data": {
      "pipelineId": "DP-1717000000000",
      "name": "ADP - MLS Property Ingestion",
      "status": "draft",
      "steps": [
        {"id": "step-1717000000000-0", "type": "source", "name": "Extract MLS API", "status": "configured"},
        {"id": "step-1717000000000-1", "type": "transform", "name": "Map & Coerce Fields", "status": "configured"},
        {"id": "step-1717000000000-2", "type": "validate", "name": "Validate Required Fields", "status": "configured"},
        {"id": "step-1717000000000-3", "type": "load", "name": "Load to Postgres", "status": "configured"},
        {"id": "step-1717000000000-4", "type": "reconcile", "name": "ADP Reconciliation", "status": "configured"}
      ]
    }
  }
  ```

  ```json 400 Bad Request theme={null}
  {
    "success": false,
    "message": "Pipeline JSON must include a name."
  }
  ```

  ```json 401 Unauthorized theme={null}
  {
    "error": "API key required"
  }
  ```
</ResponseExample>