Bulk Ingestion API

Upload and process large files containing POP events. This is ideal for backfilling historical data or processing daily/weekly exports.

Workflow

The bulk ingestion process has four steps:

Request Upload URL - Get a secure URL for uploading your file
Upload File - Upload your file (NDJSON, JSON, or CSV) to the provided URL
Start Bulk Ingest - Tell us to begin processing your uploaded file
Check Job Status - Monitor the processing status

Request Upload URL

Get a secure URL where you can upload your file.

Endpoint

http

POST /coatro/v1/pop/dynamic/bulk/upload-url

Description

Request a secure upload URL for your file. After receiving the URL, upload your file using a PUT request, then use the returned fileId to start the ingestion process.

Request Body

json

{
  "fileName": "pop_2025-12-09.jsonl",
  "format": "ndjson",
  "expiresInSeconds": 3600
}

Request Schema

Field	Type	Required	Description
`fileName`	`string`	Yes	Name of the file you want to upload
`format`	`string`	Yes	File format. Must be one of: `ndjson`, `json`, `csv`
`expiresInSeconds`	`integer`	No	How long the upload URL should remain valid (maximum 86400 = 1 day). Default: 3600 (1 hour)

Example Request

bash

curl -X POST "http://localhost:8080/coatro/v1/pop/dynamic/bulk/upload-url" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "pop_2025-12-09.jsonl",
    "format": "ndjson",
    "expiresInSeconds": 3600
  }'

Response

200 OK

Upload URL created successfully.

json

{
  "uploadUrl": "https://storage.googleapis.com/bucket/object?X-Goog-Signature=...",
  "fileId": "file-dynamic-20251209-48393",
  "contentType": "application/x-ndjson",
  "uploadInstructions": "Use PUT method with Content-Type header. Then call POST /ingest with fileId."
}

Response Schema

Field	Type	Description
`uploadUrl`	`string`	Secure URL where you should upload your file
`fileId`	`string`	File identifier - use this when starting the ingest job
`contentType`	`string`	Content-Type header to use when uploading the file
`uploadInstructions`	`string`	Instructions for uploading the file

Error Responses

400 Bad Request - Invalid request data
401 Unauthorized - Invalid or missing API key
415 Unsupported Media Type - Must be application/json
500 Internal Server Error - Server error

Upload Your File

After receiving the upload URL, you must upload your file before starting the ingestion process:

Use the uploadUrl from the response
Make a PUT request to the uploadUrl
Include the Content-Type header with the value from the contentType field in the response
Upload your file as the request body

Example Upload

bash

curl -X PUT "https://storage.googleapis.com/bucket/object?X-Goog-Signature=..." \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @pop_2025-12-09.jsonl

WARNING

You must successfully upload your file to the uploadUrl before calling the Start Bulk Ingest endpoint. The upload URL expires after the time specified in expiresInSeconds (default: 1 hour).

Start Bulk Ingest

After uploading your file, call this endpoint to begin processing.

Endpoint

http

POST /coatro/v1/pop/dynamic/bulk/ingest

Description

Start processing your uploaded file. This returns a jobId that you can use to check the processing status.

Request Body

json

{
  "fileId": "file-dynamic-20251209-48393",
  "schemaVersion": "1.0",
  "metadata": {
    "source": "daily_export",
    "timezone": "UTC"
  }
}

Request Schema

Field	Type	Required	Description
`fileId`	`string`	Yes	File ID returned from the upload-url endpoint
`schemaVersion`	`string`	Yes	Version of the event schema used in your file (e.g., "1.0", "2.1")
`metadata`	`object`	No	Optional metadata for the job (any key-value pairs)

Example Request

bash

curl -X POST "http://localhost:8080/coatro/v1/pop/dynamic/bulk/ingest" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "fileId": "file-dynamic-20251209-48393",
    "schemaVersion": "1.0",
    "metadata": {
      "source": "daily_export",
      "timezone": "UTC"
    }
  }'

Response

202 Accepted

Your ingestion job has been queued and will begin processing.

json

{
  "jobId": "job-dynamic-pop-20251209-1234",
  "message": "Job queued"
}

Response Schema

Field	Type	Description
`jobId`	`string`	Job identifier - use this to check job status
`message`	`string`	Optional message about the job

Error Responses

400 Bad Request - Invalid request (e.g., file not found)
401 Unauthorized - Invalid or missing API key
500 Internal Server Error - Server error

Get Job Status

Check the status of your bulk ingestion job.

Endpoint

http

GET /coatro/v1/pop/dynamic/bulk/jobs/{jobId}

Description

Get the current status of your bulk ingestion job, including processing progress and any errors.

Path Parameters

Parameter	Type	Required	Description
`jobId`	`string`	Yes	Job identifier returned from the ingest endpoint

Example Request

bash

curl -X GET "http://localhost:8080/coatro/v1/pop/dynamic/bulk/jobs/job-dynamic-pop-20251209-1234" \
  -H "X-API-Key: your-api-key"

Response

200 OK

Job status retrieved successfully.

json

{
  "jobId": "job-dynamic-pop-20251209-1234",
  "status": "processing",
  "totalRows": 10000,
  "processedRows": 5000,
  "errorRows": 5,
  "errors": [
    {
      "row": 123,
      "error": "Invalid schema"
    }
  ],
  "startedAt": "2025-01-27T10:00:00Z",
  "completedAt": null
}

Response Schema

Field	Type	Description
`jobId`	`string`	Job identifier
`status`	`string`	Job status. Possible values: `queued`, `processing`, `completed`, `failed`, `partially_completed`
`totalRows`	`integer`	Total rows in the file (when known)
`processedRows`	`integer`	Rows processed successfully
`errorRows`	`integer`	Rows that failed
`errors`	`array`	Per-row errors when available
`startedAt`	`string`	When processing started (ISO 8601 date-time)
`completedAt`	`string`	When processing completed (ISO 8601 date-time, null if not completed)

Job Status Values

Status	Description
`queued`	Job is waiting to be processed
`processing`	Job is currently being processed
`completed`	Job completed successfully
`failed`	Job failed completely
`partially_completed`	Job completed but some rows failed

Error Responses

401 Unauthorized - Invalid or missing API key
404 Not Found - Job not found
500 Internal Server Error - Server error

Complete Example Workflow

Here's a complete example of the bulk ingestion workflow:

Step 1: Request Upload URL

bash

curl -X POST "http://localhost:8080/coatro/v1/pop/dynamic/bulk/upload-url" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "pop_events_2025-01-27.jsonl",
    "format": "ndjson"
  }'

Response:

json

{
  "uploadUrl": "https://storage.googleapis.com/...",
  "fileId": "file-dynamic-20250127-12345",
  "contentType": "application/x-ndjson",
  "uploadInstructions": "Use PUT method with Content-Type header..."
}

Step 2: Upload File

bash

curl -X PUT "https://storage.googleapis.com/..." \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @pop_events_2025-01-27.jsonl

Step 3: Start Ingestion

bash

curl -X POST "http://localhost:8080/coatro/v1/pop/dynamic/bulk/ingest" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "fileId": "file-dynamic-20250127-12345",
    "schemaVersion": "1.0"
  }'

Response:

json

{
  "jobId": "job-dynamic-pop-20250127-67890",
  "message": "Job queued"
}

Step 4: Check Status

bash

curl -X GET "http://localhost:8080/coatro/v1/pop/dynamic/bulk/jobs/job-dynamic-pop-20250127-67890" \
  -H "X-API-Key: your-api-key"

Response:

json

{
  "jobId": "job-dynamic-pop-20250127-67890",
  "status": "completed",
  "totalRows": 1000,
  "processedRows": 1000,
  "errorRows": 0,
  "startedAt": "2025-01-27T10:00:00Z",
  "completedAt": "2025-01-27T10:05:00Z"
}

Bulk Ingestion API ​

Workflow ​

Request Upload URL ​

Endpoint ​

Description ​

Request Body ​

Request Schema ​

Example Request ​

Response ​

200 OK ​

Response Schema ​

Error Responses ​

Upload Your File ​

Example Upload ​

Start Bulk Ingest ​

Endpoint ​

Description ​

Request Body ​

Request Schema ​

Example Request ​

Response ​

202 Accepted ​

Response Schema ​

Error Responses ​

Get Job Status ​

Endpoint ​

Description ​

Path Parameters ​

Example Request ​

Response ​

200 OK ​

Response Schema ​

Job Status Values ​

Error Responses ​

Complete Example Workflow ​

Step 1: Request Upload URL ​

Step 2: Upload File ​

Step 3: Start Ingestion ​

Step 4: Check Status ​

Bulk Ingestion API

Workflow

Request Upload URL

Endpoint

Description

Request Body

Request Schema

Example Request

Response

200 OK

Response Schema

Error Responses

Upload Your File

Example Upload

Start Bulk Ingest

Endpoint

Description

Request Body

Request Schema

Example Request

Response

202 Accepted

Response Schema

Error Responses

Get Job Status

Endpoint

Description

Path Parameters

Example Request

Response

200 OK

Response Schema

Job Status Values

Error Responses

Complete Example Workflow

Step 1: Request Upload URL

Step 2: Upload File

Step 3: Start Ingestion

Step 4: Check Status