Datasets

Versioned test case collections for systematic evaluation. Import from CSV/JSONL, sample from production logs, and run evals against dataset versions.

Enterprise feature

Datasets require an Enterprise plan.

Overview

Datasets are collections of test cases organized into immutable versions. Each version contains a set of input/output pairs that can be used to evaluate LLM behavior systematically.

  • Create datasets to organize test cases by use case
  • Import test cases from CSV or JSONL files
  • Sample test cases from production logs
  • Commit immutable versions for reproducible evaluation
  • Run eval definitions against dataset versions

Create a Dataset

POST
/api/v1/datasets

Create a new dataset. Requires member role.

bash
curl -X POST https://api.ingateai.com/api/v1/datasets \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "Customer Support QA", "description": "Test cases for the support chatbot"}'

Dataset CRUD

GET
/api/v1/datasets

List datasets.

GET
/api/v1/datasets/:id

Get a dataset.

PUT
/api/v1/datasets/:id

Update dataset metadata.

DELETE
/api/v1/datasets/:id

Delete a dataset and all its versions.

Versions & Test Cases

Test cases are grouped into immutable versions. Once committed, a version cannot be changed. this ensures reproducible evaluation results.

Commit a Version

POST
/api/v1/datasets/:id/versions

Commit a new version with test cases.

bash
curl -X POST https://api.ingateai.com/api/v1/datasets/{id}/versions \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "note": "Initial test suite",
    "cases": [
      {"input": "What are your hours?", "expected_output": "We are open 9am-5pm"},
      {"input": "How do I reset my password?", "expected_output": "Go to Settings > Security"}
    ]
  }'
GET
/api/v1/datasets/:id/versions

List all versions of a dataset.

GET
/api/v1/datasets/:id/versions/:vid/cases

List test cases in a version.

Import Test Cases

POST
/api/v1/datasets/:id/import?format=jsonl

Import test cases from CSV or JSONL. Auto-commits a new version.

Supported formats: csv and jsonl.

bash
# Import from JSONL file
curl -X POST "https://api.ingateai.com/api/v1/datasets/{id}/import?format=jsonl&note=Imported+v2" \
  -H "Authorization: Bearer <token>" \
  -F "file=@test_cases.jsonl"

Sample from Production Logs

POST
/api/v1/datasets/:id/sample

Create test cases from production log entries.

bash
curl -X POST https://api.ingateai.com/api/v1/datasets/{id}/sample \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"log_ids": [42, 43, 44, 45], "note": "Sampled from production"}'

Build golden datasets

Sample interesting production interactions, manually curate them, then use the resulting dataset for regression testing when you change prompts or models.

Run Evals Against a Dataset

POST
/api/v1/datasets/:id/versions/:vid/eval

Run an eval definition against all test cases in a version.

bash
curl -X POST https://api.ingateai.com/api/v1/datasets/{id}/versions/{vid}/eval \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"definition_id": "eval-definition-uuid"}'
jsonResponse
{
  "run": { "id": "...", "status": "completed" },
  "total": 50,
  "pass_count": 47,
  "fail_count": 3,
  "pass_rate": 94.0
}
Previous
Eval Engine
Next
Playground