Skill Marketplace

Overview

The Skill Marketplace lets an org author reusable agent skills (Markdown + YAML frontmatter), have them auto-evaluated against a rubric, and publish the ones that pass to a cross-org public shelf other orgs can browse and adopt. Every mutation (create, update, adopt) dispatches an async re-evaluation against the org's rubric; the request itself never blocks on that evaluation completing.

The Skill Lifecycle

• Create. A new skill starts private, at version: 1. You supply either raw_source (frontmatter + body) or discrete fields, and creation dispatches the first evaluation.
• Update. Every update logs a new SkillVersion snapshot and dispatches a re-evaluation against the org's rubric.
• Publish. Publishing is gated: the skill's current version must have a passing evaluation from one of the org's allowed verdict sources. If it doesn't, the request fails with 409 not_publishable. Publishing an already-public skill is idempotent and returns 200.
• Public shelf. Published skills appear on the cross-org public shelf, where other orgs can browse and inspect them.
• Adopt. Adopting deep-copies a public shelf skill into the caller's org as a new, private skill (version: 1, source_skill_id set to the shelf entry it was copied from). On a name collision the copy is renamed (-adopted-1, -adopted-2, …).

The Skill Object

{
  "id": "d290f1ee-6c54-4b01-90e6-d701748f0851",
  "org_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "name": "pdf-summarizer",
  "headline": "Summarize a PDF into three bullet points",
  "description": "Extracts text and produces a concise summary...",
  "body": "## Steps\n1. ...",
  "raw_source": "---\nname: pdf-summarizer\n...\n---\n## Steps\n...",
  "version": 3,
  "visibility": "private",
  "published_at": null,
  "source_skill_id": null,
  "created_by": "1c1c1c1c-...",
  "created_at": "2026-08-29T12:00:00Z",
  "updated_at": "2026-08-29T12:00:00Z"
}

visibility is "private" or "public". source_skill_id is set only on skills created via adopt, pointing back at the shelf entry they were copied from. When creating or updating a skill, you can either supply raw_source (frontmatter + body, parsed as-is) or discrete fields (name, description required; headline, body optional), which get synthesized into that shape.

Rubric Config

Each org has a rubric config controlling how its skills are judged and embedded: which judge and embedding providers and models to use, per-axis weights, the pass threshold, and which evaluation sources the publish gate accepts.

{
  "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "org_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "name": "default",
  "judge_provider": "openai",
  "judge_model": "gpt-4o-mini",
  "embed_provider": "openai",
  "embed_model": "text-embedding-3-small",
  "embed_dims": 1536,
  "weights": {
    "name_clarity": 0.2,
    "description_task_descriptiveness": 0.3,
    "semantic_uniqueness_vs_siblings": 0.3,
    "headline_informativeness": 0.2
  },
  "pass_threshold": 0.7,
  "allowed_verdict_sources": ["internal"],
  "created_at": "2026-08-29T12:00:00Z",
  "updated_at": "2026-08-29T12:00:00Z"
}

• Axes and weights. Skills are scored on four axes: name_clarity, description_task_descriptiveness, semantic_uniqueness_vs_siblings, and headline_informativeness, each with a configurable weight.
• Pass threshold. pass_threshold defaults to 0.7.
• Allowed verdict sources. allowed_verdict_sources lists the evaluation sources the publish gate accepts as evidence of a passing evaluation. It defaults to ["internal"], only Ingate's own judge.

Evaluations

Each evaluation records per-axis scores with rationales, an overall score, a pass/fail verdict against the rubric's threshold, and the judge provider and model that produced it:

{
  "id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "org_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "skill_id": "d290f1ee-6c54-4b01-90e6-d701748f0851",
  "skill_version": 3,
  "rubric_config_id": "7c9e6...",
  "axes": {
    "name_clarity": {"score": 0.9, "rationale": "..."},
    "description_task_descriptiveness": {"score": 0.85, "rationale": "..."}
  },
  "overall_score": 0.88,
  "passed": true,
  "judge_provider": "openai",
  "judge_model": "gpt-4o-mini",
  "latency_ms": 842,
  "prompt_tokens": 512,
  "completion_tokens": 96,
  "source": "internal",
  "created_at": "2026-08-29T12:00:00Z"
}

source is "internal" for Ingate's built-in judge, or an external evaluator name (e.g. "promptfoo") for verdicts ingested via the Eval Results API. external_ref (omitted when empty) is an opaque pointer into that external system.

External Verdicts

Promptfoo, or any CI judge, can submit pass/fail verdicts against a skill via the Eval Results API. Those verdicts land as source-tagged evaluations alongside Ingate's own judge runs, and if the evaluator's source name is added to the rubric's allowed_verdict_sources, its verdicts satisfy the publish gate too.

Next Steps

• Skills API reference: the full endpoint reference for skills, the public shelf, and rubric config
• Eval Results API: external verdict ingestion and the publish-gate interplay
• Promptfoo integration: wiring promptfoo verdicts into the publish gate