Promptfoo

What it is

Promptfoo is an open-source CLI/library for testing and evaluating LLM prompts: assertion-based test cases, red-team probes, and CI gating on prompt/agent quality. It talks to models through provider configs, including any OpenAI-compatible HTTP endpoint.

Point it at Ingate

Promptfoo's OpenAI-compatible provider accepts a custom apiBaseUrl. Route it through Ingate and add the X-Ingate-Provider header so Ingate knows which upstream provider config to use:

providers:
  - id: openai:gpt-4o-mini
    config:
      apiBaseUrl: https://api.ingateai.com
      apiKeyEnvar: INGATE_KEY
      headers:
        X-Ingate-Provider: openai

INGATE_KEY is a gateway API key (X-Ingate-Key auth), set as an environment variable. Never hardcode it in the config file. Because Ingate passthrough mode accepts any request shape, this works unchanged whether gpt-4o-mini is served by OpenAI directly or by a provider config Ingate routes to instead (Azure OpenAI, a self-hosted vLLM box, etc.), swap the X-Ingate-Provider value, not the promptfoo config shape.

Every promptfoo request is logged and evaluated by Ingate like any other proxied traffic, no separate instrumentation needed to see it in the gateway's dashboard.

Gate a skill from CI

If you're using the Skill Marketplace, you can run promptfoo as a CI check against a skill's behavior and let its verdict satisfy the marketplace's publish gate, no need for Ingate's built-in judge to also approve it.

1. Run promptfoo in CI, capture pass/fail

2. Submit the verdict

curl -X POST https://api.ingateai.com/api/v1/eval-results \
  -H "X-Ingate-Key: sk-ingate-..." \
  -H "Content-Type: application/json" \
  -d '{
    "subject_type": "skill",
    "subject_id": "d290f1ee-6c54-4b01-90e6-d701748f0851",
    "source": "promptfoo",
    "score": 0.92,
    "passed": true,
    "external_ref": "promptfoo-run-4471"
  }'

3. Allow promptfoo as a publish-gate source (one-time, per org)

curl -X PUT https://api.ingateai.com/api/v1/skills/rubric/<rubric-id> \
  -H "X-Ingate-Key: sk-ingate-..." \
  -H "Content-Type: application/json" \
  -d '{"allowed_verdict_sources": ["internal", "promptfoo"]}'

Once both are in place, POST /api/v1/skills/:id/publish accepts a passing promptfoo-sourced evaluation for the skill's current version, in addition to (or instead of) Ingate's own judge.

Reference

• Eval Results API: full request/response shapes, source validation rules
• Skills API: the publish gate and rubric config
• Trace Exports: stream gateway traces to a tracing backend for post-hoc analysis alongside promptfoo runs