> ## Documentation Index
> Fetch the complete documentation index at: https://openrouter.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ori Eval

> Prove the best model for your project by testing your agent on real prompts, on a pinned harness

Ori Eval answers "which model is best for what I'm building?" with evidence instead of vibes. Ask the question in plain words and the agent finds test material in your repo, writes the eval, runs your candidates, and comes back with a ranked answer. Under the hood an eval is a `*.eval.ts` file — you can write one by hand whenever you want the control, but you never have to start there.

## Why run evals through Ori

The hardest part of evals is keeping the test bench stable. Ori is a harness, not just a skill: it pins the model and settings at a level prompts can't, so a score change means your agent changed, not the environment. And because Ori routes through OpenRouter, a bakeoff can cover every major lab's models — not just the one vendor your usual harness supports.

## Before you start

Install the CLI:

```sh theme={null}
curl -fsSL https://openrouter.ai/labs/ori/install.sh | sh
```

Sign in once:

```sh theme={null}
ori
```

An eval calls a real model, so a credential is required — the browser login covers it, or export `OPENROUTER_API_KEY` directly. Running evals also needs [Bun](https://bun.sh); if it is missing, `ori eval` offers to install it on an interactive terminal.

<Tip>
  Your project does not need to be TypeScript, and TypeScript never needs a
  separate install: the eval file tests the agent, not your codebase.
</Tip>

## Start from a question

From your project directory, start a session and ask:

```sh theme={null}
cd my-project
ori
```

```text theme={null}
What's the best model for my support agent?
```

You do not need to know eval methodology or model names. The agent:

1. **Scans your repo** for test material — existing prompts, tool definitions, data files (`.jsonl`, `.csv`, chat logs), and gold answers. Tests in any language count: pytest files and Go tests get mined for cases, even though the eval itself is always TypeScript.
2. **Asks what matters** — a question or two like "accuracy first, or speed and cost?" — only when the answer changes the eval.
3. **Writes and runs the eval** — scaffolds the `*.eval.ts`, seeds candidate models from OpenRouter's live catalog under your price cap, and runs them.
4. **Recommends one model**, with the scores, latency, and cost that back it up.

Every artifact stays in your repo as plain code: re-run it when a new model drops, tighten the criteria, or put it in CI.

## Under the hood: the eval file

When you want control — or want to understand what the agent built — here is the whole surface. An eval reads like a normal `bun test`:

```ts recommends.eval.ts theme={null}
import { test } from 'bun:test';
import { setupAgent } from 'ori/eval';

const agent = setupAgent();

test('recommends restaurants using the search tool', async () => {
  const run = await agent.run('Where should I eat dinner in Lisbon?');

  run.tool('search').toBeCalled();
  run.tool('delete_file').toNotBeCalled();
  run.toComplete();
});
```

`setupAgent()` with no arguments is your workspace's own resolved harness and model — the same agent you already run. Then:

```sh theme={null}
ori eval
```

Ori discovers every `*.eval.ts` in the directory, boots an ephemeral runtime, and hands the files to `bun test`. The exit code mirrors `bun test`, so a failing eval fails CI.

### Compare models

Seed candidates from OpenRouter's live catalog instead of hand-listing slugs, cap the price, and run the same eval against each:

```ts model.eval.ts theme={null}
import { test } from 'bun:test';
import { candidateModels, setupAgent } from 'ori/eval';

const candidates = await candidateModels({
  limit: 5,
  maxPromptPrice: 0.000005,
});

for (const model of candidates) {
  test(`handles a refund request on ${model}`, async () => {
    const run = await setupAgent({ model }).run(
      'A customer wants a refund for order #1234. What do you do?',
    );
    run.tool('lookup_order').toBeCalled();
    run.toComplete();
  });
}
```

### Grade open-ended answers

When there is no single right string, grade with an LLM judge. The judge defaults to a dedicated agent on a strong grading model, so scoring stays independent of the model under test:

```ts quality.eval.ts theme={null}
import { test } from 'bun:test';
import { setupAgent, setupJudge } from 'ori/eval';

const agent = setupAgent();
const judge = setupJudge({ minScore: 0.8 });

test('gives an accurate, grounded answer', async () => {
  const run = await agent.run('What is our refund policy for digital goods?');
  await judge.autoEvals({
    criteria: 'Cites the 14-day window and does not invent exceptions.',
    run,
  });
});
```

### Eval against your own data

Real usage data beats synthetic prompts. Loop chat logs or Q\&A pairs with `test.each`:

```ts theme={null}
import supportPairs from './support-pairs.json';

test.each(supportPairs)(
  'answers: $question',
  async ({ question, mustMention }) => {
    const run = await agent.run(question);
    run.toMention(mustMention);
    run.toComplete();
  },
);
```

## Run it in CI

Evals spend model calls, so put them in a credentialed, opt-in job — not your default unit-test run — with the key as a secret. A failing eval fails the build, so a worse agent stops the ship instead of your users. `ori eval --list --allow-no-key` needs no key and cheaply asserts evals exist.