Workflow evals
A workflow changes over time. Evals are how your workspace knows whether a change made it better or worse: a saved set of test cases, a button that runs them all against a chosen version, and a history of results you can compare run to run.
Open a workflow and choose the Evals tab.
Test cases
Section titled “Test cases”A test case is an input plus a rule for what a good output looks like. You can write one by hand, or save one from a run you liked in the Runner tab. Each case has a grader — the rule that decides pass or fail:
| Grader | What it checks |
|---|---|
| Exact match | The output equals the expected text. |
| Contains | The output contains the expected text. |
| JSON path | A field inside a structured output matches a value, or exists. |
| LLM judge | A language model scores the output against a rubric you write. |
The first three are exact and free. The judge is for free-form output, where “contains” is not useful.
Running a suite
Section titled “Running a suite”Run starts every enabled case against the selected version. Each case runs as a real execution of the workflow, then its grader scores the result. You can watch cases finish live, cancel a run in progress, and open any case’s execution to see what each step produced.
Runs can also start on a schedule, and a workflow can gate activation on its suite, so a version whose latest run falls below a pass rate you choose cannot become the active one without an owner’s explicit, recorded override.
Reading a result
Section titled “Reading a result”Every run shows passed / total with a 95% interval beside it, for
example 3/3 passed (95% CI 44–100%). Read the interval as the range of true
pass rates the run cannot rule out. Three passing cases are consistent with a
workflow that fails more than half the time; thirty passing cases are not.
If you want a narrower interval, add cases.
The interval is information. The activation gate decides on the pass rate itself, against the threshold you set.
The LLM judge
Section titled “The LLM judge”For the LLM judge, you write a rubric — a short description of what a good output does — and optionally a pass threshold between 0 and 1. The judge returns a score from 0 to 1 and a sentence of reasoning; the threshold decides pass or fail. Tune the bar by moving the threshold, not by rewording the rubric until the judge agrees.
Writing a rubric that holds up:
- Make it checkable from the output alone: “cites a source” works, “is helpful” does not.
- List the criteria: “answers the question and stays under 100 words”.
- Say what should fail: invented facts, the wrong format, a missing caveat.
The judge uses a fixed, inexpensive model with randomness turned down. If it cannot produce a usable verdict, the case is marked an error — never a pass and never a fail.
Judge samples
Section titled “Judge samples”A language model is not a perfectly repeatable instrument: the same output can occasionally receive a different verdict on a different day. Suite settings → Judge samples lets your workspace measure that instead of ignoring it.
Set it above 1 (up to 5) and each case graded by the judge is scored that many times in every run:
- Pass is the majority of the verdicts. A tie counts as a fail.
- Score is the average.
- Every verdict is kept. A case whose verdicts did not agree is marked Judge disagreed, and Show verdicts opens each sample’s score and reasoning, so you can see which one disagreed and why.
- The run’s interval is widened by the disagreement and its caption says
incl. judge noise. A run whose samples all agreed shows the same interval it would at one sample.
If any one sample fails, the case is an error that names the sample. The gate still decides on the pass rate; judge samples change what the interval tells you, not what the gate does.
What it costs. Each sample is a separate judge call, so a suite at 3 samples spends three times the judge usage of the same suite at 1. Judge usage is recorded against your workspace on the usage page but is not billed to it. The setting is per workflow, and a change applies to the next run, not to one already in progress.
What it does not do. More samples measure the same judge more precisely. They do not make a vague rubric a good one — if a judge keeps passing an output you would fail, change the rubric.
Comparing runs
Section titled “Comparing runs”Pick two runs as A and B to see them side by side, case by case, with the cases whose verdict flipped highlighted. Cases added or removed between the two runs are listed separately rather than treated as failures.