Backtests and studies

A backtest runs a workflow over the past, one step at a time, and scores what it would have decided at each step against what actually happened next. A study runs many backtests at once to compare different settings or prompts, and ranks them honestly. Both live in a workflow’s Backtest tab.

Everything a backtest runs is marked HISTORICAL, so a replayed decision is never mistaken for a live one.

What a backtest does

1. You pick a config: the time range, the step size (for example one day), and the historical data to replay.
2. The workflow runs once per step. At each step it can only see what was knowable at that moment — nothing from later.
3. Each decision is scored against what happened next. The run card shows the results and a summary of the whole run.

A backtest spends money the same way a normal run does: every model call is metered to your workspace. The run card and Settings → Usage show what it cost.

What a study does

A study is a search. You describe what to try — a grid of parameter values, a number of random draws, or alternative prompts for one step — and the study runs each candidate as a backtest over several windows of time. It then ranks the candidates on windows they were not tuned on, with the luck of trying many things subtracted.

The leaderboard shows, for every candidate:

Column	Meaning
Adjusted	The score after correcting for how many candidates were tried. This is the ranking.
Out-of-sample	The score on windows the candidate was not tuned on.
In-sample	The score on the windows it was tuned on — shown so you can see the gap.
Overfit gap	In-sample minus out-of-sample. A large gap means the candidate learned noise.
Cost	What that candidate’s runs cost, in USD.

The header also shows how many candidates were planned, run and dropped, and what pure luck would have scored as the best of that many — a top score at or below it is a search that found nothing.

The best candidate is scored once more on a holdout: a final slice of time no candidate was ever tuned on. That number is the one to trust.

Every study has a ceiling

A study multiplies spend: every candidate × every fold × two windows × every step. So a study must have a budget before it starts, and there is no “unlimited” option.

Two ceilings are always required:

Ceiling	What it bounds
Max candidates	How many candidates may run.
Max total steps	How many workflow steps the whole study may run.

A third is optional:

Ceiling	What it bounds
Budget (USD)	How much the whole study may cost.

Whichever ceiling is reached first stops the study. Nothing that already ran is thrown away: the study is marked Budget reached, the candidates that ran are ranked, and a winner is still chosen from them.

How the USD estimate works

When you enter a USD budget, the form shows an estimate — for example ≈ $0.01 for 2 candidates × 2 folds × 2 windows + the holdout. The estimate comes from a real run: the last completed backtest of this workflow version in your workspace, divided by the steps it ran, times the steps the study will run. It therefore includes everything a run actually spends — tools, retries and the scoring judge — not just a model price.

Because of that, a version that has never been backtested in your workspace cannot be given a USD budget yet. Run one backtest of it first, or create the study without a USD budget. The form tells you which case you are in.

A budget below the estimate is allowed. The study simply stops before the run that would go over it, keeps what ran, and tells you so on the leaderboard — including when the holdout was the run it could not afford.

What a study costs, and where to see it

As each run finishes, its cost is added to the study and to its candidate. The leaderboard header shows Spend — what the study has cost so far, out of its budget when it has one, beside the estimate it started with. The Cost column shows each candidate’s share. Studies created before this feature show no cost rather than a misleading $0.

Run a study on a schedule

A study answers “which settings worked on the past”. Running it on a schedule answers the next question: does the winner stay the winner as new data arrives?

Turn it on from the Studies section of the Backtest tab: tick Re-run on a schedule, pick the finished study to re-run from, and set a cadence. The study you pick must have a USD budget — a study on a schedule is the same spend, repeated, so the ceiling is not optional. Each re-run copies that budget.

Every re-run:

• keeps the same search space as the study it descends from, and adds the previous winner as a candidate so it has something to beat;
• is scored only on data that arrived after the previous study’s holdout. The previous holdout has been used to choose a winner, so it can no longer be a fair test; the re-run’s folds may learn from it, but its holdout is always fresh. That is what makes every holdout in the chain an honest, independent sample — and what lets the lineage panel say whether the best result across all re-runs is better than luck would produce.

The lineage panel under a study’s leaderboard lists each re-run with the window it was scored on, its winner, the holdout score, and whether the winner changed. The line Expected best of k under noise is what pure chance would have produced as the best of that many re-runs. A best holdout at or below it is luck, not a winner that persisted — the panel says so when that is the case.

When a scheduled re-run picks a different winner than the run before it, the person who set the schedule up gets a “A scheduled study changes its winner” notification (adjustable in Settings → Notifications).

Why did my scheduled study skip?

Between data arrivals, the normal state of a schedule is skipped, and the Studies section shows why under the schedule controls:

Message	What it means	What to do
Not enough unseen data yet: 7 of 20 steps	Fewer new steps have arrived since the last holdout than the study’s minimum (set when you create a study: Min. unseen steps for a re-run).	Nothing — it runs once enough data exists. It never scores a shorter window.
A study or backtest run was in flight	One study at a time per workflow.	It retries at the next due time.
The previous study in the lineage did not finish	The last re-run was cancelled or failed, so there is no winner to re-run from.	Cancel the schedule and re-enable it from a finished study.
The root study has no USD budget	The study you picked was created without one.	Create a study with a USD budget and schedule from that one.
The root study was deleted	The schedule turned itself off.	Pick a new study and re-enable.
Workspace subscription lapsed / usage budget exhausted	Scheduled work is refused until the workspace is restored.	Fix the workspace; the schedule resumes on its own.

Promoting a winner

The leaderboard’s Promote winner button activates the winning candidate’s version through the normal activation path, so your evaluation gate still applies. A winner that only changed parameters (not prompts) has nothing to activate: apply its parameters to the config instead.

Common questions

Why was my study refused when I entered a budget? This version has no completed backtest in your workspace to estimate from. Run one, then try again — or leave the USD budget empty.

Why did the study stop early? A ceiling was reached: candidates, steps or USD. The study says which, and everything that ran is kept and ranked.

Why is the holdout missing? The budget was reached before the winner’s holdout run. The ranking stands; the holdout score does not exist. Raise the budget and run the study again to get it.

Can I run two studies on one workflow at once? No. One study at a time per workflow; cancel the running one or wait for it.

My scheduled study keeps saying “not enough unseen data”. Is it broken? No. A re-run only scores data that arrived after the previous holdout, and waits until there is at least the minimum you set. Add data to the dataset (or extend the config’s time range) and it runs at the next due time.

Why does the lineage warn that the best result is “within noise”? Across k re-runs, the best of k honest scores is still a maximum, and chance alone produces a maximum. The panel subtracts that. If your best holdout does not clear it, the winner has not shown it persists — keep collecting data.