Scout leads on structure
The widest margins are on multi-step tasks: repair lookahead (70 against a next best of 51.7) and the BTC strategy build (81.7 against 55 for Claude Code and Codex).
ScoutBench results
AI trading benchmark results
ScoutBench scores AI coding agents on the failures that break trading research. Same model, four harnesses, five tasks, and 20 verifier checks per task. Here are the full results, including where Scout loses.
Scores by task
Mean verifier score from 0 to 100 for each harness on each task. The same model ran inside every harness.
| Task | Scout | Claude Code | Codex | OpenCode |
|---|---|---|---|---|
| Hallucination resistance | 25 | 25 | 28.3 | 21.2 |
| Numeric discipline | 30 | 45 | 30 | 45 |
| Repair lookahead | 70 | 51.7 | 45 | 35 |
| Research memo | 60 | 55 | 62 | 55 |
| BTC strategy | 81.7 | 55 | 55 | 36.7 |
| Mean across tasks | 53.3 | 46.3 | 44.1 | 38.6 |
ScoutBench mean verifier score by task. Model held constant, n=3, 20 checks per task.
How to read the results
Scout trails on arithmetic
On numeric discipline Scout and Codex score 30 while Claude Code and OpenCode reach 45. That gap is a concrete target for the Scout harness.
Memos are a tie
Codex scores 62 to Scout's 60 on the research memo, which is inside the noise of a three-trial sample.
Nobody is reliable yet
Hallucination resistance tops out at 28.3 for every harness. Every agent fails roughly three of four checks, which is why generated code must stay inspectable and humans stay in the loop.
The five tasks
Hallucination resistance
Does the agent invent tickers, fields, prices, or API surfaces that do not exist?
Numeric discipline
Returns, position sizing, percentages versus basis points, and compounding done correctly.
Repair lookahead
Find and fix a strategy that peeks at future data without breaking the strategy's intent.
Research memo
Turn a run into a memo whose claims trace back to actual evidence rather than narrative.
BTC strategy
Take a crypto strategy idea from description to a working, testable implementation.
Related
- ScoutBench method and analysisTask design, verifier checks, and what the wins and losses mean.
- Scout Cloud, the AI quant research workspaceThe research harness the benchmark measures, in a shared browser session.
- AI quant research assistantHow Scout Research gathers evidence and tests ideas.
- AI backtesting toolThe validation and robustness checks behind the lookahead results.
Questions
- What is an AI trading benchmark?
- An AI trading benchmark scores AI agents on trading and research tasks with automated checks. ScoutBench focuses on the failures that make quantitative work untrustworthy: hallucinated data, arithmetic errors, lookahead bias, unverifiable memos, and end-to-end strategy builds.
- Which agent scored highest on ScoutBench?
- Scout had the highest mean score at 53.3, followed by Claude Code at 46.3, Codex at 44.1, and OpenCode at 38.6. Scout led on repair lookahead and the BTC strategy task, and trailed on numeric discipline and, narrowly, on the research memo.
- What does a ScoutBench score mean?
- A score is the mean share of 20 programmatic verifier checks an agent passed on a task, from 0 to 100. A score of 70 means about 70% of checks passed. It is not a win rate against other agents.
- Why hold the model constant?
- With the same model in every harness, differences in score come from the harness: its context handling, validation, and guardrails. That isolates what a trading-specific harness adds over a general coding agent.
- What are the limits of these results?
- There are three scored trials per cell and one model, so small gaps such as 62 versus 60 are within noise. Polarline Markets built both ScoutBench and Scout, so read the results with that interest in mind and check the method.