What are you deciding?
Choose an internal workflow to improve or products to compare. Build representative tasks around that decision.
04 / CUSTOM BENCHMARKS
Build a benchmark
around your work.
Build a benchmark for an internal workflow or a product comparison. Define a shared taskset, starting data, and rubrics, then compare agents, models, or product toolsets under equivalent conditions.
YOU BRING
An internal workflow or product use case, representative data, and success criteria.
YOU COMPARE
Agents, models, or product toolsets on shared tasks and rubrics.
YOU LEAVE WITH
A reusable taskset, explicit rubrics, and results grounded in your use case.
THE QUESTIONS THAT MATTER
Compare options on work that matters.
Are the comparisons fair?
Use common goals and rubrics, equivalent starting data, and the same agent and model configuration when comparing products.
Can you run it again?
Keep the taskset, fixtures, rubrics, and configuration versions so future changes can face the same benchmark.
START WITH A BRIEF
Tell your agent
what you want to learn.
It can help turn your context into tasks, rubrics, and an experiment you can run again. Review the proposed tasks and criteria before launch.
EXAMPLE PROMPT
Build a benchmark from our incident-response workflow. Use representative build failures and checkout incidents, with starting repositories and incident evidence. Define shared rubrics for the correct diagnosis and supporting evidence. Compare the agents, models, or product toolsets we are considering under equivalent starting conditions. Repeat each condition, inspect the runs, and retain the taskset and rubric versions for the next comparison.
HOW IT WORKS
From the first question
to your next change.
Define the taskset
Describe the work and provide examples. Your coding agent helps draft representative tasks and prepare the starting data.
Review the criteria
Review tasks, fixtures, and rubrics. Require evidence that distinguishes successful work from a plausible answer.
Repeat and compare
Choose the configurations or product toolsets. Repeat each condition, inspect variation and failures, and keep the benchmark for the next decision.
EXPLORE THE PLATFORM
Your next eval starts
with your coding agent.
Give it the framework and infrastructure
to design, run, and analyze your evaluations.