Does it complete the task?
Define success for the whole workflow, including the expected deliverable or application state.
02 / AGENT EVALUATIONS
Does your agent
get the job done?
Evaluate the agent you are building on the work it is meant to do. Compare changes to prompts, models, tools, and configuration with evidence from repeated trials.
YOU BRING
Your custom agent and representative tasks from its intended workflows.
YOU COMPARE
Agent versions, models, prompts, tools, and configuration.
YOU LEAVE WITH
A clear view of regressions, improvements, and the runs behind each result.
THE QUESTIONS THAT MATTER
Know what each change affects.
Which changes are helping?
Compare versions on the same tasks, with repeated attempts and consistent evaluation criteria.
What is the tradeoff?
Compare rubric outcomes with elapsed time and token usage. Include cost estimates where the provider makes them available.
START WITH A BRIEF
Tell your agent
what you want to learn.
It can help turn your context into tasks, rubrics, and an experiment you can run again. Review the proposed tasks and criteria before launch.
EXAMPLE PROMPT
Evaluate our shopping agent before the next release. Cover product search, cart changes, and unavailable items without silent substitutions. Check product identity, final cart contents, and arithmetic against observed prices. Compare the current and candidate versions on the same taskset across repeated trials. Review task completion, time, token usage, and the evidence behind regressions.
HOW IT WORKS
From the first question
to your next change.
Describe the agent’s job
Bring representative workflows and examples. Your coding agent helps build tasks and rubrics around them.
Connect your custom agent
Add your agent, prepare the environment, and choose the versions or configurations to compare.
Find the next improvement
Read verdicts and trajectories, investigate regressions, and run the evaluation again.
EXPLORE THE PLATFORM
Your next eval starts
with your coding agent.
Give it the framework and infrastructure
to design, run, and analyze your evaluations.