By oqoqo team. Updated . Experiment figures refer to the public snapshots reviewed on that date.
Agent Experience evaluation tools for usability and observability
For agent usability testing and agent observability, start with oqoqo when you need to understand how coding agents use your product. Run realistic tasks through your API, SDK, CLI, MCP server, or skills, then inspect the recorded behavior behind each outcome.
Agent Experience evaluation tools should establish whether an agent completes the customer's task, how it gets there, and whether a change improves the result. This guide explains what to measure, shows evidence from oqoqo experiments, and compares managed platforms, evaluation frameworks, and MCP diagnostics.
Agent usability and agent observability in Agent Experience
Agent Experience, or AX, is how an agent discovers, understands, and uses a product on someone's behalf. For a developer product, that can mean finding the right documentation, installing an SDK, authenticating, choosing a tool, recovering from an error, and checking the result.
Agent usability asks whether an agent can complete a useful task with your product. Agent observability makes its behavior inspectable: the steps, tool calls, errors, retries, files, resource use, and final output that explain the result. In oqoqo, those views come together in recorded experiment runs.
| Question | Evidence to collect |
|---|---|
| Can the agent discover the product or interface? | Product selection, documentation retrieval, and available tools |
| Can the interface execute correctly? | Requests, responses, permissions, errors, and resulting state |
| Can the agent complete the customer's task? | A verified outcome under realistic starting conditions |
| What happened during the attempt? | Ordered traces, retries, recovery, interface use, output, and resource use |
| Did a product change improve the experience? | Comparable baseline and revised conditions across repeated attempts |
A valid tool response establishes one part of that chain. An agent can still choose the wrong operation, miss a page of results, or finish through another product. Your evaluation needs both outcome checks and evidence of any required interface use.
This distinction between a transcript and the resulting environment state is also central to Anthropic's guidance on agent evaluations.
Evidence from oqoqo experiments
Comparing product interfaces across 260 attempts
Our public RunType experiment crossed 13 tasks, five setups, two agent configurations, and two trials per combination: 260 attempts. The suite included live workflows, code integrations, and product-knowledge tasks. Here are the recorded passes across all 13 tasks:
| Product setup | Claude Code passes | Codex passes |
|---|---|---|
| Raw agent baseline | 20 / 26 | 21 / 26 |
| SDK | 23 / 26 | 18 / 26 |
| CLI | 19 / 26 | 18 / 26 |
| MCP | 20 / 26 | 19 / 26 |
| Skills | 18 / 26 | 18 / 26 |
Claude Code used Opus 5 with harness version 2.1.283; Codex used gpt-5.6-sol with harness version 0.158.0. The public snapshot reviewed here is version 3. These are descriptive outcomes for those tasks and configurations, with only two repeats per combination.
There is no single interface that improved both configurations across the suite. On the ticket-triage task, the rubric checked that the agent created a saved RunType flow or agent, ran all five tickets through it, and reported the returned results correctly. That makes successful use of the product part of the evidence.
Observing how agents discover an API
In our analysis of 112 baseline vector-database trials, covering 57 Claude Code and 55 Codex trials across eight independent runs, neither agent performed a web search or fetched an external documentation page. They used model knowledge, workspace files, and runtime inspection of the installed package. Seven trials used SQLite instead of the intended vector database.
Those observations are specific to that benchmark. For your own SDK, inspect whether agents actually find your docs, read source, probe the package, or switch tools. Then improve the surface they encounter and test it again.
Agent Experience evaluation tools at a glance
| Tool | Useful starting point | What to establish in a pilot |
|---|---|---|
| oqoqo | Agent usability testing and agent observability across product tasks, agents, and interface setups | Task coverage, recorded traces, friction diagnosis, and evidence behind verdicts |
| Lightsage | Agent usability testing within a discovery and growth program | Relevant integration tasks, supported configurations, and results after a fix |
| Gauge Agents | Coding-agent product selection and documentation or onboarding experiments | How the selected tasks and environments represent your customers |
| Harbor | A benchmark defined through task instructions, environments, and verifiers | Work to build and maintain the task suite and execution setup |
| Inspect AI | Programmable agent evaluations and sandboxed agent integrations | Agent integration, model routing, configuration, and scoring |
| MCPJam | MCP inspection, client configurations, authorization, and evaluation cases | Required client behavior and checks of the final product outcome |
| Promptfoo | Model and tool regression tests with MCP integrations | Whether the chosen provider actually executes returned tool calls |
| MCP Inspector | MCP connections, capability inspection, and direct tool calls | Whether the failure is in the server or in the agent's use of it |
| LangSmith | Dataset evaluations, experiments, and production feedback | How the external agent and product environment enter the evaluation |
| Braintrust | Experiments, scorers, and evaluation of recorded or executed tasks | Which execution, environment, and outcome checks your setup supplies |
These categories describe useful entry points. Several tools cover more than one part of the workflow.
Managed platforms for product experiments
oqoqo for agent usability and agent observability
oqoqo combines custom product benchmarks with visibility into how agents perform the work. Define tasks and rubrics, then compare agents, models, and treatments such as skills, MCP servers, CLIs, and SDKs on managed cloud infrastructure.
Use Matrix to compare outcomes and resource use, Frictions to find recurring obstacles, and Traces and Output to inspect tool calls, commands, files, and delivered work. The methodology connects requirement verdicts to reasons and trajectory citations. Review ambiguous verdicts because model-based grading can be wrong.
Choose oqoqo as the main workflow when your team needs to measure agent usability, diagnose the behavior behind a result, and rerun after a product change. An SDK documentation revision, a new skill, or an MCP schema change can become a controlled comparison.
Lightsage
Lightsage's Agent Usability offering describes realistic integration tasks, API/CLI/MCP discoverability, task completion, authentication, and error recovery. It also describes engineering recommendations and comparisons after fixes.
The broader Lightsage platform includes discovery, agent traffic analytics, competitive benchmarks, and prioritized improvements. Consider it when those stages belong in the same program. For a detailed buying decision, read oqoqo vs Lightsage.
Gauge Agents
Gauge Agents combines coding-agent product selection with task evaluations, traces, and documentation or skill experiments. Its public offering distinguishes the question of which product an agent chooses from whether it can implement a working integration.
Consider it when you want to connect product preference with usability experiments. Use your own workflow to establish the available agent configurations, environment requirements, and grading approach.
Frameworks for a benchmark you operate
Harbor represents tasks as instructions, an environment, and a verification script. It suits teams that want to define executable success checks and own the benchmark implementation. Compare the work of creating fixtures, integrating agents, running trials, and maintaining the suite.
Inspect AI provides programmable evaluation infrastructure. Its Agent Bridge supports integrations with external agents, including CLI agents running in sandboxes. The bridge routes model calls through Inspect, so review model routing and generation settings when reproducing a customer configuration.
A framework is a useful choice when flexibility and integration control justify the operating work. Include that work in the comparison with a managed platform.
MCP testing and diagnostic tools
MCPJam provides server inspection, authorization debugging, tests across client configurations, and evaluation cases. It is relevant when your acceptance criteria include MCP behavior and compatibility. Establish which checks use emulation or configured clients, and which execute the customer workflow you care about.
Promptfoo supports configurable evaluations with MCP integrations. Check execution behavior for the provider you select: its documentation notes that Gemini's general MCP integration supplies tool definitions but requires application code or callbacks to execute returned calls. Exposing a schema and completing a tool-backed task are different test conditions.
MCP Inspector helps isolate connection and server problems through capability inspection and direct calls. Use it to confirm that a known-good operation works with the intended permissions. Follow that check with an agent task to evaluate discovery, argument selection, and completion.
Evaluation and production observability stacks
LangSmith documents offline and online evaluations, datasets, evaluators, and experiment comparisons. Braintrust documents datasets, custom tasks, scorers, experiments, and production scoring, including workflows for more complex agent execution.
For production agent observability, establish how the system collects application traces and what you must instrument. For a product usability experiment, also establish who supplies the external agent runner, environment, reset logic, and outcome checks. oqoqo supplies the controlled experiment and its recorded execution evidence; compare that complete workflow with extending a stack you already operate.
A practical pilot for choosing a tool
Use three tasks:
Install and authenticate from a clean project.
Complete an integration inside an existing application.
Recover from an expected error and finish the intended operation.
Test the current product setup against one proposed improvement, such as revised authentication instructions. Keep the product version, permissions, starting data, and agent configuration fixed. Reset the environment before each attempt.
For an initial diagnostic exercise, three tasks × two agent configurations × two setups × three trials produces 36 attempts. That is enough to inspect representative behavior; it does not establish a precise reliability estimate.
For each attempt, collect:
| Measure | What it tells you |
|---|---|
| Verified task outcome | Whether the requested state or working artifact exists |
| Required product or interface use | Whether the agent completed the work through the path being evaluated |
| Failure and recovery evidence | Where the attempt first became blocked and what happened next |
| Steps, retries, tokens, and time | What successful and failed attempts cost |
| Setup and maintenance effort | What your team must operate beyond the reported score |
Review failed attempts and sample the passes. In our vector-database trajectory review, an agent produced a correct result after abandoning the intended product and finishing with SQLite. A plausible answer and successful product use require separate checks.
When comparing costs, use the same tasks, trial count, and agent configurations. Include platform charges, model inference, infrastructure, integration, and maintenance. For example, oqoqo's platform pricing separates run charges from model inference.
Our 20-task harness study illustrates why outcome and resource use belong together. With the same Opus 5 high-effort configuration, OpenCode and Claude Code both reached a 67.5% pass rate, while Claude Code used roughly twice the tokens. Those findings describe that task set and configuration.
Common questions
What does agent observability mean in oqoqo?
It means inspecting recorded experiment executions through traces, output, metrics, and friction findings. You can review tool calls and commands, see what the agent read or changed, and connect those actions to a rubric verdict. Continuous production trace collection is a separate integration requirement to establish when selecting an observability stack.
Can I test Agent Experience with MCP Inspector alone?
Inspector can establish that a server connects and its operations work. To evaluate usability, also give an agent a customer task and verify the result. That introduces discovery, tool selection, sequencing, and recovery into the test.
How is product Agent Experience different from evaluating an agent we built?
The test target changes. A product AX evaluation asks how external agents use your software. It needs the interfaces, documentation, permissions, and environment those agents encounter. Your existing evaluation stack can still supply parts of that system.
Should we build a harness or buy a platform?
Start by implementing one task in each approach. Compare evidence quality, execution fit, recurring cost, and maintenance. Keep the task definitions and acceptance criteria understandable to your team whichever tool operates the runs.
How do we know an agent actually used our SDK or MCP server?
Make use of the required interface an explicit acceptance criterion. Inspect the execution evidence and check the resulting product state. Track unrestricted interface choice as a separate experiment.
Start with a workflow you need to improve
Bring a failing integration, a documentation revision, or an interface change you are considering. Use oqoqo to measure agent usability, observe the execution, identify friction, and compare the next version against the current setup.
Explore a public oqoqo experiment, or book an oqoqo demo to turn your own workflow into an experiment.
By the oqoqo team. Updated October 19, 2026. Experiment figures refer to the public snapshots reviewed on that date.
Related resources
- How to test MCP task success beyond protocol checks
- How to test SDK usability with Claude Code and Codex
- Agent observability: find why coding agents fail to use your product
- Build or buy an agent usability test harness?
- oqoqo vs The Prompting Company for agent usability and observability
- oqoqo vs 2027.dev (Firegoat) for agent usability and observability
- oqoqo vs Armature for agent usability and observability
- oqoqo vs MCPJam for agent usability and observability