By Oqoqo team. Updated . Experiment figures identify their source snapshots and review dates below.
Agent usability testing for APIs, SDKs, CLIs and MCP
Oqoqo is an agent usability testing platform for product teams. Run coding agents against your API, SDK, CLI, MCP server or skills in managed cloud environments. Compare task outcomes, inspect the trajectories behind them, and identify the interface or documentation changes to test next.
Agent usability measures how successfully an agent uses your product on a customer's behalf: discovering the interface, installing dependencies, authenticating, choosing operations, recovering from errors, and producing the required result. Agent observability supplies the execution evidence that explains the attempt.
Start an Oqoqo experiment or bring a workflow to a demo. To compare platforms and frameworks first, use the agent usability testing tools guide.
Test the workflow your customers need
Give agents such as Claude Code and Codex a realistic task, the same starting state, and the product access a customer would have. A treatment is the toolset or instruction condition available during an attempt. Compare conditions within the same agent configuration, then inspect differences across agents.
| Product question | Example task | Evidence to retain |
|---|---|---|
| Can agents use our SDK? | Build an integration in a clean project | Install and authentication steps, code, required SDK use, executable checks |
| Can agents use our MCP server? | Find a deployment's source file and return its contents | Tool discovery, identifiers, arguments, errors, recovery and returned file |
| Which interface should we expose? | Complete the same product workflow through SDK, CLI or MCP conditions | Verified outcome and product use, calls, tokens and elapsed time per condition |
| Did our documentation or skill help? | Repeat a workflow with original and revised instructions | Exposure and engagement evidence, outcomes, recovery and resource use |
| Will this release break agent workflows? | Repeat an agreed task set against the current and candidate release | Versioned fixtures, requirement verdicts, artifacts and failure trajectories |
Your team defines acceptance criteria and prepares valid fixtures. Check the supported runtime, dependencies and access during the pilot. Oqoqo's core concepts and experiment workflow describe the experiment model.
Inspect passes as well as failures
A final answer can look correct while the agent bypasses your product, retries a confusing operation, or leaves the wrong state behind. Keep task completion and required product use as separate checks. Our research on successful answers that bypass a product explains why this matters.
Oqoqo records trajectories and outputs and provides friction analysis for experiment runs. This connects agent observability to usability questions: which action first went wrong, what the agent tried next, and which change is worth evaluating. Where success depends on remote product state, capture an authoritative read-back or executable check as part of the experiment evidence; a trajectory-based verdict alone does not establish that state.
Public experiments you can inspect
260 attempts comparing product interfaces
The RunType experiment, public snapshot v3, crossed 13 tasks, five setups, two agent configurations and two repeats. Claude Code recorded 23/26 passes in the SDK condition versus 20/26 in its raw baseline; Codex recorded 18/26 versus 21/26 respectively. These small-sample descriptive results show why interface choice needs testing within each configuration. They do not establish a universal SDK advantage. See the interface comparison guide for versions and limits.
A passing Vercel MCP task still contained a failed lookup
In the Vercel experiment, a historical passing attempt used a placeholder file identifier, received a 404, listed files, and then passed the returned uid as fileId. The inspected sequence contained one extra failed MCP read beyond a valid list-then-read path.
The two model turns following the failed-request turn generated 995 output tokens and spanned 23.319 seconds, including a legitimate source-hash check. Those are observed recovery-sequence totals. A controlled before/after experiment is needed to estimate avoidable cost. Read the annotated file-ID handoff.
32 attempts evaluating supplied SDK documentation
Our SDK documentation pilot held the tasks, fixture and SDK fixed while changing supplied reference content. Independent checks passed all 24 submitted integration implementations; the original model grader had failed one. The pilot did not establish a documentation uplift. Methods, artifacts and the grading discrepancy are available for review in the report and research index.
Choose the evaluation workflow
Evaluate Oqoqo when you want managed experiments comparing how external coding agents use your product, with shared trajectories and friction findings. Your team continues to own the tasks, fixtures and outcome checks.
MCPJam is an option for MCP-focused inspection, authentication debugging and evaluations. Promptfoo's coding-agent guide describes configurable providers, test workspaces and assertions. Choose based on the interfaces, execution mode and evidence your workflow requires.
Braintrust and LangSmith document evaluation infrastructure built around datasets and evaluators. When considering them for product usability, specify the agent runner, fixture isolation and product-state checks your team would operate. They can also support an existing evaluation program alongside specialized product experiments.
This Oqoqo-authored page describes fit and inspectable evidence. It does not report a head-to-head platform benchmark. Use the build, adopt or buy guide to compare ownership and operating work.
Run a small pilot
- Choose two customer workflows and two product conditions you want to compare.
- Prepare clean, independent fixtures and check a valid reference implementation.
- Select two agent configurations and run three repeats per cell: 24 proposed attempts.
- Check the resulting state and any required interface use. Inspect the trajectories behind passes and failures.
- Compare success, input/output tokens and elapsed time separately. Use the findings to design the next product change.
This is a proposed diagnostic design, not a completed result or a reliability guarantee. Start with the executable SDK starter and blank attempt ledger, or define an experiment with the Oqoqo team.
Common questions
Does protocol compliance establish agent usability?
Protocol checks establish a connection and contract baseline. Real-task evaluation also checks discovery, tool choice, arguments, recovery, required product use and the resulting outcome. Use both layers.
Does agent observability mean production monitoring?
Oqoqo's observability here concerns controlled experiment runs and the evidence behind their results. If your requirements include continuous production trace ingestion, assess that integration separately. See the agent observability guide.
Can Oqoqo guarantee a lower token count?
A usability improvement needs measurement on matched tasks and configurations. Report successful outcomes and resource use together, keep missing measurements explicit, and attribute savings only after testing the revised condition.