By oqoqo team. Updated October 19, 2026. Experiment figures refer to the public snapshots reviewed on that date.

Agent observability: find why coding agents fail to use your product

Agent observability helps a product team see what an agent attempted, what happened at each step, and whether the product workflow succeeded. For agent usability testing, connect the recorded trajectory with verified outcomes, product participation, and friction. A final answer or aggregate pass rate leaves important parts of that picture unexplained.

oqoqo combines agent usability testing and agent observability in controlled product experiments. Its recorded traces, output, requirement-level evaluations, and friction analysis help teams diagnose how coding agents use APIs, SDKs, CLIs, MCP servers, documentation, and skills. This guide covers those recorded experiment runs; continuous telemetry from every customer's production agent session is a separate deployment question.

Start with four kinds of evidence

Keep these signals linked but separately interpretable:

Evidence	Question it answers	What it cannot establish alone
Trace	Which commands, calls, results, and file operations were recorded?	Whether every business requirement was satisfied
Verified outcome	Did the required product state or artifact exist?	Whether the intended interface was used
Product participation	Did the agent actually use the product or interface under test?	Whether the user's whole task was completed correctly
Friction	Where did the agent encounter errors, repeated work, or missing context?	Whether the overall run failed

oqoqo's Traces documentation describes ordered tool calls and results, commands, files read or changed, and token use. The Output view shows the final assistant message and recorded workspace changes; Evals score rubric requirements. These views support different questions about the same attempt.

Why a passing answer can hide product abandonment

In oqoqo's study of coding-agent product navigation, researchers analyzed 112 baseline attempts across Claude Code and Codex. Seven used SQLite in place of the target vector database. The study also observed zero external documentation fetches or web searches across those attempts.

Those findings describe that task suite and setup. They do not establish that coding agents generally avoid documentation or that every SQLite solution was wrong for a user's broader goal. They show why product teams need evidence of the path taken.

If your evaluation is intended to measure a vector database integration, an output-only check can miss a substitute implementation. The trajectory analysis article develops this product-use problem. Add an explicit product-participation requirement when it is part of the decision, and retain the underlying task outcome as a separate result.

Find the first failure in setup and authentication

When an agent gets stuck, reconstruct the transition from the last verified success to the first failed or unverified step. A useful onboarding sequence is discovery, installation, configuration, credential loading, authenticated operation, and outcome verification.

For example, an illustrative SDK run might install the right package, initialize a client, receive an authorization error, retry with the same configuration, and stop. The failed transition is the first authenticated operation. The authorization error alone does not prove whether the cause was missing permission, an incorrect account, expired access, or confusing instructions.

Retain enough evidence to investigate the alternatives:

The run, task, treatment, harness, model, and product versions.
The relevant command or tool call, its timestamp, response status, and sanitized error.
The credential source and permission scope, without exposing the secret value.
The selected account, project, and endpoint where these can be safely recorded.
The next action, whether it changed the suspected cause, and whether recovery succeeded.

Check environment preparation before assigning fault to the product. Missing test credentials, an unavailable service, and a broken fixture deserve explicit labels. They should remain visible without silently becoming evidence that the agent cannot use your interface.

A recovered error can still be worth fixing

The public Vercel MCP experiment, snapshot v4 reviewed October 19, 2026, includes four recorded occurrences of an invalid deployment target value; all recovered. It also records six occurrences of a case-sensitive fixture filename mismatch, all recovered.

These friction counts are occurrences, not counts of failed runs. They describe the configured integration and fixtures rather than a universal Vercel defect. Their value is diagnostic: the trace can show why an otherwise successful workflow needed extra attempts and whether the proposed fix belongs in the product interface, test fixture, or evaluation environment.

Track whether a friction recurs, whether the agent recovers, and the extra work it requires. Fixing a repeated detour may improve the experience without changing the pass count. Conversely, a quicker run that no longer checks the outcome may not represent an improvement.

Use observability to test a product change

Turn a suspected cause into a bounded experiment. Keep the task, files, machine, permissions, and rubric fixed. Compare the current interface or instructions with a revised treatment under the same agent configurations and repeated trials.

Review the evidence in this order:

Check verified task outcomes and product participation.
Locate the relevant failure or detour in the trace.
Inspect whether the revised condition changed that behavior.
Compare retries, time, and token use alongside the outcome.
Review a sample of passing attempts for hidden workarounds or unsupported claims.

oqoqo's Frictions workflow analyzes saved traces and output through Insights, separately from rubric Evals. Teams can trigger Insights after a run and compare two or three runs side by side in Traces, Output, and Evals. A friction finding is a starting point for investigation; the rerun tests whether a proposed change helped.

What to ask of an observability tool

Ask whether it can move from a result to the relevant recorded interaction, show the actual responses behind a claim, preserve configuration details, and export the trajectory for review. Ask how it represents missing evidence and infrastructure failures. Verify retention and credential-handling requirements for the environment you intend to test.

For a product team, prioritize evidence that leads to a reproducible fix. An attractive score without task-level evidence is difficult to use in an engineering decision. The agent usability and observability tools guide provides a broader selection framework.

Questions teams ask

How is agent observability related to agent usability testing?

Usability testing measures whether an agent can complete a representative product workflow under specified conditions. Observability supplies execution evidence that helps explain the outcome and select the next change to test. Use both to connect a result with an actionable diagnosis.

Does every friction mean the run failed?

No. An agent can recover and finish successfully. Keep friction, recovery, outcome, and resource use separate so the report shows both completion and the work required to get there.

Can oqoqo show the exact cause of every failure?

Recorded evidence can narrow a failure to a specific interaction and support a diagnosis. Missing context or multiple plausible causes may require another controlled run. Start with one failing workflow in oqoqo's experiment model and retain the evidence needed to test your explanation.

Related resources

Book an oqoqo demo with one workflow, its current setup, and the change you want to evaluate.