How to measure whether documentation helps coding agents

To measure whether documentation helps AI agents, compare the same customer tasks with the original and revised documentation, then independently check the resulting code or product state. Keep the model, harness, effort, SDK version, starting environment, and documentation exposure consistent. Read the trajectory to explain outcomes; report task success, input tokens, output tokens, and agent work time separately.

Oqoqo combines agent usability testing and agent observability for these controlled product experiments. We tested this method on qdrant-client 1.15.1 in local mode across 32 attempts. This small pilot did not establish that adding workflow notes improved success. It also exposed a disagreement between a model grader and executable checks, which is a useful reason to inspect the evidence before claiming a documentation uplift.

What we tested

The public documentation experiment, snapshot v1 published November 2, 2026, crosses four tasks, two supplied documentation conditions, two agent configurations, and two repeats: 4 × 2 × 2 × 2 = 32 attempts. One environment/documentation probe accounts for eight attempts. The remaining 24 attempts cover three integration workflows:

• Tenant-filtered retrieval: return the nearest records within the requested tenant, honoring the limit and an empty tenant.
• Complete export: return every matching record across multiple pages exactly once.
• Persistent payload updates: open existing local data, update requested fields, close the client, and independently read the saved state after reopening.

Both conditions provide the upstream README pinned to Qdrant Python client v1.15.1. The revised reference appends Oqoqo-authored notes about local persistence, collection reuse, filtering before ranking, pagination offsets, and payload updates. The SKILL.md name, description, body, reference path, SDK, fixture, and task rubrics remain the same. This is a comparison of supplied documentation content; it does not test how agents discover a product through web search.

The remote experiment used Python 3.12.3, Claude Code 2.1.310 with Opus 5 at high effort, and Codex 0.164.0 with gpt-5.6-sol at high effort. Each task/configuration/documentation cell has two attempts. A preliminary pagination pilot is excluded from these 32. The public snapshot and method record identify the versioned inputs.

Results: preserve model grades and executable checks separately

The original model grader passed all eight probes and 23 of 24 integration attempts. We then retrieved each integration attempt's exact submitted implementation and replayed its original workflow check independently. All 24 implementations passed their corresponding supplied check. The replay used the same pinned SDK in fresh Docker containers with networking disabled, read-only inputs, and temporary state; its Python 3.13.16 runtime differs from the remote experiment.

These denominators contain only the three integration tasks, with two repeats each. Passing the supplied check does not prove every possible requirement. The original model verdicts remain intact in the public snapshot; the follow-up checks are a separate evidence layer.

Why the persistence verdict needs investigation

The flagged attempt updated the requested payload and retained the records, and its supplied persistence check passed. The evaluator nevertheless rejected vector preservation after a search-then-patch sequence because a stored cosine vector's magnitude changed. That check only asserted vector dimensionality; it did not assert exact vector components.

Our follow-up diagnostic repeated that sequence against all eight persistence implementations and our deterministic reference solution. All nine showed a magnitude change for the patched vector while preserving cosine direction. This diagnostic used a fresh persisted baseline and Python 3.9 with the pinned SDK; it is not a replacement for the original grades. Qdrant documents normalization for cosine similarity, so exact magnitude needs an explicit, justified contract before treating a change as an application failure.

We do not interpret the 5/6 versus 6/6 grader counts as a demonstrated documentation regression. The next experiment should calibrate the vector-preservation criterion against the SDK's semantics and assert the intended invariant directly. This is why agent observability needs both the trajectory and the resulting artifact, rather than a single pass/fail label.

Did the extra documentation reduce tokens or latency?

The following are arithmetic means across the six integration attempts in each row. They describe this pilot, not proven savings. Input and output tokens are reported separately; the downloadable data includes cached input as a subset, without adding it a second time.

Codex recorded lower mean token counts with the notes; Claude Code recorded higher mean input. Agent work time was similar within each configuration. With only two repeats per task, no estimate of uncertainty, and no complete audit of every external lookup, these differences do not establish a causal documentation effect. Keep the agent configurations separate. Reported full attempt duration, which includes additional execution overhead, is a separate column in the CSV.

A reusable documentation evaluation process

1. Start with a real workflow and an independent outcome. For export, check every record across pages, not whether the agent called scroll. For persistence, close and reopen the store before reading it back.
2. Change only the documentation you intend to test. Keep wrapper descriptions and discovery paths identical when studying content. Test discoverability as a separate change.
3. Record exposure and product participation. The run records mark the supplied documentation as used in all 24 integration attempts. Inspect the actual reads and any alternate documentation or source-code lookup before attributing an effect.
4. Save outputs and review disagreements. Use exact attempt identifiers when retrieving repeated runs. Preserve original grades and follow-up findings instead of silently replacing one with the other.
5. Expand the task set before claiming an uplift. These checks already passed under both conditions. Harder representative workflows and more repeats may be needed to detect a meaningful difference.

The SDK agent usability guide covers clean-project setup and product participation. The SKILL.md evaluation guide explains how to separate an instruction change from added context. Our earlier skills research shows why instructions should be tested against the workflow they are meant to help.

Inspect the evidence or run your own comparison

• Public experiment: original grades, configurations, traces, and outputs.
• Download all 32 attempt observations as CSV; probes and integration attempts are labeled separately.
• Download the method record, source hashes, and follow-up audit scope.
• Executable agent usability starter: SDK fixture, independent verifier, fresh documentation workspaces, and JSON/CSV/Markdown reports. Its deterministic demo validates the fixture; it does not generate model benchmark results.

For a managed experiment on your product, bring one SDK workflow and the documentation change you want to test. Oqoqo records what the agent tried, what the product returned, what the artifact contains, and where recovery effort accumulated. Browse the research index for other public studies and the agent observability guide for the evidence to retain.

Common questions

Do longer agent docs improve task success?

That is a testable hypothesis. This pilot did not establish an improvement: all 24 submitted integration implementations passed the supplied workflow checks under either documentation condition, and resource use varied by agent configuration.

Is a model grader enough to evaluate agent documentation?

A model grader can assess requirements and point to evidence, but calibrate it against executable state checks and product semantics. In this experiment, a vector-preservation verdict needs that additional review.

Can this local SDK fixture measure authentication or production reliability?

No. It uses synthetic records and local storage without a product account or network service. Add separate controlled tasks for authentication, server behavior, discovery, and production constraints.