How Oqoqo runs eval experiments

Why runs

Agents are a class of user you cannot observe. When an agent fails on a product, it does not file a ticket or answer a survey. It works around the product or moves on, and the only trace it leaves is a request log that looks like noise.

So we measure instead of guessing: real agents doing real work, in controlled and repeated runs. This page explains how.

The experiment

An experiment crosses tasks, treatments, and agents, and repeats each combination. Oqoqo launches and reports that grid as one experiment, so the comparison stays fair when you change a product, an agent, or a model.

Every number in the report traces back to one cell of that grid: this task, this treatment, this agent, this trial.

The environment

Each trial runs in its own environment: a fresh sandbox with the project files, credentials, and the tools the agent needs. Four rules hold on every trial:

Evaluation

Every run is graded requirement by requirement. An evaluator reads the trajectory — each step, tool call, and output — along with the files the agent changed, and decides each requirement: pass or fail, with a reason and a citation into the run. The task passes only when every requirement passes.

Recorded work stays fixed. Changing a rubric does not run the agent again: fix the rubric and choose Re-trigger Evals to grade the same recorded work under the updated requirements, or launch a new experiment when the task itself changed.

Evals produce the verdicts. Insights reads the same trajectories for frictions, wasted tokens, and other patterns, and runs on its own — a fresh analysis never changes a verdict.

Results

An experiment reports pass rate for each treatment and each agent, and lift: the difference in pass rate between a treatment and the baseline on the same tasks. Each run also reports its steps, tool calls, tokens, and duration, so cost sits beside outcome.

Every trial counts toward these numbers, including the failed ones.

Frictions

Pass rates say how often agents succeed. Frictions say why they do not. A friction is one specific obstacle an agent hit during a trial: documentation that contradicts the API, a setup step that fails, an error message that points the wrong way. Each friction records:

Family

What kind of obstacle it is, so recurring problems add up across trials.

Severity

What it cost, from a short detour to a failed task.

Recovery

Whether the agent got past the obstacle or was blocked by it.

Owner

Who can fix it: the product, its docs, the environment, the agent harness, the agent's own reasoning, or the task itself.

Evidence

The exact trajectory steps that produced it.

Repeated frictions are grouped across trials, so one underlying defect appears as one finding. Fix it, then re-run.

Versioning and access

Tasks are versioned, and launching an experiment snapshots the exact treatments, skills, and environment it uses — so a result always records what it measured. A published result changes only when a new run replaces it, never by silent recalculation.

People launch experiments from the web app; agents launch them from the CLI or MCP. Your projects hold your tasks, treatments, credentials, and results. Your tasks stay out of training corpora, so agents cannot pass them from memory.

Limitations

An experiment cannot tell you everything. These limits apply:

Task scope

An experiment measures only its tasks. A high pass rate on 24 tasks tells you about those workflows, not about every workflow the product supports. Choosing tasks well matters more than anything computed from them.

Agent drift

Results depend on the agents and models used, and both change quickly. Every report names its agents, models, and effort settings, and results should be refreshed as those change.

Evaluator fallibility

The evaluator is a model and can be wrong. The reason and citation on each verdict make it easy to check, but the risk does not go away.