By oqoqo team. Updated October 19, 2026. Experiment figures refer to the public snapshots reviewed on that date.

Build or buy an agent usability test harness?

Use an agent usability platform when your team needs repeatable product experiments, shared evidence, and failure analysis without taking on the whole evaluation system. Adopt an existing framework when you need more control over execution and can own its operation. Build additional infrastructure when a pilot reveals a concrete requirement those options cannot meet.

oqoqo is a platform for agent usability testing and agent observability. It is a fit to evaluate when the question is whether coding agents can use your API, SDK, CLI, MCP server, or skills, and which product change improves that experience. Your team still needs to own the customer workflows and the definition of a correct result.

First decide what you are testing

There are two common evaluation jobs. One team is building an agent and wants to compare its behavior. Another team is building a product and wants to know whether external agents can use it. They may share tracing and grading techniques, but require different execution setups.

For product usability, test the actual agent environments customers rely on. Give them realistic access and documentation. Verify the resulting product state and inspect whether they used the interface under evaluation. A test of a chat-model response or suggested tool call does not establish that a coding agent can complete an authenticated integration.

Compare approaches on the same workflow

Approach	Appropriate starting point	Responsibility to plan for
oqoqo product experiments	Controlled comparisons of product interfaces and instructions, with recorded run evidence and friction analysis	Representative tasks, rubrics, valid fixtures, and supported execution requirements
Netlify AXIS framework	An open-source agent-experience evaluation approach your team can adopt and extend	Scenario configuration, environment fit, grading review, and ongoing operation
Harbor framework	Engineering-owned tasks with defined environments and verification logic	Environment images, fixtures, agent setup, task verifiers, and benchmark upkeep
Custom system	A demonstrated execution, integration, or operating constraint existing options cannot satisfy	Adapters, isolation, scheduling, storage, grading, analysis, reliability, and maintenance

These are operating choices, not a performance ranking. No head-to-head benchmark is implied. Review current documentation and test the supported configuration rather than treating an undocumented feature as absent.

Netlify AXIS documents scenarios run through agents, captured transcripts, and scoring across goal achievement, environment, service, and agent dimensions. It also supports repeated runs and baseline comparison. This makes it a substantive framework option rather than a reason to start from an empty repository.

Harbor environments can include the packages, services, and MCP configuration a task requires. Its verifier workflow provides task-specific checking. That approach is useful when your engineers want to control the environment and the acceptance logic, with the corresponding responsibility to maintain them.

What your team should own in either approach

Keep the tasks, starting state, permissions, rubrics, and release requirements reviewable. Define success as observable behavior. For example, a cancellation workflow should check that the intended subscription changed state and that unrelated subscriptions did not change. An SDK integration should check a real product operation and the required interface usage.

Separate the acceptance criteria from the instructions the agent receives. Supplying private endpoints, a complete solution, or hidden maintainer setup changes the test. Preserve a record of the materials and access each condition actually received.

Review passes as well as failures. oqoqo's trajectory research shows why an acceptable final output can conceal a different path through the task. The platform or framework does not remove the need for a fair rubric and evidence review.

What an oqoqo evaluation looks like

In oqoqo's documented model, a task contains the instructions, rubric, files, and machine. A treatment contains the interface or assets being compared, such as an SDK, CLI, MCP server, or skill. Agents are chosen at launch, and trials repeat the combinations.

The public RunType experiment, snapshot v3 reviewed October 19, 2026, illustrates that structure: 13 tasks, five treatments, two agent configurations, and two trials produced 260 attempts. This demonstrates a published experiment's scope; it does not establish that every new customer needs that batch size or that oqoqo outperforms another runner.

For diagnosis, recorded traces expose commands, tool calls and results, files, and token use. Insights identify friction separately from rubric scoring. Use those capabilities to assess whether your team can go from a failed requirement to evidence and a testable change.

Run a bounded pilot before committing

An illustrative pilot could use three representative workflows, two agent configurations, two interface conditions, and three repeats: 36 attempts per candidate. This is a diagnostic design, not a completed study or a statistical reliability guarantee.

Choose a clean setup task, a normal integration, and a known difficult workflow. Provide equivalent fixtures and permissions. Pin the product and agent versions where possible. Add a deliberately broken example to check whether the acceptance test catches the right failure, and confirm that a valid reference solution passes.

For each candidate, answer these questions with actual run evidence:

Can it run the agent configurations and authenticated workflow you need?
Does each attempt start from valid, independent state?
Can the result be checked against product state and required interface use?
Can an engineer inspect the responsible calls, errors, and generated artifacts?
Are configuration records, task definitions, and run evidence available for review or export?
Can the team rerun the comparison after a product change without rebuilding the setup?

Mark unsupported or unverified requirements explicitly. A polished demo and a successful custom pilot answer different questions.

Compare total operating work

Include platform or hosting fees, execution and model charges, integration, fixture resets, agent-version maintenance, rubric review, failed-run triage, and the engineering time spent operating the system. Task authoring and product expertise remain necessary in every approach.

Estimate recurring work from the pilot: who fixes an expired credential, adapts to a changed harness, validates the grader, and investigates a blocked run? A low-cost first script does not estimate the upkeep of a reliable internal service. Equally, a platform is a poor fit if your required execution or evidence cannot be supported in the configuration you would use.

Build the missing component when it resolves a specific constraint. Replace more infrastructure only when the evidence justifies taking ownership of it.

Questions teams ask

When is a managed platform the better fit?

When its supported execution matches your workflows and the team benefits from repeatable experiments, shared review, and diagnosis with less infrastructure ownership. Confirm this with your own task, credentials, and acceptance criteria.

When should we use an open-source framework?

When execution control or integration requirements favor an engineering-owned setup and your team can maintain it. AXIS and Harbor are useful candidates to evaluate for different forms of product and coding-agent testing; they should be piloted rather than assumed interchangeable.

What should we test with oqoqo first?

Start with one workflow that currently fails and one change you want to evaluate. Bring the fixture and a product-state check. Use the agent usability and observability tools guide for a broader checklist and oqoqo vs Lightsage when comparing those product workflows.

Related resources

Book an oqoqo demo with one workflow, its current setup, and the change you want to evaluate.