By oqoqo team. Updated October 19, 2026. Experiment figures refer to the public snapshots reviewed on that date.

How to test SDK usability with Claude Code and Codex

To check whether Claude Code or Codex can use your SDK, give each agent a customer task in a clean project, provide only the documentation and access a customer would have, and run an independent acceptance check. Measure installation, authentication, integration, and the resulting product state separately. Repeat the test with pinned configurations.

oqoqo provides agent usability testing and agent observability for SDK workflows. Teams can compare interface treatments across coding-agent configurations and inspect the recorded attempts behind each result. The purpose is to learn whether an agent can use your product and what change would make the workflow more reliable.

Define what a clean project contains

A clean project needs an explicit boundary. Record the language and runtime, package manager, starter files, network access, account permissions, and documentation available to the agent. Exclude private implementation hints and hidden maintainer setup unless customers receive them too.

Use two labeled tracks when necessary:

Onboarding: the agent must discover the installation and authentication procedure, install the SDK, and complete the task.
Configured integration: the correct SDK and scoped test credentials are already available; the agent must use them in an application.

The first track measures the whole first-use experience. The second isolates the integration work after setup. An authentication outage should not be mistaken for a method-naming problem.

Test a customer goal beyond the quickstart

Choose a workflow with an observable outcome. For example, ask the agent to integrate a document-processing SDK into a starter app, submit a supplied fixture, wait for processing, and save the resulting output. Keep the acceptance test outside the instructions given to the agent.

This illustrative task can expose missing initialization, invented methods, incorrect asynchronous handling, and output that looks convincing but was never produced by the product. Include an existing-project task as well as a clean installation task if customers commonly integrate into an established codebase.

Avoid writing a hidden sequence of required function calls into the rubric when multiple correct SDK paths exist. Require a specific path only when that path is the behavior you intend to test.

Verify the result and use of the SDK

Score these questions independently:

Check	What would establish it?
Installation	The intended package and version were installed in the test project
Authentication	A real authorized operation succeeded with the scoped test account
Integration	The generated application runs against the intended product
Correct outcome	Returned or persisted state satisfies the task's requirements
SDK participation	Execution evidence shows the required SDK path was used
Accurate reporting	Claimed identifiers and results match the observed operation

An import statement alone does not establish SDK participation. Conversely, a direct API request may be a valid solution to the user's broad problem while failing an SDK-specific usability test. Decide how to report that before seeing the results.

The public RunType experiment includes a ticket-triage task that makes product participation explicit: the rubric checks for a saved flow or agent and its returned ID, five actual ticket executions, and a report consistent with those results. This illustrates how a rubric can check product use beyond plausible final text.

Real results: the SDK condition varied by agent configuration

Published RunType snapshot v3, reviewed October 19, 2026, contains 260 attempts: 13 tasks, five treatments, two agent configurations, and two trials per combination. The SDK and raw-agent conditions each contain 26 attempts per configuration.

Agent configuration	Raw-agent condition	RunType SDK condition
Claude Code 2.1.283, Opus 5	20 passed / 26 attempts	23 passed / 26 attempts
Codex 0.158.0, gpt-5.6-sol	21 passed / 26 attempts	18 passed / 26 attempts

These are aggregate counts from the published task suite, not a dedicated clean-project SDK study or a universal interface ranking. The snapshot does not specify an effort level for these configurations. The opposing directions are a reason to inspect individual tasks and configurations before declaring an SDK treatment broadly better.

For your own SDK, run the clean-project tracks explicitly. Keep the baseline and revised condition comparable, and report counts at the workflow level so one easy task does not hide a difficult integration.

Use a matrix you can inspect

A useful diagnostic pilot might contain three workflows, two agent configurations, two setups, and three repeats: 36 attempts. This is an illustrative design, not a completed experiment or a statistical certification threshold.

One setup could use the current SDK materials and another a revised example or SDK version. Keep permissions, starter files, product state, and budgets fixed. If the revision changes the package and its documentation together, describe it as a combined intervention rather than attributing any difference to one component.

In oqoqo, tasks hold instructions, rubrics, files, and machines; treatments hold the SDK, skills, or other assets being compared. Agents are selected when the experiment launches. Preserve the actual harness version, model, effort configuration, and trial count in the result.

Diagnose failures before rewriting all the documentation

Find the first transition that failed: package discovery, installation, credential loading, initialization, method choice, execution, or verification. Then read the relevant commands, responses, and generated files in the recorded trace.

Check whether the agent encountered the relevant documentation at all. oqoqo's 112-trial navigation study observed no external documentation fetches or web searches in that particular study. That finding does not mean agents never read documentation; it makes the discovery path something to measure in your own setup.

If the agent read the correct example and still used it incorrectly, investigate prerequisites, ordering, types, and errors. If it never found the example, adding more prose to the same page may leave the original obstacle untouched.

Make the suite useful for the next release

Retain the tasks and verification checks, then compare current and candidate SDK versions under the same agent configurations. Review known failures and a sample of passes. Treat tokens, retries, and time as supporting measures alongside successful outcomes. Keep unavailable infrastructure and invalid fixtures distinct from SDK usability failures.

Set release requirements around the importance of the workflow and the evidence you have. A small repeated suite helps catch regressions; it does not justify a precise reliability claim for all customers and agents.

Questions teams ask

Is generated code enough to evaluate our SDK?

Only if the relevant task ends at code generation. For an integration task, run the code and check the actual product operation and result. Correct-looking imports and final explanations can coexist with a broken integration.

Should we compare Claude Code and Codex with the same prompt?

Use the same customer goal and equivalent starting conditions. Record each harness and model configuration. Compare task outcomes and failure evidence, while preserving any environment differences that affect interpretation.

What should we bring to an oqoqo evaluation?

Bring a representative integration task, a resettable test account, the SDK materials customers receive, and a check for the correct product outcome. The agent usability and observability tools guide explains how to assess the execution and evidence workflow.

Related resources

Book an oqoqo demo with one workflow, its current setup, and the change you want to evaluate.