Agent usability research
Public product experiments with inspectable configurations, task outcomes, trajectories and limitations. Use the evidence to investigate agent usability and understand what a passing task can hide.
These are studies run by Oqoqo, with different dates, versions and task sets. They are not a cross-product leaderboard. Read the experiment workflow and the methods and limitations of each study before comparing results.
32 attempts · published November 2, 2026
Do workflow notes help coding agents use an SDK?
A Qdrant local-mode documentation pilot: 24 integration attempts and eight probes. Compare original grader verdicts with independent artifact checks; inspect the vector-normalization grading discrepancy.
260 attempts · RunType · snapshot v3
How agent behavior changes across product interfaces
Thirteen tasks, five product setups, two agent configurations and two repeats. Results vary by configuration; verified product participation matters alongside task completion.
300 attempts · Bright Data · snapshot v1
Comparing five ways to expose a product to agents
Fifteen tasks across five treatments, two agent configurations and two repeats. The suite mixes inventory, integration, live operation and knowledge tasks; inspect task-level outcomes before drawing an interface conclusion.
60-attempt Vercel experiment · annotated single trajectory
A passing MCP task with a failed file-ID lookup
One passing attempt used a placeholder file ID, received a 404, then recovered through a file listing. The observed recovery sequence is evidence to investigate, rather than a proven savings estimate.
Start your own artifact comparison with our executable SDK agent usability starter, or use the agent observability guide to decide what evidence to retain.