By Oqoqo team. Updated . Experiment figures identify their source snapshots and review dates below.
CLI vs MCP vs SDK: how to choose interfaces for coding agents
Choose a CLI, MCP server, or SDK by testing the customer workflows your coding agents need to complete. A CLI is a useful candidate for terminal operations and automation; an SDK for application integration; and MCP for discoverable operations in compatible agent clients. Test skills alongside these interfaces as a guidance layer. Verify outcomes, setup friction, recovery, and effort on your own product.
Oqoqo provides agent usability testing and agent observability across these setups. Our public RunType experiment contains 260 attempts across five setups and two agent configurations. Its results show why an interface decision needs both a task-level comparison and the trajectory behind each outcome.
CLI, MCP, SDK, and skills: what changes for the agent?
| Surface | Useful starting point | What to test |
|---|---|---|
| CLI | Terminal workflows, scripts, operational tasks | Help discovery, structured output, exit codes, pagination, authentication, and composability |
| MCP server | Operations exposed to a compatible agent client | Tool discovery, descriptions, input schemas, response semantics, authorization, and recovery |
| SDK | Product behavior embedded in application code | Installation, runtime compatibility, method discovery, types, examples, errors, and working integration |
| Skill | Instructions and workflow knowledge used with available tools | Natural activation, correct guidance, reference access, and the effect on task success and effort |
These are starting hypotheses. An SDK can support an operational script, a CLI can generate integration code, and an MCP server can expose documentation as well as actions. The MCP architecture describes its client/server boundary and primitives; the Agent Skills overview describes portable instructions and resources.
A skill can tell the agent when and how to use a CLI, MCP server, or SDK. It does not supply an equivalent execution interface by itself. A comparison of “SDK” against “MCP plus skill” measures those complete packages, including the extra guidance.
Evidence: five RunType setups across 260 attempts
The public RunType experiment crossed 13 tasks × five setups × two agent configurations × two repeats. The suite included product discovery, live workflows, code integration, and knowledge tasks.
| RunType setup | Claude Code passes | Codex passes |
|---|---|---|
| Raw agent baseline | 20 / 26 | 21 / 26 |
| SDK | 23 / 26 | 18 / 26 |
| CLI | 19 / 26 | 18 / 26 |
| MCP | 20 / 26 | 19 / 26 |
| Skills | 18 / 26 | 18 / 26 |
Source: published snapshot version 3, completed October 11, 2026 and reviewed November 2. Claude Code used harness 2.1.283 with Opus 5; Codex used harness 0.158.0 with gpt-5.6-sol. Effort was unspecified. Each setup/configuration has 26 attempts: two per task.
In this suite, the SDK condition had the highest observed pass count for Claude Code. The raw-agent baseline had the highest for Codex. Those observations describe these tasks, versions, access conditions, and rubrics. They do not establish a universal interface ranking, statistical significance, or the isolated cause of a difference.
The raw baseline still had product access. Its results do not mean the agent completed the suite without the product. Review the task, its requirements, and the recorded operations before interpreting the treatment label.
Why one successful operation is not a successful integration
The ticket-triage task asked the agent to create a saved flow or agent, run five tickets, and report the categories and priorities returned by RunType. The rubric checked saved-object creation, execution/report fidelity, valid labels, and required urgent priorities separately.
In the SDK condition, Claude Code passed both trials and Codex passed neither fully. Both Codex attempts passed the saved-object and valid-label requirements. A created object alone would have overstated success.
In one Codex evaluation, captured execution output was truncated, preventing complete verification of the returned results; a separate ticket priority was also wrong. This does not isolate the SDK as the source of truncation. It shows why interface comparisons need evidence quality and business correctness alongside API-call success.
How to compare interfaces on your product
1. Select customer outcomes
Choose a first-time setup task, an application integration, and recovery from a realistic error. Keep discovery and instructed use distinct. “Build this with our SDK” tests integration under a named constraint. A natural customer request also tests whether the agent finds and chooses the product.
Give every condition the same customer outcome and independent acceptance check. If using the intended product is a requirement, verify participation explicitly. Our trajectory analysis explains how plausible output can hide product substitution.
2. Check capability and access parity
List the operations each interface supports before interpreting pass rates. Keep the account, data, permission scope, product version, and network access equivalent. Document any missing operation as part of the setup difference.
Decide what the comparison includes. An onboarding test can include installation and authentication. A steady-state operation test can begin with an authenticated environment. Record the boundary and preserve setup timings separately from agent work where possible.
3. Define named setups
Start with baseline, CLI, MCP, and SDK conditions. Pin tool versions and keep unrelated tools out of the comparison. If customers normally combine tools, add that combination as its own condition.
Test a skill as an additional condition on a chosen interface, keeping the underlying tool fixed. The SKILL.md evaluation guide explains how to separate availability, activation, and usefulness.
4. Repeat in clean environments
Use fresh workspaces and agent sessions, isolated external state, and the same task fixtures. Record each harness, model, effort level, permissions, time budget, and documentation policy. Interleave conditions and keep technical failures in the attempt ledger.
A proposed pilot with three tasks, four setups, two agent configurations, and three repeats contains 72 attempts. This is a diagnostic starting design, not a completed study or a guarantee of sufficient statistical power. Report each agent configuration separately and inspect the workflows that matter most to customers.
5. Score quality and explain friction
| Dimension | Useful measurement | Engineering question |
|---|---|---|
| Outcome | Verified passes and per-requirement verdicts | Did the customer's work actually get done? |
| Product participation | Required operations or executed integration | Did the agent use the intended product? |
| Discovery and setup | First correct operation, setup failures, documentation access | Could a new agent find and enter the workflow? |
| Recovery | Failed calls, repeated errors, detours, and eventual outcome | Did the interface help the agent find a valid path? |
| Effort | Token categories, model cost, calls, and elapsed time | What did complete attempts cost, including failures? |
| Evidence coverage | Missing outputs, unverified state, or ambiguous verdicts | Can the result support the decision? |
Inspect passes as well as failures. In a passing Vercel MCP attempt, the agent made a failed file read before learning the file-ID handoff. The valid endpoint worked after discovery. That is a specific description/error-message experiment, rather than evidence that MCP as a format cannot support the task.
How to turn the results into an interface decision
Prioritize the workflows and agent configurations your customers use. A suite average is a starting view; a critical authentication failure or silent data omission can matter more than several easy passing tasks.
Keep pass counts and denominators next to effort measurements. Report within-task variation, and expand the experiment when the observed differences are small or unstable. Failed attempts and retries contribute to operating cost even when a successful run is fast.
Use agent observability to decide what to change: an unclear CLI flag, a missing SDK example, an MCP response that omits the next valid action, or a skill that routes the agent to stale instructions. Change one suspected cause, repeat equivalent tasks, and check that the agent encountered the revision.
Your next release decision might be to improve one existing interface, add a missing operation, or ship a complementary skill. Tie it to the observed customer workflow and the maintenance work your team can support.
Download an interface comparison plan
The interface comparison plan (JSON) describes the proposed 72-attempt pilot. Use the blank attempt ledger (CSV) to retain outcomes, configuration, resource use, and evidence links. These are planning and recording templates, not executable Oqoqo launcher files.
Replace the example task descriptions with real customer jobs and write the verifier before launching. Keep expected answers outside the agent's editable workspace. Test the verifier on correct, incorrect, incomplete, and substituted outputs.
Compare interfaces in Oqoqo
Oqoqo models the comparison as tasks, machines and files, agent configurations, and treatments. Define a treatment for each toolset, repeat the task grid, then inspect requirement verdicts, trajectories, and frictions alongside resource use. The experiment workflow explains how these pieces fit together.
Use Oqoqo when your team needs a managed workflow for measuring agent usability and diagnosing the behavior behind the result. Bring one customer task and the interfaces you want to compare to discuss an evaluation.
Common questions
Is MCP always better than a CLI for agents?
No interface label establishes task success. MCP changes discovery and execution through compatible clients; a CLI changes terminal interaction. Test their actual operations, descriptions, permissions, results, and recovery on matched tasks.
Should a skill replace our SDK documentation?
A skill can supply workflow guidance and route the agent to references. Keep the underlying SDK and reference material accurate and test whether the agent reaches and uses the relevant information. A skill does not remove the need for a working integration surface.
Can we compare interfaces using different models?
Report each model/harness configuration separately. To estimate an interface effect, compare interfaces within the same configuration first. The RunType results show how an aggregate across agents can conceal different observed outcomes.