By Oqoqo team. Updated . Experiment figures identify their source snapshots and review dates below.
How to evaluate SKILL.md: task success, tokens, and latency
To evaluate SKILL.md, run the same customer tasks with and without the skill, verify the resulting work, and compare task success, skill activation, token usage, and elapsed time. Keep the agent, product version, permissions, and starting state fixed. A skill helps when it improves a useful outcome or reduces the work needed to reach it.
Oqoqo provides agent usability testing and agent observability for this comparison: repeated product tasks, named toolsets, recorded trajectories, requirement verdicts, and resource use. This guide explains what to measure and how to interpret the evidence, using our published experiments.
A documentation-content comparison with an unchanged skill wrapper
Our 32-attempt SDK documentation pilot keeps the SKILL.md wrapper identical and changes only its reference content. This separates the supplied documentation treatment from a wrapper-description change. The pilot does not establish a success uplift; it illustrates why task checks, model grades, tokens and agent work time should be reported separately. The executable starter prepares independent workspaces using that design.
What should a skill evaluation measure?
Treat installation, activation, and usefulness as separate questions. An installed skill may never load. A loaded skill may give the wrong advice. A correct answer may come from a workaround that never uses your product.
| Question | Measurement | Evidence to retain |
|---|---|---|
| Was the skill available? | Installed version and startup visibility | Skill files, content hash, and exposed metadata |
| Did it activate when relevant? | Relevant tasks with observed activation / relevant tasks with activation evidence | Skill-read or invocation event, with the harness's visibility limits |
| Did it activate unnecessarily? | Unrelated tasks with observed activation / unrelated tasks with activation evidence | Near-match prompts and the corresponding trajectory |
| Did the work succeed? | Fully passing attempts / evaluated attempts | Independent outcome checks and requirement verdicts |
| Was your product used? | Required product operations actually observed | Tool results, executed integration, or state read-back |
| What did it cost? | Tokens, cost, calls, and elapsed time per attempt | Complete attempts, including failures and retries |
Mark activation unknown when the runtime does not expose enough evidence. A skill's name appearing in startup context does not establish that its body was read. Keep blocked and unevaluated attempts visible, with their own counts.
What our experiments show
A published skill case reduced exploration cost
Our skills analysis reports a MongoDB/Codex condition using roughly 27% fewer tokens and finishing 16% faster than its baseline. Other product and skill conditions in that analysis added reading overhead without proportional correctness gains.
Those are historical findings attributed to that study. The article does not provide a complete per-attempt ledger for that comparison, so these figures should not be treated as a reproducible estimate of the savings your skill will produce. Use them as a reason to measure both quality and effort.
A skill treatment did not improve every task suite
The public RunType experiment crossed 13 tasks, five setups, two agent configurations, and two trials per combination: 260 attempts. Here are the recorded passes for its raw-agent and skills setups:
| RunType setup | Claude Code passes | Codex passes |
|---|---|---|
| Raw agent baseline | 20 / 26 | 21 / 26 |
| Skills treatment | 18 / 26 | 18 / 26 |
Snapshot version 3 used Claude Code 2.1.283 with Opus 5 and Codex 0.158.0 with gpt-5.6-sol; effort was unspecified. These are descriptive results across the 13-task suite, with only two repeats per task and configuration. They compare the supplied setups as a whole. They do not isolate skill wording, establish statistical significance, or prove the skill caused each failure.
Task-level inspection changes the interpretation. On the ticket-triage task, both configurations passed both trials in the skills condition. A single aggregate hides that useful task-specific result.
The lesson is to inspect which workflows improved, which regressed, whether the skill was encountered, and what happened during the attempt. A content audit alone cannot answer those questions.
How to run a controlled SKILL.md evaluation
1. Choose tasks before editing the skill
Use real support questions, integration jobs, and recurring mistakes. Include ordinary success paths, recovery from an error, and prompts close enough to the topic that a generic description might trigger unnecessarily. Keep expected results and grading fixtures outside the agent's editable workspace.
For example, a database skill might need to help with connection reuse and a query-plan problem, while staying out of an unrelated CSV-formatting task. Write the task as a customer would ask it. A prompt that explicitly names the skill measures instructed use; a natural prompt measures discovery as well.
2. Freeze the comparison
Start with a no-skill baseline and the current skill. Keep product access equivalent: the same SDK or CLI, account permissions, credentials, network access, documentation availability, and fixtures. Record the skill version and the agent harness, model, effort, and budget.
When testing a description change, keep the skill body fixed. When testing the body, keep the description and exposure fixed. You can evaluate a complete revised package, but attribute the result to that package rather than to one sentence inside it.
3. Use fresh attempts and repeat both conditions
Give every attempt a clean workspace and fresh agent session. Reset external state or use a separate namespace. Interleave conditions to reduce the chance that a temporary service problem affects only one side. Record timeouts and infrastructure failures explicitly.
A proposed diagnostic pilot is four tasks, two conditions, two agent configurations, and three repeats: 48 attempts. If two tasks are relevant and two are negative controls, report activation on those groups separately. Three repeats can reveal failure patterns; precision and release confidence may require more data.
The Agent Skills evaluation guidance also recommends comparing skill and baseline behavior in fresh contexts and retaining timing and token evidence.
4. Verify the work independently
For generated code, execute it against fixtures or an isolated product instance. For live actions, read back the resulting state through a trusted check. For knowledge tasks, check the required facts against a versioned source. Keep successful product participation separate from a plausible final message.
Review ambiguous verdicts and a sample of passing attempts. Our SDK usability guide shows why a created object, valid labels, and a fully correct workflow are different requirements.
5. Explain the difference through the trajectory
Inspect the earliest point where the conditions diverge. Did the skill direct the agent to the correct operation? Did it prevent repeated exploration? Did a stale instruction create an unnecessary call? Did the agent read the reference but ignore its important constraint?
This is where agent observability makes an evaluation actionable. Preserve the prompt, skill-read evidence, relevant calls, error text, recovery, and outcome check together.
How to compare tokens and latency fairly
Report input, output, and cached token categories according to the provider's accounting; do not add cached tokens twice. Separate model inference cost from platform charges. Define whether time means agent work, total elapsed attempt time, or tool latency.
Compare the same task and agent configuration first. A fast failure must remain a failure in the quality report. Include costs from failed and retried attempts, then optionally report the distribution among successful attempts as a separate view.
For a fixed batch, a useful operational measure is total model cost across all attempts / independently verified successes. If there are no successes, the value is undefined. Pair this with pass counts and elapsed-time distributions so a cheaper but less reliable setup does not look like a win.
Attribute savings to a skill revision only after a controlled comparison. A long recovery sequence in one trajectory identifies a candidate fix; its full duration is not automatically avoidable overhead. Our Vercel MCP file-ID analysis illustrates that distinction.
Download a skill evaluation template
Use the skill evaluation plan (JSON) and blank attempt ledger (CSV). The plan describes the proposed 48-attempt pilot; it is a planning template, not a launcher configuration or completed result.
For each attempt, retain task and condition IDs, agent configuration, fixture and skill versions, activation evidence, verified outcome, token categories, elapsed time, and links to the trajectory and verifier. Leave unavailable measurements empty and record why.
Run skill evaluations in Oqoqo
In Oqoqo, define versioned tasks and their requirements, prepare the machine and fixtures, and create treatments for the baseline and skill conditions. A treatment is the toolset available to the agent. Keep the underlying product tools the same when isolating the effect of a skill.
Compare repeated attempts, inspect requirement verdicts, and follow trajectories to explain the outcome. Check what the selected runtime exposes before interpreting skill-engagement evidence. Oqoqo's experiment workflow and grading documentation describe the setup and assessment model; its public RunType report shows an actual comparison.
Bring a workflow that agents struggle with and a specific skill revision to discuss an Oqoqo evaluation.
Common questions
Does a shorter SKILL.md always use fewer tokens?
Measure the entire attempt. A shorter file can omit a constraint and cause more exploration or retries. A longer file can pay for itself when it guides the agent to a reliable path early. File length alone cannot establish the outcome.
Should we force the agent to invoke the skill?
Use forced invocation as a separate diagnostic condition. It can test the body when activation is the suspected problem. Keep natural activation as the primary condition if customers will rely on the agent choosing the skill itself.
Can a skill help one workflow while hurting the aggregate?
Yes. The RunType example above shows successful ticket-triage attempts inside a skills condition whose suite-level pass count was below its raw baseline. Keep task-level results alongside the aggregate and repeat the workflows that drive your decision.