Are first party harnesses any better for models than open ones?
Recently a lot of model providers have been talking about how the harness and model cannot be separated and can only be evaluated together, which is true. But it somewhat implied that the best harness for any given model is the first party option since it was built for that model specifically. This makes sense theoretically, but given the wide variety of harnesses available today, it bears to question this vendor lock-in. To understand how this translates for everyday tasks, I took 20 random coding tasks, mostly repurposed from existing public benchmarks and ran them against three harnesses with the same model configuration: OpenCode, Claude Code, and Pi all running Opus 5 on high effort against the same instructions and rubric.
- OpenCode and Claude Code tied at 67.5% pass rate. Pi was at 62.5%.
- The first-party harness (Claude Code) had no overall performance advantage.
- Claude Code used ~2x the tokens of OpenCode and ~3.5x those of Pi for the same/comparable results.
- Claude Code cost ~60% more than OpenCode while delivering the same pass rate.
- Pi was the fastest and most token-efficient in every task category although lagged behind slightly on pass rate.
- Claude Code had far fewer workspace/environment issues (its clearest first-party advantage) but that didn’t translate into more passed runs.
- The best harness changed by task type: Claude Code led on API & networking related tasks, OpenCode led on data, state & background jobs, OpenCode and Pi tied on tooling & configuration.
