Harness economics · OpenBench
Same model. Different harness bill.
Four coding harnesses ran GPT-5.6 Sol on the same 15-task benchmark. Solvency reprices each harness’s proxy-measured token usage at the model’s current verified API rates.
OpenBench-only basis · never mixed with the general model leaderboard
Matched pass rate · different usage
Codex costs 3.77× Pi per solved task at the same 72.7% pass rate.
$1.37 vs $0.363 per solved task · point estimates, not a claim of statistical superiority
Lowest point estimate
Pi
$0.264 / attempt · $0.363 / solved
Price basis
Current API
$5 input · $0.50 cache read · $30 output per million tokens
Source: OpenBench harness benchmark (MIT) · verified 2026-08-23 · GPT-5.6 Sol prices verified 2026-08-21; source usage repriced, not an observed dollar bill
One model · one benchmark
Harness cost comparison
Ranked by cost per solved task. Only the harness changes; lower is better. Infrastructure and rate-limited failures are excluded exactly as OpenBench specifies.
| # | Harness | Version | Solved | Pass | $ / attempt | $ / solved | vs lowest |
|---|---|---|---|---|---|---|---|
| 1 | Pi | 0.80.10 | 32/44 | 72.7% | $0.264 | $0.363 | 1.00× |
| 2 | Claude Code | 2.1.214 | 35/45 | 77.8% | $0.327 | $0.420 | 1.16× |
| 3 | Grok Build | 0.2.103 (89c3d36fb6f1) | 37/45 | 82.2% | $0.491 | $0.597 | 1.64× |
| 4 | Codex | 0.144.5 | 32/44 | 72.7% | $0.997 | $1.37 | 3.77× |
Source: OpenBench harness benchmark (MIT) · verified 2026-08-23 · source usage repriced: measured tokens × current verified model prices; no Solvency loop or tier assumption
Cost basis
What this comparison means
This is neither a provider invoice nor Solvency’s modelled task tier. It is an independently reproducible reprice of source-observed usage.
Calculation
cost_per_attempt =
mean_uncached_input × current_input_price
+ mean_cache_read × current_cached_input_price
+ mean_output × current_output_price
cost_per_solved = cost_per_attempt / solve_rateTask tier, assumed loops, cache-hit controls, retry cap and frontier-efficiency assumptions cannot move these rows. Tests enforce that invariant.
Read with care
- Each arm has only 44 or 45 countable attempts, so the correctness intervals are wide.
- OpenBench reports subscription-backed access; Solvency applies current API token rates for comparability, not the subscription fee.
- Cursor and OpenCode are excluded because their usage is CLI-self-reported; Devin is excluded because its split usage is incomplete.
- The source flags Pi for a rerun after a surprising per-task swing. Treat the point estimate as provisional.