Methodology

How the number is made

Solvency publishes one number that does not appear on any pricing page: what it costs to get a task finished.

cost_per_attempt     = loops × (tokens_in × input_price + tokens_out × output_price)
cost_per_solved_task = cost_per_attempt / pass_rate

Cost bases, never mixed

Where a benchmark publishes a per-task cost it actually observed, that figure is used directly and no Solvency assumption touches it, no loop count, no per-loop token estimate, no efficiency multiplier. Where only a pass rate is published, cost comes from a loop model that is an assumption, and is labelled as one. These bases are reported separately and are never averaged. Tests assert that changing an assumption cannot move a measured row.

A third basis, source_usage_repriced, starts with complete token usage observed by an external benchmark and applies the model’s current verified input, cache-read and output prices. It also bypasses Solvency’s loop, task-tier, cache-hit and efficiency assumptions. Because it is a reprice rather than an observed dollar bill, it is never labelled measured. OpenBench rows using this basis are isolated on theharness comparison and never enter the general model leaderboard.

The frontier

The scatter on the calculator plots pass rate against cost per solved task. The frontier joins the measured models that no other measured model beats on both axes at once, cheaper and at least as likely to solve the task. Modeled points are drawn hollow and are never on the line, because a modeled cost and a measured cost are not the same kind of number.

Retry variants

Three are computed. The calculator leads with naive (cost / p): unlimited independent retries.

• capped, min(1/p, K) × cost + residual × (1−p)^K; only meaningful with a real human-takeover cost set.
• truncatedGeometric, (E[N] × cost + (1−p)^K × residual) / P(solved), the rigorous truncated-geometric form.

Two proven results: with no human-takeover cost, truncatedGeometric reducesexactly to naive; and capped understatescost for low pass rates, at p = 0.2 it bills 3 attempts instead of 5 and books nothing for the 51% of tasks still unsolved, which makes weak models look cheap.

Assumption health

frontier_efficiency (0.6× loops on heavy tasks for frontier-class models) isdemoted to fallback. It has no published source, and it applies only to rows with no measured cost. Untick it in the calculator's Assumptions drawer to see unadjusted results. Solvency's own measured runs will replace it outright.

Task tiers (light / moderate / heavy) are modelling assumptions inherited from a public cost calculator that states they are not industry-standard benchmarks; Solvency inherits that caveat verbatim. They move modeled rows only.

Rules

• Prices are verified against the provider's own pricing page on the date recorded. Prices recalled from memory are never used.
• Missing is missing. A number that does not exist is reported as missing, never estimated, and a missing cache price is never treated as zero.
• Every modeled parameter is labelled an assumption, carries its provenance, and is adjustable.
• Benchmark figures are third-party, cited and linked, and are not redistributed.
• Measured rows are harness + model pairs, not properties of the bare model; the harness travels with every row.
• Source-usage-repriced rows compare harnesses only within the same model and benchmark. They are never ranked against another benchmark’s rows.

What this does not measure

A METR randomised controlled trial found experienced open-source developers were19% slower using early-2025 AI tools while believing they were20% faster. Cost per solved task measures benchmark cost. It is not a productivity measure, and the two must not be conflated.