Benchmarks

What we measured about models and effort

From our own extensive testing: 820 scored runs across 40 configurations, on the same 20 Terminal-Bench 2.0 tasks (Claude Haiku 4.5 ran twice). A configuration is one model at one reasoning-effort level, for example Claude Opus 5.5 at medium. Here is how they rank, what they cost, and what effort level is worth.

scored runs on the same 20 tasks (Claude Haiku 4.5 ran twice)
820scored runs on the same 20 tasks (Claude Haiku 4.5 ran twice)
configurations: one model at one effort level
40configurations: one model at one effort level
models scored worst or tied-worst at max effort
5 of 7models scored worst or tied-worst at max effort

Why we run our own tests

Picking the right model for a prompt means knowing how each model really behaves at every effort level: what it solves, what it costs, how long it takes, and where it runs out of time. Published scores rarely answer that, so we measure it ourselves. Findings like these inform the Efficient Router. The same attention to detail, and the same in-depth understanding of the systems underneath, goes into every part of OptiFlume: we take the time to get it right.

01

Leaderboard

Each row is one configuration. Its rating comes from comparing configurations task by task: on each task, a configuration that solved it beats one that failed it. The average configuration rates 1000.

Two configurations can solve the same number of tasks and still rate differently, because the rating looks at which tasks they solved. Solving a hard task (one that most configurations failed) earns more than solving an easy one (one that most of them passed).

With 20 tasks the 95% intervals are wide, and neighbouring rows usually overlap, so read the order in groups rather than as an exact ranking. Claude Sonnet 5.5 at low effort rates highest (1310), at $0.13 per task.

Leaderboard of 40 configurations (model and effort level) by Bradley–Terry rating
RankModelRatingSolved
1Claude Sonnet 5.5Low13101134–190116/20$0.13
2Claude Opus 5.5High12461042–197516/20$0.62
3Claude Sonnet 5.5Extra high12441030–193116/20$0.61
4Claude Opus 5High12441030–193116/20$1.86
5Claude Opus 5.5Medium12361025–195016/20$0.46
6Claude Sonnet 5.5High12241009–190316/20$0.19
7Claude Fable 5.1Low1168991–148615/20$0.81
8GPT-5.6 LunaExtra high1144933–154815/20$0.040
9Claude Sonnet 5.5Medium1142926–149715/20$0.14
10Claude Opus 5Medium1103840–156015/20$0.81
11Claude Opus 4.5Default1070813–134014/20$1.30
12Grok 4.6Low1058867–131914/20$0.20
13Claude Fable 5.1Medium1058865–130414/20$0.98
14GPT-5.6 LunaHigh1056826–134714/20$0.045
15Claude Opus 5Extra high1049829–131914/20$2.23
On the cost frontier

Tests passed: a task counts as solved only when all of its tests pass. This column shows how close a configuration got overall: the share of each task’s tests it passed, averaged over the 20 tasks. The rating uses solved or failed only. Turns: median model calls per run. Cost per task: every token (uncached input, cache writes, cache reads and output) at list prices.

02

Rating against cost

Each dot is a configuration. The frontier joins the configurations that no cheaper configuration matches: GPT-5.6 Luna at extra high effort (1144 at $0.040) and Claude Sonnet 5.5 at low effort (1310 at $0.13). Every other configuration costs more than one of these and rates lower.

  • Sonnet 5.5
  • Sonnet 4.6
  • Opus 5.5
  • Fable 5.1
  • Opus 5
  • GPT-5.6 Luna
  • Sonnet 5
  • Opus 4.6
  • One effort level only: Opus 4.5, Grok 4.6, Sonnet 4.5, Haiku 4.5, Sonnet 4
50060070080090010001100120013001400$0.03$0.10$0.30$1$3Cost per task (log scale)RatingSonnet 5.5 · LGPT-5.6 Luna · XHOpus 5.5 · HSonnet 5.5 · XHOpus 5 · HOpus 5.5 · MFable 5.1 · LOpus 5 · MSonnet 5 · MOpus 4.6 · LSonnet 4.6 · LOpus 4.5Grok 4.6 · LHaiku 4.5Sonnet 4Sonnet 5.5 · MGPT-5.6 Luna · HOpus 5 · XHFable 5.1 · HOpus 5.5 · LOpus 4.6 · MaxSonnet 5 · LFable 5.1 · MaxOpus 4.6 · HSonnet 4.6 · Max

Tap a point to see its model, effort, rating, tasks solved and cost.

Frontier: nothing cheaper rates as highL, M, H, XH: low, medium, high, extra high effort. Lines join one model’s efforts, low to max.Hover, tap or tab to a point for its numbers.Swipe sideways for the most expensive configurations.

Measurements like these are part of how the Efficient Router chooses a model and effort for each prompt.

03

How the rating works

We rate every configuration with a Bradley–Terry model, the method used by public AI leaderboards. Three formulas do the work.

P(i beats j)=pipi+pj

Each configuration i has a strength p. On a task that one of two configurations solved and the other failed, this is the chance that i is the one that solved it.

Ri=1000+400⁢log10⁡pi

The strengths are the ones that best explain every task result (maximum likelihood). They are put on a familiar scale where the average configuration is 1000.

P(i beats j)=11+10(Rj−Ri)/400

Read back from two ratings: a 100-point lead means winning about 64% of the tasks that separate them; 200 points, about 76%.

When both configurations solved a task, or both failed it, that task says nothing about which is better and is left out. Each configuration also gets one imaginary draw against an average opponent, so one that never lost (or never won) still gets a finite rating. The 95% interval comes from refitting 1,000 times on random resamples of the 20 tasks.

Every rating with its interval

30050070090011001300150017001900average 1000Claude Sonnet 5.5, low effort: 1310 (95% 1134–1901)1. Sonnet 5.5 · LowClaude Opus 5.5, high effort: 1246 (95% 1042–1975)2. Opus 5.5 · HighClaude Sonnet 5.5, extra high effort: 1244 (95% 1030–1931)3. Sonnet 5.5 · Extra highClaude Opus 5, high effort: 1244 (95% 1030–1931)4. Opus 5 · HighClaude Opus 5.5, medium effort: 1236 (95% 1025–1950)5. Opus 5.5 · MediumClaude Sonnet 5.5, high effort: 1224 (95% 1009–1903)6. Sonnet 5.5 · HighClaude Fable 5.1, low effort: 1168 (95% 991–1486)7. Fable 5.1 · LowGPT-5.6 Luna, extra high effort: 1144 (95% 933–1548)8. GPT-5.6 Luna · Extra highClaude Sonnet 5.5, medium effort: 1142 (95% 926–1497)9. Sonnet 5.5 · MediumClaude Opus 5, medium effort: 1103 (95% 840–1560)10. Opus 5 · MediumClaude Opus 4.5, default effort: 1070 (95% 813–1340)11. Opus 4.5 · DefaultGrok 4.6, low effort: 1058 (95% 867–1319)12. Grok 4.6 · LowClaude Fable 5.1, medium effort: 1058 (95% 865–1304)13. Fable 5.1 · MediumGPT-5.6 Luna, high effort: 1056 (95% 826–1347)14. GPT-5.6 Luna · HighClaude Opus 5, extra high effort: 1049 (95% 829–1319)15. Opus 5 · Extra highClaude Sonnet 5, medium effort: 1048 (95% 835–1355)16. Sonnet 5 · MediumClaude Opus 5, low effort: 1031 (95% 803–1410)17. Opus 5 · LowClaude Opus 4.6, low effort: 988 (95% 773–1248)18. Opus 4.6 · LowClaude Sonnet 5, max effort: 985 (95% 739–1246)19. Sonnet 5 · MaxClaude Fable 5.1, high effort: 981 (95% 797–1199)20. Fable 5.1 · HighClaude Fable 5.1, extra high effort: 981 (95% 797–1199)21. Fable 5.1 · Extra highClaude Sonnet 5, extra high effort: 975 (95% 775–1185)22. Sonnet 5 · Extra highClaude Opus 5.5, low effort: 973 (95% 766–1200)23. Opus 5.5 · LowClaude Opus 5, max effort: 971 (95% 737–1232)24. Opus 5 · MaxClaude Sonnet 4.6, high effort: 937 (95% 647–1165)25. Sonnet 4.6 · HighClaude Sonnet 5, high effort: 935 (95% 641–1169)26. Sonnet 5 · HighClaude Opus 4.6, max effort: 932 (95% 667–1154)27. Opus 4.6 · MaxClaude Opus 4.6, medium effort: 928 (95% 670–1151)28. Opus 4.6 · MediumClaude Sonnet 4.6, medium effort: 910 (95% 681–1098)29. Sonnet 4.6 · MediumClaude Sonnet 4.6, low effort: 906 (95% 675–1126)30. Sonnet 4.6 · LowClaude Sonnet 4.5, default effort: 904 (95% 594–1123)31. Sonnet 4.5 · DefaultClaude Opus 5.5, extra high effort: 903 (95% 690–1077)32. Opus 5.5 · Extra highClaude Sonnet 5.5, max effort: 897 (95% 695–1076)33. Sonnet 5.5 · MaxClaude Opus 5.5, max effort: 886 (95% 618–1091)34. Opus 5.5 · MaxClaude Sonnet 5, low effort: 845 (95% 593–1042)35. Sonnet 5 · LowClaude Fable 5.1, max effort: 843 (95% 587–1030)36. Fable 5.1 · MaxClaude Opus 4.6, high effort: 828 (95% 593–992)37. Opus 4.6 · HighClaude Haiku 4.5, default effort: 743 (95% 496–894)38. Haiku 4.5 · DefaultClaude Sonnet 4, default effort: 690 (95% 376–856)39. Sonnet 4 · DefaultClaude Sonnet 4.6, max effort: 629 (95% 380–780)40. Sonnet 4.6 · MaxRating (dot) and 95% interval (line)
Accent dots are on the cost frontier. Long lines mean few decisive tasks: the rating could sit anywhere along them.

Head to head: the top 12

The chance that the configuration in the row beats the one in the column on a task only one of them solves, from the third formula.

Sonnet 5.5 · LowOpus 5.5 · HighSonnet 5.5 · XHOpus 5 · HighOpus 5.5 · MediumSonnet 5.5 · HighFable 5.1 · LowGPT-5.6 Luna · XHSonnet 5.5 · MediumOpus 5 · MediumOpus 4.5 · DefaultGrok 4.6 · LowSonnet 5.5 · LowSonnet 5.5 · LowSonnet 5.5 · Low beats Opus 5.5 · High: 59%59Sonnet 5.5 · Low beats Sonnet 5.5 · XH: 59%59Sonnet 5.5 · Low beats Opus 5 · High: 59%59Sonnet 5.5 · Low beats Opus 5.5 · Medium: 60%60Sonnet 5.5 · Low beats Sonnet 5.5 · High: 62%62Sonnet 5.5 · Low beats Fable 5.1 · Low: 69%69Sonnet 5.5 · Low beats GPT-5.6 Luna · XH: 72%72Sonnet 5.5 · Low beats Sonnet 5.5 · Medium: 72%72Sonnet 5.5 · Low beats Opus 5 · Medium: 77%77Sonnet 5.5 · Low beats Opus 4.5 · Default: 80%80Sonnet 5.5 · Low beats Grok 4.6 · Low: 81%81Opus 5.5 · HighOpus 5.5 · High beats Sonnet 5.5 · Low: 41%41Opus 5.5 · HighOpus 5.5 · High beats Sonnet 5.5 · XH: 50%50Opus 5.5 · High beats Opus 5 · High: 50%50Opus 5.5 · High beats Opus 5.5 · Medium: 51%51Opus 5.5 · High beats Sonnet 5.5 · High: 53%53Opus 5.5 · High beats Fable 5.1 · Low: 61%61Opus 5.5 · High beats GPT-5.6 Luna · XH: 64%64Opus 5.5 · High beats Sonnet 5.5 · Medium: 65%65Opus 5.5 · High beats Opus 5 · Medium: 69%69Opus 5.5 · High beats Opus 4.5 · Default: 73%73Opus 5.5 · High beats Grok 4.6 · Low: 75%75Sonnet 5.5 · XHSonnet 5.5 · XH beats Sonnet 5.5 · Low: 41%41Sonnet 5.5 · XH beats Opus 5.5 · High: 50%50Sonnet 5.5 · XHSonnet 5.5 · XH beats Opus 5 · High: 50%50Sonnet 5.5 · XH beats Opus 5.5 · Medium: 51%51Sonnet 5.5 · XH beats Sonnet 5.5 · High: 53%53Sonnet 5.5 · XH beats Fable 5.1 · Low: 61%61Sonnet 5.5 · XH beats GPT-5.6 Luna · XH: 64%64Sonnet 5.5 · XH beats Sonnet 5.5 · Medium: 64%64Sonnet 5.5 · XH beats Opus 5 · Medium: 69%69Sonnet 5.5 · XH beats Opus 4.5 · Default: 73%73Sonnet 5.5 · XH beats Grok 4.6 · Low: 74%74Opus 5 · HighOpus 5 · High beats Sonnet 5.5 · Low: 41%41Opus 5 · High beats Opus 5.5 · High: 50%50Opus 5 · High beats Sonnet 5.5 · XH: 50%50Opus 5 · HighOpus 5 · High beats Opus 5.5 · Medium: 51%51Opus 5 · High beats Sonnet 5.5 · High: 53%53Opus 5 · High beats Fable 5.1 · Low: 61%61Opus 5 · High beats GPT-5.6 Luna · XH: 64%64Opus 5 · High beats Sonnet 5.5 · Medium: 64%64Opus 5 · High beats Opus 5 · Medium: 69%69Opus 5 · High beats Opus 4.5 · Default: 73%73Opus 5 · High beats Grok 4.6 · Low: 74%74Opus 5.5 · MediumOpus 5.5 · Medium beats Sonnet 5.5 · Low: 40%40Opus 5.5 · Medium beats Opus 5.5 · High: 49%49Opus 5.5 · Medium beats Sonnet 5.5 · XH: 49%49Opus 5.5 · Medium beats Opus 5 · High: 49%49Opus 5.5 · MediumOpus 5.5 · Medium beats Sonnet 5.5 · High: 52%52Opus 5.5 · Medium beats Fable 5.1 · Low: 60%60Opus 5.5 · Medium beats GPT-5.6 Luna · XH: 63%63Opus 5.5 · Medium beats Sonnet 5.5 · Medium: 63%63Opus 5.5 · Medium beats Opus 5 · Medium: 68%68Opus 5.5 · Medium beats Opus 4.5 · Default: 72%72Opus 5.5 · Medium beats Grok 4.6 · Low: 74%74Sonnet 5.5 · HighSonnet 5.5 · High beats Sonnet 5.5 · Low: 38%38Sonnet 5.5 · High beats Opus 5.5 · High: 47%47Sonnet 5.5 · High beats Sonnet 5.5 · XH: 47%47Sonnet 5.5 · High beats Opus 5 · High: 47%47Sonnet 5.5 · High beats Opus 5.5 · Medium: 48%48Sonnet 5.5 · HighSonnet 5.5 · High beats Fable 5.1 · Low: 58%58Sonnet 5.5 · High beats GPT-5.6 Luna · XH: 61%61Sonnet 5.5 · High beats Sonnet 5.5 · Medium: 62%62Sonnet 5.5 · High beats Opus 5 · Medium: 67%67Sonnet 5.5 · High beats Opus 4.5 · Default: 71%71Sonnet 5.5 · High beats Grok 4.6 · Low: 72%72Fable 5.1 · LowFable 5.1 · Low beats Sonnet 5.5 · Low: 31%31Fable 5.1 · Low beats Opus 5.5 · High: 39%39Fable 5.1 · Low beats Sonnet 5.5 · XH: 39%39Fable 5.1 · Low beats Opus 5 · High: 39%39Fable 5.1 · Low beats Opus 5.5 · Medium: 40%40Fable 5.1 · Low beats Sonnet 5.5 · High: 42%42Fable 5.1 · LowFable 5.1 · Low beats GPT-5.6 Luna · XH: 53%53Fable 5.1 · Low beats Sonnet 5.5 · Medium: 54%54Fable 5.1 · Low beats Opus 5 · Medium: 59%59Fable 5.1 · Low beats Opus 4.5 · Default: 64%64Fable 5.1 · Low beats Grok 4.6 · Low: 65%65GPT-5.6 Luna · XHGPT-5.6 Luna · XH beats Sonnet 5.5 · Low: 28%28GPT-5.6 Luna · XH beats Opus 5.5 · High: 36%36GPT-5.6 Luna · XH beats Sonnet 5.5 · XH: 36%36GPT-5.6 Luna · XH beats Opus 5 · High: 36%36GPT-5.6 Luna · XH beats Opus 5.5 · Medium: 37%37GPT-5.6 Luna · XH beats Sonnet 5.5 · High: 39%39GPT-5.6 Luna · XH beats Fable 5.1 · Low: 47%47GPT-5.6 Luna · XHGPT-5.6 Luna · XH beats Sonnet 5.5 · Medium: 50%50GPT-5.6 Luna · XH beats Opus 5 · Medium: 56%56GPT-5.6 Luna · XH beats Opus 4.5 · Default: 60%60GPT-5.6 Luna · XH beats Grok 4.6 · Low: 62%62Sonnet 5.5 · MediumSonnet 5.5 · Medium beats Sonnet 5.5 · Low: 28%28Sonnet 5.5 · Medium beats Opus 5.5 · High: 35%35Sonnet 5.5 · Medium beats Sonnet 5.5 · XH: 36%36Sonnet 5.5 · Medium beats Opus 5 · High: 36%36Sonnet 5.5 · Medium beats Opus 5.5 · Medium: 37%37Sonnet 5.5 · Medium beats Sonnet 5.5 · High: 38%38Sonnet 5.5 · Medium beats Fable 5.1 · Low: 46%46Sonnet 5.5 · Medium beats GPT-5.6 Luna · XH: 50%50Sonnet 5.5 · MediumSonnet 5.5 · Medium beats Opus 5 · Medium: 56%56Sonnet 5.5 · Medium beats Opus 4.5 · Default: 60%60Sonnet 5.5 · Medium beats Grok 4.6 · Low: 62%62Opus 5 · MediumOpus 5 · Medium beats Sonnet 5.5 · Low: 23%23Opus 5 · Medium beats Opus 5.5 · High: 31%31Opus 5 · Medium beats Sonnet 5.5 · XH: 31%31Opus 5 · Medium beats Opus 5 · High: 31%31Opus 5 · Medium beats Opus 5.5 · Medium: 32%32Opus 5 · Medium beats Sonnet 5.5 · High: 33%33Opus 5 · Medium beats Fable 5.1 · Low: 41%41Opus 5 · Medium beats GPT-5.6 Luna · XH: 44%44Opus 5 · Medium beats Sonnet 5.5 · Medium: 44%44Opus 5 · MediumOpus 5 · Medium beats Opus 4.5 · Default: 55%55Opus 5 · Medium beats Grok 4.6 · Low: 56%56Opus 4.5 · DefaultOpus 4.5 · Default beats Sonnet 5.5 · Low: 20%20Opus 4.5 · Default beats Opus 5.5 · High: 27%27Opus 4.5 · Default beats Sonnet 5.5 · XH: 27%27Opus 4.5 · Default beats Opus 5 · High: 27%27Opus 4.5 · Default beats Opus 5.5 · Medium: 28%28Opus 4.5 · Default beats Sonnet 5.5 · High: 29%29Opus 4.5 · Default beats Fable 5.1 · Low: 36%36Opus 4.5 · Default beats GPT-5.6 Luna · XH: 40%40Opus 4.5 · Default beats Sonnet 5.5 · Medium: 40%40Opus 4.5 · Default beats Opus 5 · Medium: 45%45Opus 4.5 · DefaultOpus 4.5 · Default beats Grok 4.6 · Low: 52%52Grok 4.6 · LowGrok 4.6 · Low beats Sonnet 5.5 · Low: 19%19Grok 4.6 · Low beats Opus 5.5 · High: 25%25Grok 4.6 · Low beats Sonnet 5.5 · XH: 26%26Grok 4.6 · Low beats Opus 5 · High: 26%26Grok 4.6 · Low beats Opus 5.5 · Medium: 26%26Grok 4.6 · Low beats Sonnet 5.5 · High: 28%28Grok 4.6 · Low beats Fable 5.1 · Low: 35%35Grok 4.6 · Low beats GPT-5.6 Luna · XH: 38%38Grok 4.6 · Low beats Sonnet 5.5 · Medium: 38%38Grok 4.6 · Low beats Opus 5 · Medium: 44%44Grok 4.6 · Low beats Opus 4.5 · Default: 48%48Grok 4.6 · Low
Per cent chance the row beats the column on a task only one of them solves.row favouredcolumn favoured

04

Which tasks each configuration solved

Every run, as solved or failed. Tasks run from easiest (solved by most configurations) to hardest. This is what the rating is built from, and it shows why two rows with the same count can rate differently: one of them solved harder tasks.

git-leak-recovery40pypi-server40modernize-scientific-stack39feal-differential-cryptanalysis37fix-git37kv-store-grpc37password-recovery36headless-terminal34llm-inference-batching-scheduler34mteb-leaderboard30schemelike-metacircular-eval26regex-chess25query-optimize24pytorch-model-cli22cancel-async-tasks21dna-assembly20adaptive-rejection-sampler9caffe-cifar-106train-fasttext5filter-js-from-html01. Sonnet 5.5 · LowSonnet 5.5 · Low on git-leak-recovery: solvedSonnet 5.5 · Low on pypi-server: solvedSonnet 5.5 · Low on modernize-scientific-stack: solvedSonnet 5.5 · Low on feal-differential-cryptanalysis: solvedSonnet 5.5 · Low on fix-git: solvedSonnet 5.5 · Low on kv-store-grpc: solvedSonnet 5.5 · Low on password-recovery: solvedSonnet 5.5 · Low on headless-terminal: solvedSonnet 5.5 · Low on llm-inference-batching-scheduler: solvedSonnet 5.5 · Low on mteb-leaderboard: solvedSonnet 5.5 · Low on schemelike-metacircular-eval: solvedSonnet 5.5 · Low on regex-chess: solvedSonnet 5.5 · Low on query-optimize: solvedSonnet 5.5 · Low on pytorch-model-cli: solvedSonnet 5.5 · Low on cancel-async-tasks: solvedSonnet 5.5 · Low on dna-assembly: solvedSonnet 5.5 · Low on adaptive-rejection-sampler: failedSonnet 5.5 · Low on caffe-cifar-10: failedSonnet 5.5 · Low on train-fasttext: failedSonnet 5.5 · Low on filter-js-from-html: failed162. Opus 5.5 · HighOpus 5.5 · High on git-leak-recovery: solvedOpus 5.5 · High on pypi-server: solvedOpus 5.5 · High on modernize-scientific-stack: solvedOpus 5.5 · High on feal-differential-cryptanalysis: solvedOpus 5.5 · High on fix-git: solvedOpus 5.5 · High on kv-store-grpc: solvedOpus 5.5 · High on password-recovery: solvedOpus 5.5 · High on headless-terminal: solvedOpus 5.5 · High on llm-inference-batching-scheduler: solvedOpus 5.5 · High on mteb-leaderboard: solvedOpus 5.5 · High on schemelike-metacircular-eval: solvedOpus 5.5 · High on regex-chess: solvedOpus 5.5 · High on query-optimize: solvedOpus 5.5 · High on pytorch-model-cli: solvedOpus 5.5 · High on cancel-async-tasks: solvedOpus 5.5 · High on dna-assembly: failedOpus 5.5 · High on adaptive-rejection-sampler: failedOpus 5.5 · High on caffe-cifar-10: failedOpus 5.5 · High on train-fasttext: solvedOpus 5.5 · High on filter-js-from-html: failed163. Sonnet 5.5 · Extra highSonnet 5.5 · Extra high on git-leak-recovery: solvedSonnet 5.5 · Extra high on pypi-server: solvedSonnet 5.5 · Extra high on modernize-scientific-stack: solvedSonnet 5.5 · Extra high on feal-differential-cryptanalysis: solvedSonnet 5.5 · Extra high on fix-git: solvedSonnet 5.5 · Extra high on kv-store-grpc: solvedSonnet 5.5 · Extra high on password-recovery: solvedSonnet 5.5 · Extra high on headless-terminal: solvedSonnet 5.5 · Extra high on llm-inference-batching-scheduler: solvedSonnet 5.5 · Extra high on mteb-leaderboard: solvedSonnet 5.5 · Extra high on schemelike-metacircular-eval: solvedSonnet 5.5 · Extra high on regex-chess: solvedSonnet 5.5 · Extra high on query-optimize: solvedSonnet 5.5 · Extra high on pytorch-model-cli: failedSonnet 5.5 · Extra high on cancel-async-tasks: solvedSonnet 5.5 · Extra high on dna-assembly: solvedSonnet 5.5 · Extra high on adaptive-rejection-sampler: solvedSonnet 5.5 · Extra high on caffe-cifar-10: failedSonnet 5.5 · Extra high on train-fasttext: failedSonnet 5.5 · Extra high on filter-js-from-html: failed164. Opus 5 · HighOpus 5 · High on git-leak-recovery: solvedOpus 5 · High on pypi-server: solvedOpus 5 · High on modernize-scientific-stack: solvedOpus 5 · High on feal-differential-cryptanalysis: solvedOpus 5 · High on fix-git: solvedOpus 5 · High on kv-store-grpc: solvedOpus 5 · High on password-recovery: solvedOpus 5 · High on headless-terminal: solvedOpus 5 · High on llm-inference-batching-scheduler: solvedOpus 5 · High on mteb-leaderboard: solvedOpus 5 · High on schemelike-metacircular-eval: solvedOpus 5 · High on regex-chess: solvedOpus 5 · High on query-optimize: solvedOpus 5 · High on pytorch-model-cli: failedOpus 5 · High on cancel-async-tasks: solvedOpus 5 · High on dna-assembly: solvedOpus 5 · High on adaptive-rejection-sampler: solvedOpus 5 · High on caffe-cifar-10: failedOpus 5 · High on train-fasttext: failedOpus 5 · High on filter-js-from-html: failed165. Opus 5.5 · MediumOpus 5.5 · Medium on git-leak-recovery: solvedOpus 5.5 · Medium on pypi-server: solvedOpus 5.5 · Medium on modernize-scientific-stack: solvedOpus 5.5 · Medium on feal-differential-cryptanalysis: solvedOpus 5.5 · Medium on fix-git: solvedOpus 5.5 · Medium on kv-store-grpc: solvedOpus 5.5 · Medium on password-recovery: solvedOpus 5.5 · Medium on headless-terminal: solvedOpus 5.5 · Medium on llm-inference-batching-scheduler: solvedOpus 5.5 · Medium on mteb-leaderboard: solvedOpus 5.5 · Medium on schemelike-metacircular-eval: solvedOpus 5.5 · Medium on regex-chess: solvedOpus 5.5 · Medium on query-optimize: failedOpus 5.5 · Medium on pytorch-model-cli: solvedOpus 5.5 · Medium on cancel-async-tasks: solvedOpus 5.5 · Medium on dna-assembly: solvedOpus 5.5 · Medium on adaptive-rejection-sampler: solvedOpus 5.5 · Medium on caffe-cifar-10: failedOpus 5.5 · Medium on train-fasttext: failedOpus 5.5 · Medium on filter-js-from-html: failed166. Sonnet 5.5 · HighSonnet 5.5 · High on git-leak-recovery: solvedSonnet 5.5 · High on pypi-server: solvedSonnet 5.5 · High on modernize-scientific-stack: solvedSonnet 5.5 · High on feal-differential-cryptanalysis: solvedSonnet 5.5 · High on fix-git: solvedSonnet 5.5 · High on kv-store-grpc: solvedSonnet 5.5 · High on password-recovery: solvedSonnet 5.5 · High on headless-terminal: solvedSonnet 5.5 · High on llm-inference-batching-scheduler: solvedSonnet 5.5 · High on mteb-leaderboard: solvedSonnet 5.5 · High on schemelike-metacircular-eval: solvedSonnet 5.5 · High on regex-chess: solvedSonnet 5.5 · High on query-optimize: failedSonnet 5.5 · High on pytorch-model-cli: solvedSonnet 5.5 · High on cancel-async-tasks: solvedSonnet 5.5 · High on dna-assembly: solvedSonnet 5.5 · High on adaptive-rejection-sampler: failedSonnet 5.5 · High on caffe-cifar-10: failedSonnet 5.5 · High on train-fasttext: solvedSonnet 5.5 · High on filter-js-from-html: failed167. Fable 5.1 · LowFable 5.1 · Low on git-leak-recovery: solvedFable 5.1 · Low on pypi-server: solvedFable 5.1 · Low on modernize-scientific-stack: solvedFable 5.1 · Low on feal-differential-cryptanalysis: solvedFable 5.1 · Low on fix-git: solvedFable 5.1 · Low on kv-store-grpc: solvedFable 5.1 · Low on password-recovery: solvedFable 5.1 · Low on headless-terminal: solvedFable 5.1 · Low on llm-inference-batching-scheduler: solvedFable 5.1 · Low on mteb-leaderboard: solvedFable 5.1 · Low on schemelike-metacircular-eval: solvedFable 5.1 · Low on regex-chess: solvedFable 5.1 · Low on query-optimize: solvedFable 5.1 · Low on pytorch-model-cli: solvedFable 5.1 · Low on cancel-async-tasks: solvedFable 5.1 · Low on dna-assembly: failedFable 5.1 · Low on adaptive-rejection-sampler: failedFable 5.1 · Low on caffe-cifar-10: failedFable 5.1 · Low on train-fasttext: failedFable 5.1 · Low on filter-js-from-html: failed158. GPT-5.6 Luna · Extra highGPT-5.6 Luna · Extra high on git-leak-recovery: solvedGPT-5.6 Luna · Extra high on pypi-server: solvedGPT-5.6 Luna · Extra high on modernize-scientific-stack: solvedGPT-5.6 Luna · Extra high on feal-differential-cryptanalysis: solvedGPT-5.6 Luna · Extra high on fix-git: solvedGPT-5.6 Luna · Extra high on kv-store-grpc: solvedGPT-5.6 Luna · Extra high on password-recovery: solvedGPT-5.6 Luna · Extra high on headless-terminal: solvedGPT-5.6 Luna · Extra high on llm-inference-batching-scheduler: solvedGPT-5.6 Luna · Extra high on mteb-leaderboard: solvedGPT-5.6 Luna · Extra high on schemelike-metacircular-eval: solvedGPT-5.6 Luna · Extra high on regex-chess: failedGPT-5.6 Luna · Extra high on query-optimize: solvedGPT-5.6 Luna · Extra high on pytorch-model-cli: solvedGPT-5.6 Luna · Extra high on cancel-async-tasks: failedGPT-5.6 Luna · Extra high on dna-assembly: solvedGPT-5.6 Luna · Extra high on adaptive-rejection-sampler: solvedGPT-5.6 Luna · Extra high on caffe-cifar-10: failedGPT-5.6 Luna · Extra high on train-fasttext: failedGPT-5.6 Luna · Extra high on filter-js-from-html: failed159. Sonnet 5.5 · MediumSonnet 5.5 · Medium on git-leak-recovery: solvedSonnet 5.5 · Medium on pypi-server: solvedSonnet 5.5 · Medium on modernize-scientific-stack: solvedSonnet 5.5 · Medium on feal-differential-cryptanalysis: solvedSonnet 5.5 · Medium on fix-git: solvedSonnet 5.5 · Medium on kv-store-grpc: solvedSonnet 5.5 · Medium on password-recovery: solvedSonnet 5.5 · Medium on headless-terminal: solvedSonnet 5.5 · Medium on llm-inference-batching-scheduler: solvedSonnet 5.5 · Medium on mteb-leaderboard: failedSonnet 5.5 · Medium on schemelike-metacircular-eval: solvedSonnet 5.5 · Medium on regex-chess: solvedSonnet 5.5 · Medium on query-optimize: solvedSonnet 5.5 · Medium on pytorch-model-cli: solvedSonnet 5.5 · Medium on cancel-async-tasks: solvedSonnet 5.5 · Medium on dna-assembly: solvedSonnet 5.5 · Medium on adaptive-rejection-sampler: failedSonnet 5.5 · Medium on caffe-cifar-10: failedSonnet 5.5 · Medium on train-fasttext: failedSonnet 5.5 · Medium on filter-js-from-html: failed1510. Opus 5 · MediumOpus 5 · Medium on git-leak-recovery: solvedOpus 5 · Medium on pypi-server: solvedOpus 5 · Medium on modernize-scientific-stack: solvedOpus 5 · Medium on feal-differential-cryptanalysis: solvedOpus 5 · Medium on fix-git: solvedOpus 5 · Medium on kv-store-grpc: failedOpus 5 · Medium on password-recovery: solvedOpus 5 · Medium on headless-terminal: solvedOpus 5 · Medium on llm-inference-batching-scheduler: solvedOpus 5 · Medium on mteb-leaderboard: solvedOpus 5 · Medium on schemelike-metacircular-eval: solvedOpus 5 · Medium on regex-chess: solvedOpus 5 · Medium on query-optimize: solvedOpus 5 · Medium on pytorch-model-cli: failedOpus 5 · Medium on cancel-async-tasks: solvedOpus 5 · Medium on dna-assembly: solvedOpus 5 · Medium on adaptive-rejection-sampler: failedOpus 5 · Medium on caffe-cifar-10: failedOpus 5 · Medium on train-fasttext: solvedOpus 5 · Medium on filter-js-from-html: failed1511. Opus 4.5 · DefaultOpus 4.5 · Default on git-leak-recovery: solvedOpus 4.5 · Default on pypi-server: solvedOpus 4.5 · Default on modernize-scientific-stack: solvedOpus 4.5 · Default on feal-differential-cryptanalysis: solvedOpus 4.5 · Default on fix-git: solvedOpus 4.5 · Default on kv-store-grpc: solvedOpus 4.5 · Default on password-recovery: solvedOpus 4.5 · Default on headless-terminal: solvedOpus 4.5 · Default on llm-inference-batching-scheduler: solvedOpus 4.5 · Default on mteb-leaderboard: solvedOpus 4.5 · Default on schemelike-metacircular-eval: failedOpus 4.5 · Default on regex-chess: failedOpus 4.5 · Default on query-optimize: solvedOpus 4.5 · Default on pytorch-model-cli: solvedOpus 4.5 · Default on cancel-async-tasks: failedOpus 4.5 · Default on dna-assembly: failedOpus 4.5 · Default on adaptive-rejection-sampler: solvedOpus 4.5 · Default on caffe-cifar-10: solvedOpus 4.5 · Default on train-fasttext: failedOpus 4.5 · Default on filter-js-from-html: failed1412. Grok 4.6 · LowGrok 4.6 · Low on git-leak-recovery: solvedGrok 4.6 · Low on pypi-server: solvedGrok 4.6 · Low on modernize-scientific-stack: solvedGrok 4.6 · Low on feal-differential-cryptanalysis: solvedGrok 4.6 · Low on fix-git: solvedGrok 4.6 · Low on kv-store-grpc: solvedGrok 4.6 · Low on password-recovery: solvedGrok 4.6 · Low on headless-terminal: solvedGrok 4.6 · Low on llm-inference-batching-scheduler: solvedGrok 4.6 · Low on mteb-leaderboard: solvedGrok 4.6 · Low on schemelike-metacircular-eval: solvedGrok 4.6 · Low on regex-chess: solvedGrok 4.6 · Low on query-optimize: solvedGrok 4.6 · Low on pytorch-model-cli: failedGrok 4.6 · Low on cancel-async-tasks: failedGrok 4.6 · Low on dna-assembly: solvedGrok 4.6 · Low on adaptive-rejection-sampler: failedGrok 4.6 · Low on caffe-cifar-10: failedGrok 4.6 · Low on train-fasttext: failedGrok 4.6 · Low on filter-js-from-html: failed1413. Fable 5.1 · MediumFable 5.1 · Medium on git-leak-recovery: solvedFable 5.1 · Medium on pypi-server: solvedFable 5.1 · Medium on modernize-scientific-stack: solvedFable 5.1 · Medium on feal-differential-cryptanalysis: solvedFable 5.1 · Medium on fix-git: solvedFable 5.1 · Medium on kv-store-grpc: solvedFable 5.1 · Medium on password-recovery: solvedFable 5.1 · Medium on headless-terminal: solvedFable 5.1 · Medium on llm-inference-batching-scheduler: solvedFable 5.1 · Medium on mteb-leaderboard: solvedFable 5.1 · Medium on schemelike-metacircular-eval: solvedFable 5.1 · Medium on regex-chess: solvedFable 5.1 · Medium on query-optimize: failedFable 5.1 · Medium on pytorch-model-cli: solvedFable 5.1 · Medium on cancel-async-tasks: solvedFable 5.1 · Medium on dna-assembly: failedFable 5.1 · Medium on adaptive-rejection-sampler: failedFable 5.1 · Medium on caffe-cifar-10: failedFable 5.1 · Medium on train-fasttext: failedFable 5.1 · Medium on filter-js-from-html: failed1414. GPT-5.6 Luna · HighGPT-5.6 Luna · High on git-leak-recovery: solvedGPT-5.6 Luna · High on pypi-server: solvedGPT-5.6 Luna · High on modernize-scientific-stack: solvedGPT-5.6 Luna · High on feal-differential-cryptanalysis: solvedGPT-5.6 Luna · High on fix-git: solvedGPT-5.6 Luna · High on kv-store-grpc: solvedGPT-5.6 Luna · High on password-recovery: solvedGPT-5.6 Luna · High on headless-terminal: solvedGPT-5.6 Luna · High on llm-inference-batching-scheduler: solvedGPT-5.6 Luna · High on mteb-leaderboard: solvedGPT-5.6 Luna · High on schemelike-metacircular-eval: solvedGPT-5.6 Luna · High on regex-chess: solvedGPT-5.6 Luna · High on query-optimize: failedGPT-5.6 Luna · High on pytorch-model-cli: failedGPT-5.6 Luna · High on cancel-async-tasks: failedGPT-5.6 Luna · High on dna-assembly: solvedGPT-5.6 Luna · High on adaptive-rejection-sampler: failedGPT-5.6 Luna · High on caffe-cifar-10: solvedGPT-5.6 Luna · High on train-fasttext: failedGPT-5.6 Luna · High on filter-js-from-html: failed1415. Opus 5 · Extra highOpus 5 · Extra high on git-leak-recovery: solvedOpus 5 · Extra high on pypi-server: solvedOpus 5 · Extra high on modernize-scientific-stack: solvedOpus 5 · Extra high on feal-differential-cryptanalysis: solvedOpus 5 · Extra high on fix-git: solvedOpus 5 · Extra high on kv-store-grpc: solvedOpus 5 · Extra high on password-recovery: solvedOpus 5 · Extra high on headless-terminal: solvedOpus 5 · Extra high on llm-inference-batching-scheduler: solvedOpus 5 · Extra high on mteb-leaderboard: solvedOpus 5 · Extra high on schemelike-metacircular-eval: failedOpus 5 · Extra high on regex-chess: solvedOpus 5 · Extra high on query-optimize: failedOpus 5 · Extra high on pytorch-model-cli: failedOpus 5 · Extra high on cancel-async-tasks: solvedOpus 5 · Extra high on dna-assembly: solvedOpus 5 · Extra high on adaptive-rejection-sampler: failedOpus 5 · Extra high on caffe-cifar-10: failedOpus 5 · Extra high on train-fasttext: solvedOpus 5 · Extra high on filter-js-from-html: failed1416. Sonnet 5 · MediumSonnet 5 · Medium on git-leak-recovery: solvedSonnet 5 · Medium on pypi-server: solvedSonnet 5 · Medium on modernize-scientific-stack: solvedSonnet 5 · Medium on feal-differential-cryptanalysis: solvedSonnet 5 · Medium on fix-git: solvedSonnet 5 · Medium on kv-store-grpc: solvedSonnet 5 · Medium on password-recovery: solvedSonnet 5 · Medium on headless-terminal: failedSonnet 5 · Medium on llm-inference-batching-scheduler: solvedSonnet 5 · Medium on mteb-leaderboard: solvedSonnet 5 · Medium on schemelike-metacircular-eval: solvedSonnet 5 · Medium on regex-chess: solvedSonnet 5 · Medium on query-optimize: solvedSonnet 5 · Medium on pytorch-model-cli: solvedSonnet 5 · Medium on cancel-async-tasks: failedSonnet 5 · Medium on dna-assembly: solvedSonnet 5 · Medium on adaptive-rejection-sampler: failedSonnet 5 · Medium on caffe-cifar-10: failedSonnet 5 · Medium on train-fasttext: failedSonnet 5 · Medium on filter-js-from-html: failed1417. Opus 5 · LowOpus 5 · Low on git-leak-recovery: solvedOpus 5 · Low on pypi-server: solvedOpus 5 · Low on modernize-scientific-stack: solvedOpus 5 · Low on feal-differential-cryptanalysis: solvedOpus 5 · Low on fix-git: solvedOpus 5 · Low on kv-store-grpc: failedOpus 5 · Low on password-recovery: solvedOpus 5 · Low on headless-terminal: failedOpus 5 · Low on llm-inference-batching-scheduler: solvedOpus 5 · Low on mteb-leaderboard: solvedOpus 5 · Low on schemelike-metacircular-eval: solvedOpus 5 · Low on regex-chess: solvedOpus 5 · Low on query-optimize: solvedOpus 5 · Low on pytorch-model-cli: solvedOpus 5 · Low on cancel-async-tasks: solvedOpus 5 · Low on dna-assembly: solvedOpus 5 · Low on adaptive-rejection-sampler: failedOpus 5 · Low on caffe-cifar-10: failedOpus 5 · Low on train-fasttext: failedOpus 5 · Low on filter-js-from-html: failed1418. Opus 4.6 · LowOpus 4.6 · Low on git-leak-recovery: solvedOpus 4.6 · Low on pypi-server: solvedOpus 4.6 · Low on modernize-scientific-stack: solvedOpus 4.6 · Low on feal-differential-cryptanalysis: solvedOpus 4.6 · Low on fix-git: solvedOpus 4.6 · Low on kv-store-grpc: solvedOpus 4.6 · Low on password-recovery: solvedOpus 4.6 · Low on headless-terminal: solvedOpus 4.6 · Low on llm-inference-batching-scheduler: solvedOpus 4.6 · Low on mteb-leaderboard: solvedOpus 4.6 · Low on schemelike-metacircular-eval: solvedOpus 4.6 · Low on regex-chess: failedOpus 4.6 · Low on query-optimize: failedOpus 4.6 · Low on pytorch-model-cli: solvedOpus 4.6 · Low on cancel-async-tasks: failedOpus 4.6 · Low on dna-assembly: failedOpus 4.6 · Low on adaptive-rejection-sampler: solvedOpus 4.6 · Low on caffe-cifar-10: failedOpus 4.6 · Low on train-fasttext: failedOpus 4.6 · Low on filter-js-from-html: failed1319. Sonnet 5 · MaxSonnet 5 · Max on git-leak-recovery: solvedSonnet 5 · Max on pypi-server: solvedSonnet 5 · Max on modernize-scientific-stack: solvedSonnet 5 · Max on feal-differential-cryptanalysis: solvedSonnet 5 · Max on fix-git: solvedSonnet 5 · Max on kv-store-grpc: solvedSonnet 5 · Max on password-recovery: failedSonnet 5 · Max on headless-terminal: solvedSonnet 5 · Max on llm-inference-batching-scheduler: solvedSonnet 5 · Max on mteb-leaderboard: failedSonnet 5 · Max on schemelike-metacircular-eval: solvedSonnet 5 · Max on regex-chess: solvedSonnet 5 · Max on query-optimize: solvedSonnet 5 · Max on pytorch-model-cli: failedSonnet 5 · Max on cancel-async-tasks: failedSonnet 5 · Max on dna-assembly: solvedSonnet 5 · Max on adaptive-rejection-sampler: failedSonnet 5 · Max on caffe-cifar-10: solvedSonnet 5 · Max on train-fasttext: failedSonnet 5 · Max on filter-js-from-html: failed1320. Fable 5.1 · HighFable 5.1 · High on git-leak-recovery: solvedFable 5.1 · High on pypi-server: solvedFable 5.1 · High on modernize-scientific-stack: solvedFable 5.1 · High on feal-differential-cryptanalysis: solvedFable 5.1 · High on fix-git: solvedFable 5.1 · High on kv-store-grpc: solvedFable 5.1 · High on password-recovery: solvedFable 5.1 · High on headless-terminal: solvedFable 5.1 · High on llm-inference-batching-scheduler: solvedFable 5.1 · High on mteb-leaderboard: solvedFable 5.1 · High on schemelike-metacircular-eval: failedFable 5.1 · High on regex-chess: solvedFable 5.1 · High on query-optimize: solvedFable 5.1 · High on pytorch-model-cli: failedFable 5.1 · High on cancel-async-tasks: solvedFable 5.1 · High on dna-assembly: failedFable 5.1 · High on adaptive-rejection-sampler: failedFable 5.1 · High on caffe-cifar-10: failedFable 5.1 · High on train-fasttext: failedFable 5.1 · High on filter-js-from-html: failed1321. Fable 5.1 · Extra highFable 5.1 · Extra high on git-leak-recovery: solvedFable 5.1 · Extra high on pypi-server: solvedFable 5.1 · Extra high on modernize-scientific-stack: solvedFable 5.1 · Extra high on feal-differential-cryptanalysis: solvedFable 5.1 · Extra high on fix-git: solvedFable 5.1 · Extra high on kv-store-grpc: solvedFable 5.1 · Extra high on password-recovery: solvedFable 5.1 · Extra high on headless-terminal: solvedFable 5.1 · Extra high on llm-inference-batching-scheduler: solvedFable 5.1 · Extra high on mteb-leaderboard: solvedFable 5.1 · Extra high on schemelike-metacircular-eval: failedFable 5.1 · Extra high on regex-chess: solvedFable 5.1 · Extra high on query-optimize: solvedFable 5.1 · Extra high on pytorch-model-cli: failedFable 5.1 · Extra high on cancel-async-tasks: solvedFable 5.1 · Extra high on dna-assembly: failedFable 5.1 · Extra high on adaptive-rejection-sampler: failedFable 5.1 · Extra high on caffe-cifar-10: failedFable 5.1 · Extra high on train-fasttext: failedFable 5.1 · Extra high on filter-js-from-html: failed1322. Sonnet 5 · Extra highSonnet 5 · Extra high on git-leak-recovery: solvedSonnet 5 · Extra high on pypi-server: solvedSonnet 5 · Extra high on modernize-scientific-stack: solvedSonnet 5 · Extra high on feal-differential-cryptanalysis: solvedSonnet 5 · Extra high on fix-git: solvedSonnet 5 · Extra high on kv-store-grpc: solvedSonnet 5 · Extra high on password-recovery: solvedSonnet 5 · Extra high on headless-terminal: solvedSonnet 5 · Extra high on llm-inference-batching-scheduler: solvedSonnet 5 · Extra high on mteb-leaderboard: solvedSonnet 5 · Extra high on schemelike-metacircular-eval: solvedSonnet 5 · Extra high on regex-chess: solvedSonnet 5 · Extra high on query-optimize: failedSonnet 5 · Extra high on pytorch-model-cli: solvedSonnet 5 · Extra high on cancel-async-tasks: failedSonnet 5 · Extra high on dna-assembly: failedSonnet 5 · Extra high on adaptive-rejection-sampler: failedSonnet 5 · Extra high on caffe-cifar-10: failedSonnet 5 · Extra high on train-fasttext: failedSonnet 5 · Extra high on filter-js-from-html: failed1323. Opus 5.5 · LowOpus 5.5 · Low on git-leak-recovery: solvedOpus 5.5 · Low on pypi-server: solvedOpus 5.5 · Low on modernize-scientific-stack: solvedOpus 5.5 · Low on feal-differential-cryptanalysis: solvedOpus 5.5 · Low on fix-git: solvedOpus 5.5 · Low on kv-store-grpc: solvedOpus 5.5 · Low on password-recovery: solvedOpus 5.5 · Low on headless-terminal: solvedOpus 5.5 · Low on llm-inference-batching-scheduler: solvedOpus 5.5 · Low on mteb-leaderboard: failedOpus 5.5 · Low on schemelike-metacircular-eval: solvedOpus 5.5 · Low on regex-chess: solvedOpus 5.5 · Low on query-optimize: failedOpus 5.5 · Low on pytorch-model-cli: solvedOpus 5.5 · Low on cancel-async-tasks: solvedOpus 5.5 · Low on dna-assembly: failedOpus 5.5 · Low on adaptive-rejection-sampler: failedOpus 5.5 · Low on caffe-cifar-10: failedOpus 5.5 · Low on train-fasttext: failedOpus 5.5 · Low on filter-js-from-html: failed1324. Opus 5 · MaxOpus 5 · Max on git-leak-recovery: solvedOpus 5 · Max on pypi-server: solvedOpus 5 · Max on modernize-scientific-stack: solvedOpus 5 · Max on feal-differential-cryptanalysis: solvedOpus 5 · Max on fix-git: failedOpus 5 · Max on kv-store-grpc: solvedOpus 5 · Max on password-recovery: solvedOpus 5 · Max on headless-terminal: solvedOpus 5 · Max on llm-inference-batching-scheduler: failedOpus 5 · Max on mteb-leaderboard: solvedOpus 5 · Max on schemelike-metacircular-eval: solvedOpus 5 · Max on regex-chess: solvedOpus 5 · Max on query-optimize: solvedOpus 5 · Max on pytorch-model-cli: failedOpus 5 · Max on cancel-async-tasks: solvedOpus 5 · Max on dna-assembly: solvedOpus 5 · Max on adaptive-rejection-sampler: failedOpus 5 · Max on caffe-cifar-10: failedOpus 5 · Max on train-fasttext: failedOpus 5 · Max on filter-js-from-html: failed1325. Sonnet 4.6 · HighSonnet 4.6 · High on git-leak-recovery: solvedSonnet 4.6 · High on pypi-server: solvedSonnet 4.6 · High on modernize-scientific-stack: solvedSonnet 4.6 · High on feal-differential-cryptanalysis: solvedSonnet 4.6 · High on fix-git: solvedSonnet 4.6 · High on kv-store-grpc: solvedSonnet 4.6 · High on password-recovery: solvedSonnet 4.6 · High on headless-terminal: solvedSonnet 4.6 · High on llm-inference-batching-scheduler: failedSonnet 4.6 · High on mteb-leaderboard: solvedSonnet 4.6 · High on schemelike-metacircular-eval: solvedSonnet 4.6 · High on regex-chess: failedSonnet 4.6 · High on query-optimize: failedSonnet 4.6 · High on pytorch-model-cli: solvedSonnet 4.6 · High on cancel-async-tasks: failedSonnet 4.6 · High on dna-assembly: failedSonnet 4.6 · High on adaptive-rejection-sampler: failedSonnet 4.6 · High on caffe-cifar-10: solvedSonnet 4.6 · High on train-fasttext: failedSonnet 4.6 · High on filter-js-from-html: failed1226. Sonnet 5 · HighSonnet 5 · High on git-leak-recovery: solvedSonnet 5 · High on pypi-server: solvedSonnet 5 · High on modernize-scientific-stack: solvedSonnet 5 · High on feal-differential-cryptanalysis: solvedSonnet 5 · High on fix-git: failedSonnet 5 · High on kv-store-grpc: solvedSonnet 5 · High on password-recovery: solvedSonnet 5 · High on headless-terminal: solvedSonnet 5 · High on llm-inference-batching-scheduler: solvedSonnet 5 · High on mteb-leaderboard: solvedSonnet 5 · High on schemelike-metacircular-eval: failedSonnet 5 · High on regex-chess: failedSonnet 5 · High on query-optimize: failedSonnet 5 · High on pytorch-model-cli: solvedSonnet 5 · High on cancel-async-tasks: failedSonnet 5 · High on dna-assembly: solvedSonnet 5 · High on adaptive-rejection-sampler: failedSonnet 5 · High on caffe-cifar-10: solvedSonnet 5 · High on train-fasttext: failedSonnet 5 · High on filter-js-from-html: failed1227. Opus 4.6 · MaxOpus 4.6 · Max on git-leak-recovery: solvedOpus 4.6 · Max on pypi-server: solvedOpus 4.6 · Max on modernize-scientific-stack: solvedOpus 4.6 · Max on feal-differential-cryptanalysis: solvedOpus 4.6 · Max on fix-git: solvedOpus 4.6 · Max on kv-store-grpc: solvedOpus 4.6 · Max on password-recovery: solvedOpus 4.6 · Max on headless-terminal: solvedOpus 4.6 · Max on llm-inference-batching-scheduler: solvedOpus 4.6 · Max on mteb-leaderboard: failedOpus 4.6 · Max on schemelike-metacircular-eval: failedOpus 4.6 · Max on regex-chess: failedOpus 4.6 · Max on query-optimize: solvedOpus 4.6 · Max on pytorch-model-cli: solvedOpus 4.6 · Max on cancel-async-tasks: failedOpus 4.6 · Max on dna-assembly: failedOpus 4.6 · Max on adaptive-rejection-sampler: solvedOpus 4.6 · Max on caffe-cifar-10: failedOpus 4.6 · Max on train-fasttext: failedOpus 4.6 · Max on filter-js-from-html: failed1228. Opus 4.6 · MediumOpus 4.6 · Medium on git-leak-recovery: solvedOpus 4.6 · Medium on pypi-server: solvedOpus 4.6 · Medium on modernize-scientific-stack: solvedOpus 4.6 · Medium on feal-differential-cryptanalysis: solvedOpus 4.6 · Medium on fix-git: solvedOpus 4.6 · Medium on kv-store-grpc: solvedOpus 4.6 · Medium on password-recovery: solvedOpus 4.6 · Medium on headless-terminal: solvedOpus 4.6 · Medium on llm-inference-batching-scheduler: solvedOpus 4.6 · Medium on mteb-leaderboard: failedOpus 4.6 · Medium on schemelike-metacircular-eval: failedOpus 4.6 · Medium on regex-chess: failedOpus 4.6 · Medium on query-optimize: failedOpus 4.6 · Medium on pytorch-model-cli: solvedOpus 4.6 · Medium on cancel-async-tasks: failedOpus 4.6 · Medium on dna-assembly: solvedOpus 4.6 · Medium on adaptive-rejection-sampler: solvedOpus 4.6 · Medium on caffe-cifar-10: failedOpus 4.6 · Medium on train-fasttext: failedOpus 4.6 · Medium on filter-js-from-html: failed1229. Sonnet 4.6 · MediumSonnet 4.6 · Medium on git-leak-recovery: solvedSonnet 4.6 · Medium on pypi-server: solvedSonnet 4.6 · Medium on modernize-scientific-stack: solvedSonnet 4.6 · Medium on feal-differential-cryptanalysis: solvedSonnet 4.6 · Medium on fix-git: failedSonnet 4.6 · Medium on kv-store-grpc: solvedSonnet 4.6 · Medium on password-recovery: solvedSonnet 4.6 · Medium on headless-terminal: solvedSonnet 4.6 · Medium on llm-inference-batching-scheduler: solvedSonnet 4.6 · Medium on mteb-leaderboard: solvedSonnet 4.6 · Medium on schemelike-metacircular-eval: solvedSonnet 4.6 · Medium on regex-chess: failedSonnet 4.6 · Medium on query-optimize: solvedSonnet 4.6 · Medium on pytorch-model-cli: solvedSonnet 4.6 · Medium on cancel-async-tasks: failedSonnet 4.6 · Medium on dna-assembly: failedSonnet 4.6 · Medium on adaptive-rejection-sampler: failedSonnet 4.6 · Medium on caffe-cifar-10: failedSonnet 4.6 · Medium on train-fasttext: failedSonnet 4.6 · Medium on filter-js-from-html: failed1230. Sonnet 4.6 · LowSonnet 4.6 · Low on git-leak-recovery: solvedSonnet 4.6 · Low on pypi-server: solvedSonnet 4.6 · Low on modernize-scientific-stack: solvedSonnet 4.6 · Low on feal-differential-cryptanalysis: solvedSonnet 4.6 · Low on fix-git: solvedSonnet 4.6 · Low on kv-store-grpc: solvedSonnet 4.6 · Low on password-recovery: solvedSonnet 4.6 · Low on headless-terminal: solvedSonnet 4.6 · Low on llm-inference-batching-scheduler: solvedSonnet 4.6 · Low on mteb-leaderboard: failedSonnet 4.6 · Low on schemelike-metacircular-eval: solvedSonnet 4.6 · Low on regex-chess: failedSonnet 4.6 · Low on query-optimize: solvedSonnet 4.6 · Low on pytorch-model-cli: failedSonnet 4.6 · Low on cancel-async-tasks: solvedSonnet 4.6 · Low on dna-assembly: failedSonnet 4.6 · Low on adaptive-rejection-sampler: failedSonnet 4.6 · Low on caffe-cifar-10: failedSonnet 4.6 · Low on train-fasttext: failedSonnet 4.6 · Low on filter-js-from-html: failed1231. Sonnet 4.5 · DefaultSonnet 4.5 · Default on git-leak-recovery: solvedSonnet 4.5 · Default on pypi-server: solvedSonnet 4.5 · Default on modernize-scientific-stack: solvedSonnet 4.5 · Default on feal-differential-cryptanalysis: solvedSonnet 4.5 · Default on fix-git: solvedSonnet 4.5 · Default on kv-store-grpc: failedSonnet 4.5 · Default on password-recovery: failedSonnet 4.5 · Default on headless-terminal: solvedSonnet 4.5 · Default on llm-inference-batching-scheduler: solvedSonnet 4.5 · Default on mteb-leaderboard: failedSonnet 4.5 · Default on schemelike-metacircular-eval: failedSonnet 4.5 · Default on regex-chess: failedSonnet 4.5 · Default on query-optimize: solvedSonnet 4.5 · Default on pytorch-model-cli: solvedSonnet 4.5 · Default on cancel-async-tasks: failedSonnet 4.5 · Default on dna-assembly: failedSonnet 4.5 · Default on adaptive-rejection-sampler: solvedSonnet 4.5 · Default on caffe-cifar-10: solvedSonnet 4.5 · Default on train-fasttext: failedSonnet 4.5 · Default on filter-js-from-html: failed1132. Opus 5.5 · Extra highOpus 5.5 · Extra high on git-leak-recovery: solvedOpus 5.5 · Extra high on pypi-server: solvedOpus 5.5 · Extra high on modernize-scientific-stack: solvedOpus 5.5 · Extra high on feal-differential-cryptanalysis: solvedOpus 5.5 · Extra high on fix-git: solvedOpus 5.5 · Extra high on kv-store-grpc: solvedOpus 5.5 · Extra high on password-recovery: solvedOpus 5.5 · Extra high on headless-terminal: solvedOpus 5.5 · Extra high on llm-inference-batching-scheduler: solvedOpus 5.5 · Extra high on mteb-leaderboard: failedOpus 5.5 · Extra high on schemelike-metacircular-eval: failedOpus 5.5 · Extra high on regex-chess: solvedOpus 5.5 · Extra high on query-optimize: failedOpus 5.5 · Extra high on pytorch-model-cli: failedOpus 5.5 · Extra high on cancel-async-tasks: solvedOpus 5.5 · Extra high on dna-assembly: solvedOpus 5.5 · Extra high on adaptive-rejection-sampler: failedOpus 5.5 · Extra high on caffe-cifar-10: failedOpus 5.5 · Extra high on train-fasttext: failedOpus 5.5 · Extra high on filter-js-from-html: failed1233. Sonnet 5.5 · MaxSonnet 5.5 · Max on git-leak-recovery: solvedSonnet 5.5 · Max on pypi-server: solvedSonnet 5.5 · Max on modernize-scientific-stack: solvedSonnet 5.5 · Max on feal-differential-cryptanalysis: solvedSonnet 5.5 · Max on fix-git: solvedSonnet 5.5 · Max on kv-store-grpc: solvedSonnet 5.5 · Max on password-recovery: solvedSonnet 5.5 · Max on headless-terminal: failedSonnet 5.5 · Max on llm-inference-batching-scheduler: solvedSonnet 5.5 · Max on mteb-leaderboard: solvedSonnet 5.5 · Max on schemelike-metacircular-eval: solvedSonnet 5.5 · Max on regex-chess: solvedSonnet 5.5 · Max on query-optimize: failedSonnet 5.5 · Max on pytorch-model-cli: failedSonnet 5.5 · Max on cancel-async-tasks: solvedSonnet 5.5 · Max on dna-assembly: failedSonnet 5.5 · Max on adaptive-rejection-sampler: failedSonnet 5.5 · Max on caffe-cifar-10: failedSonnet 5.5 · Max on train-fasttext: failedSonnet 5.5 · Max on filter-js-from-html: failed1234. Opus 5.5 · MaxOpus 5.5 · Max on git-leak-recovery: solvedOpus 5.5 · Max on pypi-server: solvedOpus 5.5 · Max on modernize-scientific-stack: failedOpus 5.5 · Max on feal-differential-cryptanalysis: solvedOpus 5.5 · Max on fix-git: solvedOpus 5.5 · Max on kv-store-grpc: solvedOpus 5.5 · Max on password-recovery: solvedOpus 5.5 · Max on headless-terminal: failedOpus 5.5 · Max on llm-inference-batching-scheduler: failedOpus 5.5 · Max on mteb-leaderboard: solvedOpus 5.5 · Max on schemelike-metacircular-eval: failedOpus 5.5 · Max on regex-chess: solvedOpus 5.5 · Max on query-optimize: solvedOpus 5.5 · Max on pytorch-model-cli: failedOpus 5.5 · Max on cancel-async-tasks: failedOpus 5.5 · Max on dna-assembly: solvedOpus 5.5 · Max on adaptive-rejection-sampler: failedOpus 5.5 · Max on caffe-cifar-10: failedOpus 5.5 · Max on train-fasttext: solvedOpus 5.5 · Max on filter-js-from-html: failed1135. Sonnet 5 · LowSonnet 5 · Low on git-leak-recovery: solvedSonnet 5 · Low on pypi-server: solvedSonnet 5 · Low on modernize-scientific-stack: solvedSonnet 5 · Low on feal-differential-cryptanalysis: solvedSonnet 5 · Low on fix-git: solvedSonnet 5 · Low on kv-store-grpc: solvedSonnet 5 · Low on password-recovery: solvedSonnet 5 · Low on headless-terminal: failedSonnet 5 · Low on llm-inference-batching-scheduler: solvedSonnet 5 · Low on mteb-leaderboard: failedSonnet 5 · Low on schemelike-metacircular-eval: solvedSonnet 5 · Low on regex-chess: failedSonnet 5 · Low on query-optimize: failedSonnet 5 · Low on pytorch-model-cli: solvedSonnet 5 · Low on cancel-async-tasks: failedSonnet 5 · Low on dna-assembly: solvedSonnet 5 · Low on adaptive-rejection-sampler: failedSonnet 5 · Low on caffe-cifar-10: failedSonnet 5 · Low on train-fasttext: failedSonnet 5 · Low on filter-js-from-html: failed1136. Fable 5.1 · MaxFable 5.1 · Max on git-leak-recovery: solvedFable 5.1 · Max on pypi-server: solvedFable 5.1 · Max on modernize-scientific-stack: solvedFable 5.1 · Max on feal-differential-cryptanalysis: solvedFable 5.1 · Max on fix-git: solvedFable 5.1 · Max on kv-store-grpc: solvedFable 5.1 · Max on password-recovery: solvedFable 5.1 · Max on headless-terminal: failedFable 5.1 · Max on llm-inference-batching-scheduler: solvedFable 5.1 · Max on mteb-leaderboard: failedFable 5.1 · Max on schemelike-metacircular-eval: failedFable 5.1 · Max on regex-chess: solvedFable 5.1 · Max on query-optimize: solvedFable 5.1 · Max on pytorch-model-cli: failedFable 5.1 · Max on cancel-async-tasks: solvedFable 5.1 · Max on dna-assembly: failedFable 5.1 · Max on adaptive-rejection-sampler: failedFable 5.1 · Max on caffe-cifar-10: failedFable 5.1 · Max on train-fasttext: failedFable 5.1 · Max on filter-js-from-html: failed1137. Opus 4.6 · HighOpus 4.6 · High on git-leak-recovery: solvedOpus 4.6 · High on pypi-server: solvedOpus 4.6 · High on modernize-scientific-stack: solvedOpus 4.6 · High on feal-differential-cryptanalysis: solvedOpus 4.6 · High on fix-git: solvedOpus 4.6 · High on kv-store-grpc: solvedOpus 4.6 · High on password-recovery: solvedOpus 4.6 · High on headless-terminal: solvedOpus 4.6 · High on llm-inference-batching-scheduler: solvedOpus 4.6 · High on mteb-leaderboard: solvedOpus 4.6 · High on schemelike-metacircular-eval: failedOpus 4.6 · High on regex-chess: failedOpus 4.6 · High on query-optimize: solvedOpus 4.6 · High on pytorch-model-cli: failedOpus 4.6 · High on cancel-async-tasks: failedOpus 4.6 · High on dna-assembly: failedOpus 4.6 · High on adaptive-rejection-sampler: failedOpus 4.6 · High on caffe-cifar-10: failedOpus 4.6 · High on train-fasttext: failedOpus 4.6 · High on filter-js-from-html: failed1138. Haiku 4.5 · DefaultHaiku 4.5 · Default on git-leak-recovery: solvedHaiku 4.5 · Default on pypi-server: solvedHaiku 4.5 · Default on modernize-scientific-stack: solvedHaiku 4.5 · Default on feal-differential-cryptanalysis: failedHaiku 4.5 · Default on fix-git: solvedHaiku 4.5 · Default on kv-store-grpc: solved once in two runsHaiku 4.5 · Default on password-recovery: failedHaiku 4.5 · Default on headless-terminal: solvedHaiku 4.5 · Default on llm-inference-batching-scheduler: failedHaiku 4.5 · Default on mteb-leaderboard: solved once in two runsHaiku 4.5 · Default on schemelike-metacircular-eval: solved once in two runsHaiku 4.5 · Default on regex-chess: failedHaiku 4.5 · Default on query-optimize: solvedHaiku 4.5 · Default on pytorch-model-cli: solvedHaiku 4.5 · Default on cancel-async-tasks: failedHaiku 4.5 · Default on dna-assembly: failedHaiku 4.5 · Default on adaptive-rejection-sampler: failedHaiku 4.5 · Default on caffe-cifar-10: failedHaiku 4.5 · Default on train-fasttext: failedHaiku 4.5 · Default on filter-js-from-html: failed8.539. Sonnet 4 · DefaultSonnet 4 · Default on git-leak-recovery: solvedSonnet 4 · Default on pypi-server: solvedSonnet 4 · Default on modernize-scientific-stack: solvedSonnet 4 · Default on feal-differential-cryptanalysis: failedSonnet 4 · Default on fix-git: solvedSonnet 4 · Default on kv-store-grpc: solvedSonnet 4 · Default on password-recovery: failedSonnet 4 · Default on headless-terminal: solvedSonnet 4 · Default on llm-inference-batching-scheduler: failedSonnet 4 · Default on mteb-leaderboard: solvedSonnet 4 · Default on schemelike-metacircular-eval: failedSonnet 4 · Default on regex-chess: failedSonnet 4 · Default on query-optimize: failedSonnet 4 · Default on pytorch-model-cli: failedSonnet 4 · Default on cancel-async-tasks: solvedSonnet 4 · Default on dna-assembly: failedSonnet 4 · Default on adaptive-rejection-sampler: failedSonnet 4 · Default on caffe-cifar-10: failedSonnet 4 · Default on train-fasttext: failedSonnet 4 · Default on filter-js-from-html: failed840. Sonnet 4.6 · MaxSonnet 4.6 · Max on git-leak-recovery: solvedSonnet 4.6 · Max on pypi-server: solvedSonnet 4.6 · Max on modernize-scientific-stack: solvedSonnet 4.6 · Max on feal-differential-cryptanalysis: failedSonnet 4.6 · Max on fix-git: solvedSonnet 4.6 · Max on kv-store-grpc: solvedSonnet 4.6 · Max on password-recovery: solvedSonnet 4.6 · Max on headless-terminal: solvedSonnet 4.6 · Max on llm-inference-batching-scheduler: failedSonnet 4.6 · Max on mteb-leaderboard: solvedSonnet 4.6 · Max on schemelike-metacircular-eval: failedSonnet 4.6 · Max on regex-chess: failedSonnet 4.6 · Max on query-optimize: failedSonnet 4.6 · Max on pytorch-model-cli: failedSonnet 4.6 · Max on cancel-async-tasks: failedSonnet 4.6 · Max on dna-assembly: failedSonnet 4.6 · Max on adaptive-rejection-sampler: failedSonnet 4.6 · Max on caffe-cifar-10: failedSonnet 4.6 · Max on train-fasttext: failedSonnet 4.6 · Max on filter-js-from-html: failed8← easiesthardest →
solvedfailedNumber above each task: how many of the 40 configurations solved it. Number at the end of each row: tasks solved.

filter-js-from-html is the column no configuration solved. It is solvable: the suite’s reference solution passes it. It has 2 tests: the filter must block malicious JavaScript, and it must leave clean HTML unchanged. 17 of the 41 runs of it (one per configuration, two for Claude Haiku 4.5) blocked the script but also changed clean HTML, so they passed one test of two; the other 24 passed neither.

05

Effort

These comparisons are paired: two effort levels of the same model on the same 20 tasks.

  • Across 63 such comparisons, a lower effort level was never significantly worse than a higher one.
  • 6 comparisons significantly favor the lower effort level.
  • Max effort was the worst or tied-worst for 5 of the 7 models that offer it, mostly because it ran out of time. The exceptions were Claude Sonnet 5 and Claude Opus 4.6.
  • Max cost 1.6x to 13.9x as much per task as each model’s best result per dollar.

Claude Opus 5.5

61218Claude Opus 5.5, low: 13/20 solved, $0.25 per task, 0 timeouts13Claude Opus 5.5, medium: 16/20 solved, $0.46 per task, 2 timeouts16Claude Opus 5.5, high: 16/20 solved, $0.62 per task, 2 timeouts16Claude Opus 5.5, xhigh: 12/20 solved, $0.94 per task, 3 timeouts12Claude Opus 5.5, max: 11/20 solved, $2.36 per task, 9 timeouts11L$0.25M$0.46H$0.62XH$0.94Max$2.36

Claude Sonnet 5.5

61218Claude Sonnet 5.5, low: 16/20 solved, $0.13 per task, 0 timeouts16Claude Sonnet 5.5, medium: 15/20 solved, $0.14 per task, 0 timeouts15Claude Sonnet 5.5, high: 16/20 solved, $0.19 per task, 0 timeouts16Claude Sonnet 5.5, xhigh: 16/20 solved, $0.61 per task, 4 timeouts16Claude Sonnet 5.5, max: 12/20 solved, $1.75 per task, 11 timeouts12L$0.13M$0.14H$0.19XH$0.61Max$1.75

Claude Fable 5.1

61218Claude Fable 5.1, low: 15/20 solved, $0.81 per task, 1 timeouts15Claude Fable 5.1, medium: 14/20 solved, $0.98 per task, 0 timeouts14Claude Fable 5.1, high: 13/20 solved, $1.48 per task, 1 timeouts13Claude Fable 5.1, xhigh: 13/20 solved, $3.25 per task, 5 timeouts13Claude Fable 5.1, max: 11/20 solved, $3.71 per task, 9 timeouts11L$0.81M$0.98H$1.48XH$3.25Max$3.71

Claude Opus 5

61218Claude Opus 5, low: 14/20 solved, $0.61 per task, 1 timeouts14Claude Opus 5, medium: 15/20 solved, $0.81 per task, 2 timeouts15Claude Opus 5, high: 16/20 solved, $1.86 per task, 4 timeouts16Claude Opus 5, xhigh: 14/20 solved, $2.23 per task, 5 timeouts14Claude Opus 5, max: 13/20 solved, $3.07 per task, 8 timeouts13L$0.61M$0.81H$1.86XH$2.23Max$3.07

Claude Sonnet 5

61218Claude Sonnet 5, low: 11/20 solved, $0.37 per task, 0 timeouts11Claude Sonnet 5, medium: 14/20 solved, $0.47 per task, 1 timeouts14Claude Sonnet 5, high: 12/20 solved, $0.76 per task, 1 timeouts12Claude Sonnet 5, xhigh: 13/20 solved, $0.88 per task, 0 timeouts13Claude Sonnet 5, max: 13/20 solved, $1.29 per task, 2 timeouts13L$0.37M$0.47H$0.76XH$0.88Max$1.29

Claude Opus 4.6

61218Claude Opus 4.6, low: 13/20 solved, $0.78 per task, 2 timeouts13Claude Opus 4.6, medium: 12/20 solved, $1.03 per task, 3 timeouts12Claude Opus 4.6, high: 11/20 solved, $1.18 per task, 4 timeouts11Claude Opus 4.6, max: 12/20 solved, $1.26 per task, 5 timeouts12L$0.78M$1.03H$1.18XH–Max$1.26

Claude Sonnet 4.6

61218Claude Sonnet 4.6, low: 12/20 solved, $0.59 per task, 2 timeouts12Claude Sonnet 4.6, medium: 12/20 solved, $0.76 per task, 2 timeouts12Claude Sonnet 4.6, high: 12/20 solved, $0.89 per task, 4 timeouts12Claude Sonnet 4.6, max: 8/20 solved, $0.94 per task, 5 timeouts8L$0.59M$0.76H$0.89XH–Max$0.94

GPT-5.6 Luna

61218GPT-5.6 Luna, high: 14/20 solved, $0.045 per task, 0 timeouts14GPT-5.6 Luna, xhigh: 15/20 solved, $0.040 per task, 3 timeouts15L–M–H$0.045XH$0.040Max–
Tasks solved out of 20 (number above each point) by effort level (L, M, H, XH, Max), with cost per task underneath. Same scale in every panel. Accent: that model’s best result per dollar. A dash: effort level not offered or not measured.

The clearest example: Claude Opus 5.5

Medium effort

16/20

$0.46 per task

Extra high effort

12/20

$0.94 per task

Paired difference

+20 points

95% interval [+5, +40], significant

The 6 significant comparisons

ModelLower effortHigher effortDifference [95%]
Claude Fable 5.1Low: 15/20, $0.81Max: 11/20, $3.71+20 [+5, +40]
Claude Opus 5.5Medium: 16/20, $0.46Extra high: 12/20, $0.94+20 [+5, +40]
Claude Sonnet 4.6High: 12/20, $0.89Max: 8/20, $0.94+20 [+5, +40]
Claude Sonnet 5.5High: 16/20, $0.19Max: 12/20, $1.75+20 [+5, +40]
Claude Sonnet 5.5Low: 16/20, $0.13Max: 12/20, $1.75+20 [+5, +40]
Claude Sonnet 5.5Extra high: 16/20, $0.61Max: 12/20, $1.75+20 [+5, +40]

Timeouts by effort

How many of the 20 runs hit the task’s time limit. A timed-out run is still graded on what the agent left behind: of 108 timed-out runs, 76 failed and 32 passed because the work was already done. A dash means the model doesn’t offer that effort level.

ModelLowMediumHighExtra highMax
Claude Opus 5.502239
Claude Sonnet 5.5000411
Claude Fable 5.110159
Claude Opus 512458
Claude Sonnet 501102
Claude Opus 4.6234–5
Claude Sonnet 4.6224–5

How far the 76 failed timeouts got: share of the task’s tests passed

  • No tests 42
  • Under half 9
  • Half to three-quarters 15
  • Three-quarters or more 10
Runs that hit the time limit were usually further from done than other failures: on average they passed 25% of the task’s tests, against 43% for failed runs that finished in time. So max effort mostly lost by running out of time, not by falling just short. 49 of the 108 timeouts were at max effort.

Best result per dollar in our runs

For each model, the cheapest effort level that solved within one task of that model’s best, set against max effort where the model offers it.

ModelEffortSolved$ / taskMax: solvedMax: $ / taskMax costs
Claude Opus 5.5Medium16/20$0.4611/20$2.365.1x
Claude Sonnet 5.5Low16/20$0.1312/20$1.7513.9x
Claude Fable 5.1Low15/20$0.8111/20$3.714.6x
Claude Opus 5Medium15/20$0.8113/20$3.073.8x
Claude Sonnet 5Medium14/20$0.4713/20$1.292.7x
Claude Opus 4.6Low13/20$0.7812/20$1.261.6x
Claude Sonnet 4.6Low12/20$0.598/20$0.941.6x
GPT-5.6 LunaExtra high15/20$0.040–––

GPT-5.6 Luna was measured at high and extra high effort only.

06

Turns and cache

A turn is one model call: the model reads the conversation so far and replies, usually by asking for a command or an edit. Higher effort took more turns. The median run made 7 calls at low effort, 8 at medium, 10 at high, and 12 at both extra high and max.

More turns did not mean more tasks solved. Solved and failed runs took about the same number of turns (medians 10 and 11). On 8 of the 17 tasks that had both kinds of run, the solved runs took fewer.

Each turn re-reads the conversation, and most of it comes from the prompt cache: 95% of all input tokens were cache reads (77% to 98% across configurations). Cache reads are cheap, so they were only 20% of the cost. Writing to the cache was 26% and output 54%.

Median turns per run, by effort level

  • Low7
  • Medium8
  • High10
  • Extra high12
  • Max12
A turn is one model call. Runs at each level: low 160, medium 140, high 160, extra high 120, max 140. Models without an effort level are left out.

Input tokens the models read

  • Uncached input 0.2%
  • Cache writes 5%
  • Cache reads 95%

Token cost

  • Uncached input 0.3%
  • Cache writes 26%
  • Cache reads 20%
  • Output 54%
All 820 runs, every token priced at list price.

07

Method and caveats

How we ran and rated it

  • Terminal-Bench 2.0: the same 20 tasks for every configuration, each run once (Claude Haiku 4.5 twice; its runs are pooled). The suite’s own tests decide pass or fail.
  • Ratings: the Bradley–Terry model set out in section 03, fitted by maximum likelihood, with a 95% interval from 1,000 resamples of the tasks.
  • When a run failed because of our own setup (a restart, or a grader that ran out of time), we re-ran it and did not score the failed attempt: 52 runs in all. A run the model itself failed was never re-run.
  • Paired differences in the effort section use a 95% bootstrap interval over tasks; “significant” means that interval excludes zero.
  • Cost per task counts every token type (uncached input, cache writes, cache reads and output), each at the model’s list price. For 601 runs the agent kept its own running cost total, which also includes side calls such as web searches, and we use that. It matches our own pricing of the logged calls to within a cent on 538 of those runs. For the other runs, mostly ones cut off by a timeout, we price the logged calls ourselves.
  • In 25 runs (Claude Fable 5.1 10, Claude Opus 5.5 12, Claude Opus 5 3), the model’s safety checks declined part of the work, and Claude Opus 4.8 answered those calls instead. The runs still count for the configuration they started with, and those calls are priced at Opus 4.8’s rate.
The 20 tasks

adaptive-rejection-sampler, caffe-cifar-10, cancel-async-tasks, dna-assembly, feal-differential-cryptanalysis, filter-js-from-html, fix-git, git-leak-recovery, headless-terminal, kv-store-grpc, llm-inference-batching-scheduler, modernize-scientific-stack, mteb-leaderboard, password-recovery, pypi-server, pytorch-model-cli, query-optimize, regex-chess, schemelike-metacircular-eval, train-fasttext

What to keep in mind

  • Small sample. 20 tasks give wide intervals: most ratings span a few hundred points, so many configurations overlap. Read the leaderboard in groups, not as an exact order. In pass-rate terms, one task is 5 points and intervals are about ±20.
  • Our own scale. Ratings compare these 40 configurations on these tasks. They can’t be set beside another leaderboard’s numbers.
  • One suite. These are agentic tasks in a terminal. Other kinds of work may rank models and effort levels differently.
  • Time limits drive the max result. Max mostly lost by running out of time. With longer limits it could look better; when you are waiting on the answer, the time is a real cost either way.
  • Costs are list prices, not billed amounts. Plan allowances, discounts and team fees are not included.
  • A snapshot. Models and their defaults change; these results describe the versions available when we ran them.