Measure before choosing a model.
Compare model capability, prompts, and tools using measured results. Short experiments are reported separately from the full-year store benchmark.
Explore the full-year store experiment →
Research task comparisons
These are synthetic cases without independent expert review. Repeated answers are not additional independent cases. Development-set improvements do not establish performance on unseen tasks.
| Stage | Configuration | Passed / trials | Cases | Median latency | Accounted USD |
|---|---|---|---|---|---|
| development baseline | gpt-4.1-mini-2025-04-14 | 12 / 16 | 8 × 2 | 0.68s | $0.0015 |
| development baseline | gpt-5-mini-2025-08-07 | 14 / 16 | 8 × 2 | 2.09s | $0.0052 |
| development baseline | gpt-6-astra | 16 / 16 | 8 × 2 | 1.95s | $0.0511 |
| development ablation | GPT-4.1 Mini + TCG harness | 16 / 16 | 8 × 2 | 1.03s | $0.0035 |
| development ablation | GPT-4.1 Mini + revised prompt | 12 / 16 | 8 × 2 | 0.74s | $0.0018 |
| fresh synthetic challenge | GPT-6 Astra | 24 / 24 | 12 × 2 | 2.11s | $0.0979 |
| fresh synthetic challenge | GPT-4.1 Mini | 13 / 24 | 12 × 2 | 0.90s | $0.0027 |
| fresh synthetic challenge | GPT-4.1 Mini + TCG harness | 24 / 24 | 12 × 2 | 1.80s | $0.0084 |
The harness may use additional model calls, so its improvement is not free. These samples remain too small to establish general superiority.
Store decision pilot
Matched development scenarios, 1 days each, up to 8 actions per day, with the same information limits. This is a short diagnostic, not a 365-day ranking.
| Model | Scenario | Status | Days | Errors / tool calls | Difference vs hold | Accounted USD |
|---|---|---|---|---|---|---|
| gpt-4.1-mini-2025-04-14 | 2201 | complete | 1 | 0 / 7 | $0.00 | $0.0140 |
| gpt-4.1-mini-2025-04-14 | 2202 | complete | 1 | 0 / 6 | $0.00 | $0.0121 |
| gpt-5-mini-2025-08-07 | 2201 | complete | 1 | 2 / 8 | $0.00 | $0.0127 |
| gpt-5-mini-2025-08-07 | 2202 | complete | 1 | 2 / 8 | $0.00 | $0.0119 |
| gpt-6-astra | 2201 | complete | 1 | 0 / 8 | $519.04 | $0.4687 |
| gpt-6-astra | 2202 | complete | 1 | 0 / 8 | $650.14 | $0.4533 |
Incomplete attempts receive no ranking. Original failure records are retained.
store-baseline (unpaced): incomplete · Completed trials: 1 · Errors: 1
store-baseline-paced (3-day chat profile): incomplete · Completed trials: 4 · Errors: 1
Cost and next steps
Pilot cap: $5. Conservatively accounted cost: $2.182.
We are building a shared research harness first. Fine-tuning should target persistent errors demonstrated on fresh evaluation data. Newer models may perform better; matched testing must establish where.
Read the full-year benchmark methodology →Updated: 2026-09-13T19:01:42.213689+00:00Make sense of the evidence.
Answers use the measured benchmark reports and their stated limitations.
Checking assistant availability…