Methodology
No invented index. Real numbers, tracked over time.
Every figure in the benchmark tracker is either a real, independently-sourced quantity (a price, a benchmark score, a usage share, an uptime percentage) or a plain, transparent ratio of two such quantities - quality ÷ price, tokens ÷ context window. There is no weighted composite score in the tracker. Field Tests publish first-party task scores using the separate method below.
Why there's no composite index (the honest version)
This site tried three composite indices before settling on the current approach, and each was killed after a real-data audit, not kept out of caution:
- A weighted linear sum of Quality/Economy/Adoption/Readiness - killed because a linear sum lets a model buy a high score with one strong component regardless of how bad another is.
- A weighted geometric mean of the same four components - fixed the compensation problem, but its dominant component (Quality) turned out to be little more than a normalized restatement of Artificial Analysis's own published intelligence index, so the composite wasn't structurally unique versus a benchmark that already exists.
- A usage-vs-quality divergence index ("usage rank minus quality rank") - genuinely unique (no competitor fuses real usage data with real benchmark data this way), but a real-data test showed it couldn't distinguish a genuinely overlooked model from an expensive one that's rationally avoided on price. Fixing that confound would have meant modeling "expected adoption" from price and availability - reintroducing the exact unauditable, invented-weight problem the first two attempts already failed on.
The pattern across all three: every fix pushed further from "real numbers, sorted" toward "our opinion, disguised as math." That's the lesson this site now runs on.
What we publish instead
- Quality per dollar, over time
- Artificial Analysis Intelligence Index ÷ blended price per 1M tokens (75% prompt / 25% completion), recomputed daily and plotted as a real time series - not a static snapshot like every competing chart in this space.
- Open vs. closed usage share
- Real OpenRouter token volume, split by whether a model's weights are open or closed (classified by publisher - see
server/weights-class.ts), tracked daily. - Provider reliability
- Real per-provider uptime (trailing 30 minutes) and latency, per model, tracked daily - nobody else publishes this as a historical series.
- Real throughput
- Tokens/second from real provider endpoint data, tracked daily rather than benchmarked once.
- Task usage vs. benchmark score (coding and agentic)
- Real per-task usage share from OpenRouter's classified-traffic data, shown alongside (never blended with) the relevant Artificial Analysis domain index.
Data freshness & new models
A daily job at 02:00 UTC re-fetches OpenRouter's models list, usage rankings, benchmark scores, provider endpoints, task classifications, and app rankings - and discovers newly released models automatically before syncing the rest, so a brand-new model is picked up without any manual step. Every response is archived before parsing.
SmophyAI Field Tests · Method v1
How we test models on real work
Field Tests are first-party tests run by Kamil Kępiński through SmophyAI Multi-Chat. They complement the tracker's secondary benchmark, pricing and usage data.
Prompts are written and frozen before any run. Each new prompt gets one screening run; only prompts that separate the models go to the full test. The publish rule is fixed before the full run and does not depend on which model wins.
Every model answers the same frozen prompt, three runs per model. Web search is one setting for all models in a run and is recorded. Web on and web off are reported separately. Model labels are those shown in SmophyAI; each model version remains separate.
Scoring uses hidden tests, answer keys, URL checks and constraint checklists, with a blind human ranking where a task needs judgement. Temperature cannot be set in the product, so we report the median and range of three runs.
For each task and model, we divide the verified score by the maximum and multiply by 100, then take the median of three runs. The Field Test Board reports the mean of those task scores and the supplied range, with separate category scores. Models need at least six tasks across at least three categories before they receive a tier. Models whose means are within five points of each other and whose ranges overlap share a tier; there are no numbered positions.
Tests below the publish rule are listed as "no material difference", without a test page. Published test pages are frozen. Corrections appear visibly on the page with their date and explanation, never silently. Monthly board snapshots never change after the month closes.