leaderboards on tomrochette.com

LMArena

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

LMArena ranks AI models by blind human preference votes: two anonymous models answer the same prompt, you pick the winner, and Bradley-Terry-style statistics turn millions of those picks into leaderboards spanning text, image, video, vision, search, web development, and agents. Facts below verified as of 2026-09-24; lmarena.ai is a client-rendered app, so vote and model counts beyond the founding paper’s figures could not be read from its HTML.

It is the field’s mood ring: the most cited signal of which model people prefer, and the easiest leaderboard in existence to game.

What it is
#

A website (lmarena.ai), a family of leaderboards (Agent Overall, Text, WebDev, Image, Video, Vision, Document, Search), a WebDev arena (web.lmarena.ai), a blog, and open methodology repositories, run by Arena Intelligence Inc., the company that grew out of the UC Berkeley and LMSYS Chatbot Arena project. The founding paper (arXiv:2403.04132, March 2024) describes the pairwise crowdsourcing method and 240K+ votes at the time; the leaderboard methodology source is published as the arena-rank repository, pushed August 2026. The company raised $100M at a $600M valuation in May 2025, led by Andreessen Horowitz and UC Investments, and labs including OpenAI, Google, and Anthropic partner with it to put flagship models in front of voters. Recent product motion includes AutoEval scores added to the leaderboards (to complement slowly collected human votes), agent leaderboard categories with task costs (August 2026), and a HarnessTax research post (September 2026).

Status
#

The most-cited leaderboard in the field: a critical paper describing Chatbot Arena as “the go-to leaderboard for ranking the most capable AI systems” is itself the best evidence of that status. The blog posts within days of verification, the arenas run continuously, and HN threads routinely open with its rankings as the premise. The company is well capitalized and has converted the academic project into a venture-scale business, which is also the source of its hardest questions.

Strengths
#

Preference at scale is a measurement no benchmark suite replaces: it captures whatever makes people pick one answer over another, including style, format, and thoroughness.
The methodology code and the voting procedure are public, so the statistics are checkable even when the data pipelines are not.
The arena expansion (WebDev, agents, image, video) follows usage: it measures the surfaces people actually use models on.

Cautions
#

The gaming record is documented: Meta’s Llama 4 Maverick episode (an “experimental chat version” tested on the arena that differed from the shipped model) forced a policy update, and LMArena’s own statement conceded “Meta’s interpretation of our policy did not match what we expect from model providers”.
The Leaderboard Illusion paper (arXiv:2504.20879) documents “undisclosed private testing practices” that “benefit a handful of providers”, counting 27 private Meta variants tested before the Llama 4 release and sampling-rate asymmetries favoring closed models.
The sharpest criticism (Surge AI’s “LMArena is a cancer on AI”, 246 points on HN in January 2026) argues the format “rewards superficiality over accuracy” because “the easiest way to climb the leaderboard isn’t to be smarter; it’s to hack human attention span”.
A preference rank is not a capability claim: verbosity and sycophancy win votes that lose tasks, so the number is routinely over-read by headlines.

Pricing
#

Free to use and to vote. No public pricing page exists; the company is venture funded, and its “Try Arena” product surfaces are free at the time of verification. No reader-facing price is stated, so no price history applies.

Compared to
#

Artificial Analysis: controlled first-party evals with published weights; choose it when you need price and speed, the arena when you need preference.
OpenRouter Rankings: revealed preference (spend) versus stated preference (votes).
LLM Stats: benchmark aggregation, which at least labels what it cannot verify.

Bottom line
#

Recommended as the fastest read on which models feel better to people, and as a required filter on any headline of the form “X tops the arena”; not as the number you commit money against. My disagreeable claim: the leaderboard is the least valuable thing the arenas produce, because the vote stream is quietly one of the largest human-feedback datasets ever assembled, and whoever holds it holds a training asset, not just a ranking.

Changes
#

2026-09-24 - Created.

References
#

https://lmarena.ai/ - homepage meta: blind comparison, vote-driven leaderboards across text, image, and code (fetched 200, 2026-09-24)
https://blog.lmarena.ai/ - Arena Intelligence Inc. identity, leaderboard families, 2026 post dates (fetched 200, 2026-09-24)
https://blog.lmarena.ai/how-it-works/ - the vote flow and identity reveal procedure (fetched 200, 2026-09-24)
https://arxiv.org/abs/2403.04132 - the founding paper: method and 240K+ votes (fetched 200, 2026-09-24)
https://arxiv.org/abs/2504.20879 - The Leaderboard Illusion: private testing, 27 Meta variants, sampling asymmetries (fetched 200, 2026-09-24)
https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/ - $100M seed, $600M valuation, investors, Berkeley origin (fetched 200, 2026-09-24)
https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks-gaming - the Maverick gaming episode and LMArena’s policy response (fetched 200, 2026-09-24)
https://surgehq.ai/blog/lmarena-is-a-plague-on-ai - the strongest critical essay on the format (fetched 200, 2026-09-24)
https://github.com/lmarena - methodology repositories including arena-rank, pushed 2026-08-04 (fetched 200, 2026-09-24)
https://web.lmarena.ai/ - the WebDev arena surface (fetched 200, 2026-09-24)

Trackers and Leaderboards Feature Matrix

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

This matrix compares the six members of the Trackers and leaderboards category: sites whose product is a continuously refreshed number about the AI field itself. The members split cleanly on what their number measures: what shipped (AI Release Tracker), what the operator measured (Artificial Analysis), how the field moves (Epoch AI), what public evidence aggregates to (LLM Stats), what people prefer (LMArena), and what people pay for (OpenRouter Rankings).

No row of this matrix crowns a winner, because the rows are different questions; the failure mode is citing a site for a question it does not answer, usually LMArena ranks quoted as capability or OpenRouter tokens quoted as market share.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell traces to a source cited there or in the references.

The matrix
#

	AI Release Tracker	Artificial Analysis	Epoch AI	LLM Stats	LMArena	OpenRouter Rankings
Kind	release timeline plus flat-file corpus	independent benchmarking site and data business	research nonprofit with open datasets	composite aggregator with agent-facing API	blind preference arena	gateway usage rankings
The number measures	launch-day facts: what shipped, when, with which claimed scores	the operator’s own controlled evals, prices, and speed runs	long-run trends: compute, cost, capability over time	public benchmark evidence, normalized with uncertainty	blind human preference votes	tokens processed through one gateway
Run by	To sider ApS, a Danish side project (one visible operator)	venture-backed independent company (AI Grant seed)	501(c)(3) nonprofit, itemized donors, about 50 people	ZeroEval Inc. (self-displayed YC backing)	Arena Intelligence Inc. ($100M seed at $600M)	OpenRouter, a gateway acquired-by-Stripe (announced 2026-08-19)
Coverage	231 releases, 10 labs (as of 2026-08-26)	671 models, 100+ providers, 1,000+ endpoints, chips	3,200+ models since 1950, data centers, chips, companies	398 canonical models, 50+ benchmarks claimed	text, image, video, vision, search, webdev, agent arenas	500+ models via the gateway, 80+ providers
Update cadence	on each release; dataset last updated 2026-08-26	daily changelog; index versions iterate weekly	near-daily data updates, page-stamped	continuously; “within hours of release” claimed	continuous votes; product posts within days	daily UTC buckets, about one day of lag
Methodology published	~ FAQ and data notes, no formal methodology	✓ methodology hub with evaluation lists and weights	✓ transparency page, papers, and data documentation	~ score construction published, details gated	✓ methodology repo (arena-rank) plus founding paper	✓ on-page caveats plus Data API docs
Data access	✓ /models.json and llms-full.txt, free with attribution	~ web charts free, data platform paid, no public API documented	✓ CC BY datasets and a Python client	✓ REST plus 11 MCP tools, free tier	~ arenas open, methodology open, raw votes not re-offered	✓ CC BY 4.0 JSON via Data API, history to 2025-01-01
Reader pricing	free, no ads; Pro instant alerts, price unpublished	free site; enterprise and data-platform tiers, prices unpublished	free, donation funded	free tier; Builder $99/month; Commercial contract	free	free, attribution required
Independence caveat	one operator, one company registration, no second pair of eyes	sells private benchmarking to the labs it ranks (disclosed policy)	consulted for OpenAI and DeepMind, disclosed and audited	paid eval services and a consumer sibling in the same company	lab partnerships and a documented private-testing history	owned by the gateway it measures, being absorbed by Stripe
Verification hooks	✓ full corpus re-downloadable and diffable	~ changelog and methodology public, raw runs private	✓ data, notebooks, and code public	~ API re-queryable, scoring internals partly gated	✓ methodology code public, vote data not re-offered	✓ daily snapshots re-fetchable under CC BY

Reading the matrix
#

The row that sorts the category is “the number measures”, and it doubles as a citation guide: release questions go to the tracker, present-tense quality to Artificial Analysis, trend claims to Epoch, quick composites to LLM Stats, preference to LMArena, spend to OpenRouter. The two most-quoted members are also the two most misquoted: LMArena ranks get read as capability, and gateway tokens get read as market share, when the matrix’s own rows show both measure something narrower.

Verification strength tracks institutional type, not popularity. The nonprofit (Epoch) publishes its funders, its consultations, and its data; the gateway (OpenRouter) publishes its raw daily numbers under CC BY; the side project (AI Release Tracker) lets you diff its whole corpus; while the three venture-scale companies publish methodology and products but keep raw runs, votes, or scoring internals private. If you need to check the work rather than read the work, the checkable columns are the nonprofit’s, the gateway’s, and the side project’s, which is the opposite of what citation frequency would predict.

Every column carries a conflict row, and the conflicts are structural rather than scandals: the benchmark firm sells benchmarking to the benchmarked, the arena partners with the labs it ranks, the aggregator sells evaluation services, the gateway’s data is its own marketing, and the nonprofit consults for the labs. Epoch’s FrontierMath episode and LMArena’s Maverick episode are the two documented failures, and both produced their institution’s strongest disclosure artifacts, which is the pattern worth watching on every refresh of this page.

Choosing from the matrix
#

Needing a dated record of what shipped and what the lab claimed that day: AI Release Tracker, via its flat files.
Picking a model or provider this week and needing quality against price and speed: Artificial Analysis, with the Endpoint Accuracy Index for the cheap-provider question.
Citing how fast the field moves, with downloadable data: Epoch AI.
Wiring leaderboard data into an agent or a script: LLM Stats’s free API and MCP tools, accepting the gated internals.
Sensing which model people prefer right now: LMArena, read as a mood ring and never as a capability claim.
Asking what developers actually spend on: OpenRouter Rankings, quoted with its as-of date and its single-gateway caveat.

Changes
#

2026-09-24 - Created with six columns (AI Release Tracker, Artificial Analysis, Epoch AI, LLM Stats, LMArena, OpenRouter Rankings) when the category was seeded at the owner’s request.

References
#

https://web.archive.org/web/20260826134434/https://aireleasetracker.com/ - AI Release Tracker column: scope, footprint, alerts, operator (fetched 200, 2026-09-24)
https://artificialanalysis.ai/ - Artificial Analysis column: model count, indexes, provider coverage (fetched 200, 2026-09-24)
https://artificialanalysis.ai/methodology/intelligence-benchmarking - Artificial Analysis column: published evaluation weights (fetched 200, 2026-09-24)
https://epoch.ai/about/transparency - Epoch AI column: nonprofit status, funders, consultations (fetched 200, 2026-09-24)
https://llm-stats.com/developer - LLM Stats column: API tiers, MCP tools, quotas (fetched 200, 2026-09-24)
https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/ - LMArena column: funding and company identity (fetched 200, 2026-09-24)
https://arxiv.org/abs/2504.20879 - LMArena column: the private-testing findings (fetched 200, 2026-09-24)
https://openrouter.ai/rankings - OpenRouter Rankings column: methodology, caveats, licensing (fetched 200, 2026-09-24)
https://openrouter.ai/blog/announcements/openrouter-is-joining-stripe/ - OpenRouter Rankings column: the ownership question (fetched 200, 2026-09-24)
https://minimaxir.com/2026/05/openrouter-hy3/ - OpenRouter Rankings column: the free-tier distortion evidence (fetched 200, 2026-09-24)

leaderboards on tomrochette.com

LMArena

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

Trackers and Leaderboards Feature Matrix

The matrix
#

Reading the matrix
#

Choosing from the matrix
#

Changes
#

See also
#

References
#

leaderboards on tomrochette.com

LMArena

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

Trackers and Leaderboards Feature Matrix

The matrix #

Reading the matrix #

Choosing from the matrix #

Changes #

See also #

References #

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

The matrix
#

Reading the matrix
#

Choosing from the matrix
#

Changes
#

See also
#

References
#