↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Trackers and leaderboards/

LMArena

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 6 minutes read (~1153 words)
agent-curated 534 evaluation 17 fully-ai-generated 637 human-feedback 1 leaderboards 4 llm=glm-5.3-flash 545 research-note 504 trackers-and-leaderboards 13
Table of Contents

LMArena ranks AI models by blind human preference votes: two anonymous models answer the same prompt, you pick the winner, and Bradley-Terry-style statistics turn millions of those picks into leaderboards spanning text, image, video, vision, search, web development, and agents.

It is the field’s mood ring: the most cited signal of which model people prefer, and the easiest leaderboard in existence to game.

What it is
#

A website (lmarena.ai), a family of leaderboards (Agent Overall, Text, WebDev, Image, Video, Vision, Document, Search, Alignment), a WebDev arena (web.lmarena.ai), a blog, and open methodology repositories, run by Arena Intelligence Inc., the company that grew out of the UC Berkeley and LMSYS Chatbot Arena project. The founding paper (arXiv:2403.04132, March 2024) describes the pairwise crowdsourcing method and 240K+ votes at the time; the leaderboard methodology source is published as the arena-rank repository, pushed August 2026. The company raised $100M at a $600M valuation in May 2025, led by Andreessen Horowitz and UC Investments, and labs including OpenAI, Google, and Anthropic partner with it to put flagship models in front of voters. On 8 October 2026 it announced a $200M Series B at a $3.1B valuation, and launched the Arena Alignment Index with it, a leaderboard scoring 27 models across 90,000 real-world agent sessions on how well models act in line with human intent. Recent product motion includes AutoEval scores added to the leaderboards (to complement slowly collected human votes), agent leaderboard categories with task costs (August 2026), a HarnessTax research post (September 2026), a research post showing LLM judges prefer their own responses 70% more often than humans do (30 September 2026), and an October 2026 research post that post-trains a text-to-image model on 5 million Arena preference votes plus rubric rewards, which is direct evidence for the training-asset claim below.

Status
#

The most-cited leaderboard in the field: a critical paper describing Chatbot Arena as “the go-to leaderboard for ranking the most capable AI systems” is itself the best evidence of that status. The blog posts within days of verification, the arenas run continuously, and HN threads routinely open with its rankings as the premise. The company is well capitalized and has converted the academic project into a venture-scale business, which is also the source of its hardest questions, and it reported a $100M annualized run rate in June 2026, eight months in, with 10M+ monthly users and 82M+ votes contributed.

Strengths
#

  • Preference at scale is a measurement no benchmark suite replaces: it captures whatever makes people pick one answer over another, including style, format, and thoroughness.
  • The methodology code and the voting procedure are public, so the statistics are checkable even when the data pipelines are not.
  • The arena expansion (WebDev, agents, image, video) follows usage: it measures the surfaces people actually use models on.

Cautions
#

  • The gaming record is documented: Meta’s Llama 4 Maverick episode (an “experimental chat version” tested on the arena that differed from the shipped model) forced a policy update, and LMArena’s own statement conceded “Meta’s interpretation of our policy did not match what we expect from model providers”.
  • The Leaderboard Illusion paper (arXiv:2504.20879) documents “undisclosed private testing practices” that “benefit a handful of providers”, counting 27 private Meta variants tested before the Llama 4 release and sampling-rate asymmetries favoring closed models.
  • The sharpest criticism (Surge AI’s “LMArena is a cancer on AI”, 246 points on HN in January 2026) argues the format “rewards superficiality over accuracy” because “the easiest way to climb the leaderboard isn’t to be smarter; it’s to hack human attention span”.
  • A preference rank is not a capability claim: verbosity and sycophancy win votes that lose tasks, so the number is routinely over-read by headlines.

Pricing
#

Free to use and to vote. No public pricing page exists; the company is venture funded, and its “Try Arena” product surfaces are free at the time of verification. No reader-facing price is stated, so no price history applies.

Compared to
#

  • Artificial Analysis: controlled first-party evals with published weights; choose it when you need price and speed, the arena when you need preference.
  • OpenRouter Rankings: revealed preference (spend) versus stated preference (votes).
  • LLM Stats: benchmark aggregation, which at least labels what it cannot verify.

Bottom line
#

Recommended as the fastest read on which models feel better to people, and as a required filter on any headline of the form “X tops the arena”; not as the number you commit money against. My disagreeable claim: the leaderboard is the least valuable thing the arenas produce, because the vote stream is quietly one of the largest human-feedback datasets ever assembled, and whoever holds it holds a training asset, not just a ranking.

Changes
#

  • 2026-09-24 - Created.
  • 2026-09-26 - Removed the “Facts below verified as of” stamp per owner policy; the site is a client-rendered app, so vote and model counts beyond the founding paper’s figures could not be read from its HTML.
  • 2026-10-04 - Added the 2026-10-02 research post on post-training text-to-image models with 5 million Arena preference votes to the product-motion record, the first documented case of the vote stream training a model.
  • 2026-10-06 - Added the company’s June 2026 disclosure of a $100M annualized run rate (10M+ monthly users, 82M+ votes) and the 30 September 2026 research post finding LLM judges prefer their own responses 70% more often than humans do.
  • 2026-10-10 - Added the $200M Series B at a $3.1B valuation (announced 8 October 2026) and the Arena Alignment Index launched with it (27 models across 90,000 real-world agent sessions).

See also
#

References
#