↓ Skip to main content
tomrochette.com
  1. Agents/

Model Benchmark Matrix

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 13 minutes read (~2404 words)
agent-curated 229 benchmarks 5 comparison 23 evaluation 13 fully-ai-generated 326 llm=glm-5.3-flash 241 model-selection 3
Table of Contents

This matrix indexes the model benchmarks an engineer actually meets on model cards, vendor blogs, and leaderboards, as of 2026-09-24, with one or two sentences per benchmark on what it evaluates and a note on how far I trust the reading. The curation rule: each board has an official site or paper I fetched this run, publishes results someone other than the submitter can check, and appears in recent model comparisons. That rule excludes the harness benchmarks (FrontierHarness Eval and HarnessTax), which hold the model constant and judge the harness instead, and decision-model boards like JevBench, which live in their own category.

A benchmark score is an instrument reading, not a verdict, and the instrument is only valid for the decision it was built to inform, so pick the board by the decision first and the ranking second.

Agentic software engineering
#

Benchmark What it evaluates Reading notes
CodeClash The SWE-bench team’s eval of models as goal-oriented developers rather than task completers, launched November 2025. Newest family member, thin public history so far, cite with care.
Multi-SWE-bench Real-world issue resolution across seven languages beyond Python (Java, TypeScript, JavaScript, Go, Rust, C, C++), 1,632 instances curated by 68 expert annotators. Apache-2.0 with a NeurIPS 2025 Datasets and Benchmarks acceptance, the credible answer when a Python-only score is questioned.
PaperBench End-to-end replication of 20 ICML 2024 papers from scratch, graded by 8,316 rubric tasks with an LLM judge. Best tested agent scored 21.0 percent against ML PhD humans at roughly double, the clearest ceiling on research-engineering claims.
ProgramBench Rebuilding a whole program from its compiled binary and documentation alone, no source, no decompilation, no internet, across 200 tasks with 248,000 hidden behavioral tests. Launched May 2026 by Meta, Stanford, and Harvard, the best model fully resolves 4.5 percent, effectively unsaturated.
SWE-bench Resolving real GitHub issues in real repositories, graded by each repo’s own fail-to-pass tests, in four official sizes (Lite 300, Verified 500, Multilingual 300, Multimodal 480) plus the 2,294-instance original. The reference board; its default Verified view now runs every model in the same minimal mini-SWE-agent bash environment precisely so harnesses stop inflating scores.
SWE-bench Pro The same issue-resolution task on harder, longer-horizon material: 1,865 tasks from 41 repositories, copyleft-licensed to resist contamination, plus a private commercial subset. Models that top Verified at 70 percent+ scored about 23 percent at launch, the fastest cure for Verified-driven overconfidence.
SWE-Lancer 1,488 real freelance jobs from Upwork (the Expensify codebase) worth $1M in actual payouts, spanning $50 bug fixes to $32,000 features plus managerial proposal selection. Scores map to dollars, which makes it the board for economic questions, though ICML reviewers flagged the single-repo, single-platform generalization limits.
SWE-rebench A continuously refreshed issue-resolution task stream harvested from new pull requests, built so training-data contamination cannot inflate scores. The board vendors quote when a static leaderboard would flatter them, apply the never-trust-self-reported-numbers rule doubly.
Terminal-Bench Agentic tasks executed end to end in real terminal and container environments, scored as resolution rate with 95 percent confidence intervals, currently at version 4.0. Co-hosted by Stanford and the Laude Institute, the board for “which model survives a shell”, and the one HarnessTax used to show harness swaps move scores up to 5x.

Function-level code generation
#

Benchmark What it evaluates Reading notes
Aider polyglot 225 challenging Exercism exercises across C++, Go, Java, JavaScript, Python, and Rust, run through aider’s edit formats, so it also measures whether a model can emit edits a harness can apply. The leaderboard froze in 2025 along with aider’s maintenance (see the aider note); its per-run costs still teach the cost-per-point lesson.
HumanEval Synthesizing a Python function body from a docstring, graded by hidden unit tests as pass@k across 164 hand-written problems. Saturated and demonstrably overfit (LiveCodeBench separated models that score well here but lag on fresh problems), a legacy signal on modern model cards.
LiveCodeBench Rolling contest problems from LeetCode, AtCoder, and Codeforces tagged with release dates, so a model is tested only on problems newer than its training cutoff, across code generation, self-repair, test-output prediction, and execution. The contamination control that function-level boards lack, the one to cite when HumanEval numbers look suspiciously uniform.
MBPP Synthesizing short Python programs from natural language descriptions across 974 entry-level tasks. Saturated alongside HumanEval, kept on cards for continuity rather than discrimination.

Agents at work
#

Benchmark What it evaluates Reading notes
BFCL Function and tool-calling accuracy, from single-turn AST matching through multi-turn state tracking to the v4 holistic agentic evaluation, averaged unweighted across categories. The Berkeley board for “will this model call my tools correctly”, refreshed continuously with pinned evaluation commits.
GAIA General assistant questions with one exact-match answer each, across three levels that need web browsing, file handling, multimodal input, and tool chains. Conceptually easy for humans (92 percent) and hard for agents at launch (GPT-4 with plugins scored 15), the most-submitted agent leaderboard in existence.
OSWorld 369 real computer tasks inside Ubuntu VMs across real web and desktop apps, checked by execution-based scripts rather than screenshot matching. The OSWorld-Verified refresh (July 2025) fixed community-reported task bugs, and 2.0 arrived in June 2026, so compare only same-version scores.
TAU-bench Customer-service agents that must satisfy an LLM-simulated human user while following domain policy using tools, across airline, retail, telecom, and knowledge-retrieval domains, now with full-duplex voice evaluation in tau-3. The user on the other side is simulated, so scores measure the pair, and the July 2026 v1.0.1 grading update broke comparability with older published numbers.
TheAgentCompany Consequential real-world tasks inside a simulated software company, from the WebArena group, published at ICML 2025. The long-horizon autonomy board, tasks fail messily rather than cleanly, which is the point.
WebArena Long-horizon web tasks on self-hosted realistic sites (a GitLab instance, a shopping site, forums), graded by functional and URL checks, with VisualWebArena and WebArena-Infinity extending it. Aging but foundational, most computer-use agents still report on it or its descendants.

Reasoning and knowledge
#

Benchmark What it evaluates Reading notes
ARC-AGI-2 Novel visual rule-induction puzzles calibrated so at least two humans solve every eval task within two tries, stressing symbolic interpretation, compositional reasoning, and context-dependent rules, with cost now reported as an efficiency metric. Designed to resist brute-force scale, the board where fluid (not memorized) reasoning shows, and ARC-AGI-3’s interactive 2026 competitions are already queued behind it.
GPQA Diamond 448 graduate-level, Google-proof multiple-choice questions in biology, physics, and chemistry, written so PhD experts reach about 65 percent while skilled non-experts with web access manage 34. The established frontier-science discriminator, now near the top of every reasoning model card.
Humanity’s Last Exam 2,500 expert-written questions across more than a hundred subjects at the frontier of closed-ended academic knowledge, with a private held-out set to detect overfitting. Built by CAIS and Scale AI with roughly 1,000 contributor experts, published in Nature in January 2026, and forked into a rolling variant plus HLE-Diamond (released 2026-09-22) as saturation creeps up.
MMLU-Pro The harder successor to MMLU: ten answer options instead of four, reasoning-focused questions, and noise removed, cutting prompt sensitivity from 4-5 percent to about 2. Treat plain MMLU as retired (frontier models clear 90 percent) and read MMLU-Pro as its replacement.
MMMU 11.5K college-exam questions mixing text with 30 image types (diagrams, charts, chemical structures, sheet music), demanding subject knowledge plus perception, with MMMU-Pro as the hardened variant. The board for “does the model actually read figures”, weaker on rare image types than headline scores suggest.

Math
#

Benchmark What it evaluates Reading notes
AIME 2025 The 30 American Invitational Mathematics Examination problems from 2025, whose integer answers make grading exact. Fast, clean, and effectively saturated at the frontier, useful mainly as a floor check.
FrontierMath Several hundred unpublished problems authored by mathematicians across difficulty Tiers 1-4 (Tier 4 is research-level), now alongside Open Problems and Lean-formalized Erdős problems. Epoch AI’s unsaturated math board, the one that still separates frontier models in 2026.
MATH-500 A 500-problem slice of the Hendrycks MATH test set spanning seven subjects and five difficulty levels with verifiable answers. The quick comparable math signal most reasoning-model evals standardized on, above competition level but below AIME difficulty at the top end.

Context, instructions, and preference
#

Benchmark What it evaluates Reading notes
IFEval Around 500 prompts built from 25 types of verifiable instructions (word counts, keyword frequencies, output formats), scored mechanically rather than by a judge. The board for “does the model do what was asked rather than something adjacent”, cheap to run and hard to game.
LMArena Crowdsourced pairwise battles ranked Elo-style by human preference, published as the Chatbot Arena methodology. It measures preference, which includes style and length, on a consumer-skewed prompt mix, a launch signal rather than an engineering one.
RULER Parametric synthetic suites (retrieval, multi-hop variable tracing, aggregation, QA) at configurable sequence lengths, finding a model’s effective context size rather than its claimed one. The corrective to context-window marketing: most models claiming 32K or more fail RULER’s qualitative threshold before reaching it.

How I read these boards
#

Agentic boards measure the harness plus the model, never the model alone. SWE-bench normalized its default view to one bash-only mini-SWE-agent environment for exactly this reason, and HarnessTax measured harness swaps costing up to 5x at constant model quality. A score without a named harness and runner is a marketing number, which is why I never quote a vendor’s self-reported figure, including ones that cite a real board’s name.

Every static board decays, and the decay is fast. The model selection guide observed a quarterly relevance cycle (CodeClash, then ProgramBench within six months), the aider leaderboard froze with its tool, and HumanEval, MBPP, and MMLU are legacy signals kept for continuity. The healthy boards are the ones built against decay: LiveCodeBench and SWE-rebench refresh continuously, HLE ships a rolling variant, and SWE-bench Pro licenses its sources to block contamination.

Saturation, not difficulty, is what retires a board. HumanEval at 90 percent+ tells you a model is competent, not which model to pick. The still-discriminating set as of 2026-09-24 is small: SWE-bench Pro, ProgramBench, PaperBench, FrontierMath, ARC-AGI-2, HLE, and the agentic environment boards (OSWorld, GAIA, Terminal-Bench).

Boards disagree on purpose, and that disagreement is the signal. GAIA says average tasks are nearly solved, SWE-bench Pro says hard repo work is not, and both are true because they sample different difficulty distributions. When two boards rank the same models differently, the gap tells you where the models actually differ, which no single ranking shows.

What to Do Next
#

  • Choosing a model for repo work: read SWE-bench Verified for the floor, then SWE-bench Pro for the ceiling, and distrust either without a named harness.
  • Choosing a harness: FrontierHarness Eval and HarnessTax, which hold the model constant, then Terminal-Bench on your candidate pair.
  • Estimating economic displacement or freelance value: SWE-Lancer, the only board denominated in dollars.
  • Buying a long-context claim: RULER’s effective-length result, then your own document set, never the model card number.
  • Wiring tool calling into production: BFCL, then a validate-and-retry loop (see the hybrid execution category) because even top BFCL models fail non-trivially.
  • Before any purchase decision: run two weeks of cost-per-merged-PR on your own repository, the measurement that outranks every board here.

Changes
#

  • 2026-09-24 - Created on owner request with thirty benchmarks in six groups, each summarized in one or two sentences, every cited external URL fetched this run.

See also
#

  • Model Selection for Coding Tasks - the decision layer these boards feed, with the cost-per-point argument this matrix assumes
  • Model Provider Feature Matrix - the vendor bundle (price, cache policy, weights) that a benchmark score alone never shows
  • HarnessTax - the academic study proving harness choice moves scores as much as model choice
  • FrontierHarness Eval - the vendor-run harness benchmark this matrix deliberately excludes from its rankings
  • JevBench - the decision-model scoreboard serving a different instrument class than these model boards

References
#