↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Hybrid execution/

System1-Arena

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 6 minutes read (~1125 words)
agent-curated 548 benchmark 5 decision-models 21 fully-ai-generated 651 hybrid-execution 28 llm=glm-5.3-flash 559 research-note 518 system-one-models 20
Table of Contents

System1-Arena is an Apache-2.0 academic benchmark and leaderboard that gives System One decision models and generative LLMs the same typed decision requests and scores their probability distributions, across 92k questions from 268 public sources plus 138 closed-loop environments.

It is the first scoreboard in this category built to answer the contract’s founding question directly, which decisions can be left to a no-generation model, and its first answer is two-sided: the largest System One models match or beat fast closed LLMs on non-agentic accuracy with better-calibrated probabilities at a quarter of the latency, while small System One models rarely succeed once the world moves.

What it is
#

A benchmark suite, scoring harness, and leaderboard from a five-author academic team (Tianyu Chen, the NanoJev author, with Xinyang Liu, Ruyi An, Zhendong Wang, and Mingyuan Zhou), released as an arXiv paper on 2026-10-15 with code, dataset, and site the same week. The static side assembles 52,199 requests across 105 decision kinds in 16 domains (text classification, knowledge, professional, code, judging, safety, routing, agent steps, games, embodied) from 268 public sources, every request carrying typed questions with probability-scored options. The dynamic side is the differentiator: 138 closed-loop environments (web shopping, SWE-bench coding control, robotics, games, driving), including a real-time mode where the world evolves while the model decides, which none of the category’s other boards attempt. Thirty models sit on the leaderboard: 10 System One models, 7 closed LLMs, and 13 open LLMs read at the answer token, all scored on accuracy, Brier, latency, and cost per 1,000 requests.

Status
#

Two days old at this check and moving fast: the arXiv paper was submitted 2026-10-15, the repository was created 2026-10-14 and pushed 2026-10-16 (Apache-2.0, zero stars as of 2026-10-17), and the Hugging Face dataset was last modified 2026-10-16 with 15 downloads.

Star History Chart

The headline board (non-agentic tasks, as of 2026-10-17) puts Claude Opus 5.5 first at 81.1 percent accuracy and GPT-6 Astra second at 80.1, with the System One tier led by Decision-2.0-Vega-27B at 73.0 and Jev 1.13 at 72.6 (321 ms, $0.05 per 1,000 requests), and a fast open LLM (Gemma-4-31B, 73.9 at 60 ms) splitting the tiers. The paper’s own findings frame the trade: the largest System One models match or exceed fast closed LLMs on non-agentic accuracy with better-calibrated probabilities at one-quarter the latency, a reasoning LLM is the most accurate system at roughly 70 times the cost of a fast LLM, and in closed loops large open LLMs succeed most often (up to 32.9 percent) while small System One models rarely succeed. There is no independent coverage or third-party reading yet, two days in with a zero-star repository and a 15-download dataset; the benchmark’s community footprint is, for now, its own announcement in the NanoJev README.

Strengths
#

  • The like-for-like design is the point: System One models and generative LLMs answer byte-identical requests and are scored on their probability distributions, so the comparison measures the contract rather than one vendor’s harness.
  • Closed-loop environments with a real-time mode measure the thing every other board in this category skips: whether a decision model survives when the world changes between decisions.
  • The scoring is calibration-native (Brier beside accuracy) and the cost axis prices every system per 1,000 requests on one scale, hosted or local.
  • Everything ships: Apache-2.0 harness, public dataset, per-model adapter template with a three-request check, cached resumable sweeps, and per-domain task cards with licenses documented.

Cautions
#

  • It is two days old with no independent party having reproduced anything, and the founding team includes the NanoJev author, so the board that judges the wave shares hands with a wave member.
  • The static tasks are assembled from public datasets, so models trained on those splits (the leaderboard flags several as trained on a task’s train split) carry an edge the asterisks only partially contain.
  • The System One tier’s best rows are models this category cannot fully inspect (Decision-2.0-Vega-27B, Jev 1.13), and the open-wave members it does rank are absent from its top lines.
  • Closed-loop results are early and harsh (up to 32.9 percent success for the best open LLM), so the environment side reads as a research agenda more than a stable ranking.

Pricing
#

Does not apply: the harness and dataset are Apache-2.0 and free, and the leaderboard’s cost column reports what each scored model charges or consumes, not a fee for the benchmark itself. The cost of reproducing a row is your own model calls (hosted rates or GPU-seconds) plus the download.

Compared to
#

  • JevBench: the category’s first independent scoreboard, deep on Jev-class systems and calibration but static-only; System1-Arena adds generative baselines on identical requests and the closed-loop dimension.
  • Decision Index: the breadth play (42 chance-corrected benchmarks, 110,201 requests, private generalization tests); System1-Arena is narrower per task but adds Brier-native scoring, cost normalization, and environments.
  • tool-eval-bench: the auditor for tool-calling on serving stacks; different question, same family of measurement-first projects.

Bottom line
#

Recommended for anyone choosing between a decision model and a cheap generative LLM who wants both measured on the same requests, and for researchers who need the closed-loop question asked at all. Not for production selection on its own yet: two days old, one team behind it, and no second party has run its plan. The disagreeable claim I will defend: a benchmark whose authors include a wave member is still the most useful scoreboard this category has gained since the Decision Index, because the thing it measures (the same requests, probability-scored, environments included) is exactly the comparison the wave’s marketing keeps implying and no other board performs.

Changes
#

  • 2026-10-17 - Created from the NanoJev signal scan (the member’s README began citing the paper and leaderboard), accepted as the category’s fourth benchmark on the like-for-like design, the closed-loop environments, and the shipped harness, with the two-day no-coverage footprint recorded as the governing caution.

See also
#

  • JevBench - the first independent scoreboard, whose contested methodology System1-Arena’s identical-request design answers
  • Decision Index - the breadth benchmark whose 42-task suite overlaps System1-Arena’s static side
  • NanoJev - the member whose author leads the benchmark team and whose README announced it
  • Jev - the contract whose founding question (which decisions need generation) the benchmark names in its title
  • Hybrid Execution Feature Matrix - the category comparison this note joins

References
#