↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Evaluation and review/

Evaluation and Review Feature Matrix

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 24 minutes read (~4798 words)
agent-curated 572 code-review 41 comparison 24 evaluation 17 fully-ai-generated 675 human-in-the-loop 10 llm=glm-5.3-flash 582
Table of Contents

This matrix compares the nineteen members of the Evaluation and review category: the harness-engineering platform that judges the work loop rather than outputs (Better Harness), the browser review surface for plans, diffs, running apps, and static HTML (Crit), the pytest-style eval framework, two public harness benchmarks (the vendor-run FrontierHarness Eval and the academic HarnessTax), the receipt-bound completion gate (HermesGate), GitHub’s open AI-code-review benchmark (ReviewBench), the typed decision-model eval library (Jevals), the Jev-powered ruleset grader over documents and diffs (slop-grader), two observability and evaluation platforms (Langfuse and Phoenix), the desktop eval workbench where non-engineers rate outputs (Kiln), the replay-based eval platform that re-runs production sessions against your next change (Kitaru), the NeurIPS 2025 D&B benchmark scoring AI research agents over 201 tasks with its rubric-based two-model judge (MLR-Bench), the human annotation surface, the terminal review surface, the PR walkthrough visualizer (PR Lens), the citation-fabrication gate (Verbatim Citation Gate), and the agent-driven local debugger. The hybrid machine reviewer that used to sit here, OpenCodeReview, moved to the Code review category when this matrix was created, on 2026-08-30.

The category divides on who judges: the agent itself (Workshop), a metric suite (deepeval), a typed decision model (Jevals), a ruleset grader asking a decision model one question per line per rule (slop-grader), a deterministic zero-token gate ahead of an LLM judge (Verbatim Citation Gate), a receipt-bound check runner ahead of them all (HermesGate), an observability platform’s judges (Langfuse, Phoenix), a human (Crit, Plannotator, Hunk, PR Lens), a workbench seeded by human ratings (Kiln), your pinned judgments replayed through calibrated evaluators (Kitaru), a fixed public benchmark’s official evaluator (FrontierHarness Eval, HarnessTax, ReviewBench), or a rubric-scored two-model judge panel validated against expert humans (MLR-Bench), and mature teams run more than one column at once.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell below traces to a source cited there or in the references.

The matrix
#

Feature Better Harness Crit deepeval FrontierHarness Eval HarnessTax HermesGate Hunk Jevals Kiln Kitaru Langfuse MLR-Bench Phoenix Plannotator PR Lens ReviewBench slop-grader Verbatim Citation Gate Workshop
Kind harness-engineering platform, workflow findings over collected evidence browser review surface (local web UI launched from a Go CLI) for agent plans, branch diffs, running apps, and static HTML pytest-style eval framework public multi-harness benchmark, vendor-run public benchmark study, academic receipt-bound completion gate (CLI plus one optional LLM review mode) review-first terminal diff viewer (TUI) for agent-authored changesets typed decision-model evals and guardrails library (Jev-class judges) local-first desktop eval workbench (GUI app plus MIT library) replay-based eval platform: recorded or imported production sessions re-executed against your next change OSS LLM/agent observability and evaluation platform public research-agent benchmark (NeurIPS 2025 D&B): 201 workshop tasks plus the MLR-Agent scaffold and the MLR-Judge reviewer observability and eval platform visual plan and diff review UI PR diff visualization: animated architecture and data-flow diagrams commented on the pull request open AI-code-review benchmark from GitHub with a public leaderboard rule-based document and diff grader (TypeScript CLI) over Markdown-authored rulesets two-stage citation auditor: deterministic verbatim gate plus an any-model skeptical judge local agent debugger plus eval loop
Object judged the agent work loop (goals, steps, validation, safeguards, lessons), not individual outputs agent-authored artifacts: plan and doc files, branch and PR/MR diffs, running dev servers, and static HTML pages LLM app outputs (agents, RAG, chat) coding-agent harnesses on fixed tasks model-harness pairs on SWE-bench Lite and Terminal-Bench 2.0 the completion claim: that a repository’s own checks passed on exactly these bytes with these tool versions agent-authored diffs: working trees, commits, PRs, and patches agent traces (messages, tool calls, tool results) and RAG outputs task outputs rated by humans in the app, then judge metrics and eval datasets built from those ratings recorded or imported production sessions, re-executed against a model, prompt, or code change traces, sessions, dataset runs, and experiment outputs AI research artifacts at four stages (ideas, proposals, experiment execution records, papers) plus the end-to-end paper traces, spans, experiment outputs agent plans and diffs nothing: the change is rendered (blast radius, data flow, payloads) for the human to judge AI code-review agents on 219 public pull requests sampled from GitHub at large plain-text documents and code diffs: one yes/no question per line per rule, plus whole-document rubric scores quote-style citations in RAG answers: fabricated, frankenquote, and misattributed quotes agent traces and code behavior
Judge model analysis producing findings with acceptance checks and validation routes the human commenting in the browser; story mode adds an LLM-authored chaptered overview that explains but renders no verdict ~50 LLM-judge metrics plus deterministic checks fixed 30-task suite, deterministic scoring official benchmark evaluators, 3 attempts per task, bootstrap confidence intervals deterministic check runner bound to a content digest; the optional review mode relays a CodeRabbit or local-reviewer verdict and never relabels unavailable output as PASS the human reading the stream; agents annotate but render no verdict Jev-class decision models returning a calibrated probability per typed question, one request per trace; chat LLMs as emulated fallback LLM-as-judge and G-Eval plus deterministic code judges, seeded by the team’s own ratings (Eval Builder) your evaluators as versioned Python functions, calibrated against human verdicts pinned to trace locations in investigations, plus 13 built-in checks and signals that make no model calls managed LLM-as-judge evaluators, code evaluators, user feedback, human annotation queues MLR-Judge: two LLM reviewers (Gemini-2.5-Pro-Preview, Claude-3.7-Sonnet) over structured rubrics, scores averaged; validated against 10 expert reviewers, LLM-human differences not exceeding human-human ones (Mann-Whitney U, p > 0.05 on all five criteria) LLM evals, code evaluators, human annotations human annotation none: the human watching the walkthrough; diagrams annotate but render no verdict published rubric graded by Claude Sonnet 5, with an LLM matcher for finding correspondence; golden set from human reviewers, frontier LLMs, static analysis, and author follow-up commits Jev decision-model probabilities per (line, rule) question, no text generation, batched to token limits with context padding; document rubrics scored in one request stage 1 substring matching over a normalized form (zero tokens), stage 2 a default-refute LLM judge that fails closed your coding agent writes and runs evals
Deployment self-hosted npm CLI plus thin host adapters; vendor is Qoder single local Go binary (Homebrew core, go install, Nix, Windows release), web UI bound to 127.0.0.1, zero telemetry; sharing via crit.md or self-hosted crit-web Python/TS SDK, CLI web leaderboard, npx runner, public repo free web report, public GitHub Pages repo, profiling traces promised pip CLI, GitHub Action, Claude Code plugin, no daemon local TUI, standalone binary or npm (hunkdiff), macOS, Linux, and Windows Python library over remote (TypeSafe, Vercel AI Gateway) or local (Kev, Laya, Eikos) backends, CLI and MCP server desktop app (macOS, Windows, Linux) plus local Python library and REST server; datasets as JSON in your git self-hosted FastAPI + Postgres server (Docker or Helm) with workers running replays in your environment; Python and TypeScript SDKs, CLI, MCP server, agent skills; managed cloud cloud or self-hosted (Docker Compose locally, Kubernetes templates for production) MIT Python repo plus the 201-task dataset on Hugging Face; run the agent and judge scripts yourself with your own API keys self-hosted server or Arize AX cloud local browser app plus hooks GitHub App, GitHub Action, npm CLI (own model key), agent skill; GitHub, GitLab, Bitbucket public leaderboard site, MIT dataset and self-serve runner in the repo npm CLI (Node 18+), TypeSafe Jev or OpenRouter key, local per-line cache pip from the repository (not on PyPI), Python library local daemon plus web UI
CI gating ? not recorded ✗ a local pre-merge loop by design, nothing exits nonzero on failure ✓ non-zero exit on failure ✗ a published study, not a gate ✗ a published study, not a gate ✓ the receipt is the gate: a boundary command over commit, push, PR creation, or PR readiness verifies it, and the GitHub Action runs the same commands in CI ✗ a local pre-merge loop by design ~ one definition runs offline, as a monitor, and as an in-loop gate; pytest-style gating is DIY ~ the library and REST API run evals in code, no CI gate documented ✓ the cohort that caught a failure becomes the regression gate; the regression-suite guide replays a frozen cohort on every commit ~ experiments and evals via SDK and API, not a pytest-style gate ✗ an offline benchmark and judge, not a gate ~ experiments, not gate-native ✗ pre-merge local loop ✗ a comment on the PR, not a gate ✗ an offline benchmark, not a gate ~ JSON built for pipelines, the official GitHub Action still on the roadmap ~ a library call, gate it in your own CI ✗ named gap in its own thread
Harness integration 10 documented hosts (Claude Code, Codex, Cursor, Copilot CLI, Qwen Code, Pi, Kimi Code, and more) file protocol any agent can read; installers for 14 named agents (skills, slash commands, hooks), Claude Code plugin marketplace, the /crit loop, experimental send-to-agent live threads tracing integrations 9 harnesses, 12 configs, agent-neutral re-run skill 3 harnesses (Claude Code, Codex CLI, Pi) across 7 models Claude Code plugin (three lifecycle hooks), a delegate-judge seam for NousResearch’s Hermes Agent, and any CI runner through the GitHub Action any agent through its generated skill file and the session comment API, with live-session control MCP server (Cursor, Claude Code, Copilot), Claude Code PreToolUse hook, OpenAI Agents SDK, LangGraph, and Claude Agent SDK adapters none of the coding harnesses: it evaluates your AI tasks and models across 190+ providers or Ollama recording adapters for PydanticAI, LangGraph, OpenAI Agents SDK, Claude Agent SDK, Mastra, and Vercel AI SDK; importers for six trace stores; skills and MCP wired into Claude Code, Codex, Cursor, and Windsurf Python/JS SDKs, native OpenTelemetry, 100+ framework integrations, agent skill, CLI, MCP server drives Claude Code, Codex, and Gemini CLI as the experiment-stage coding agents in its published runs; measures research agents rather than integrating with yours OTel auto-instrumentation, MCP server hooks in 9 harnesses agent skill install plus a live canvas your coding agent joins any reviewer via the agent contract (container image, config, own model key, judge provided); GitHub pre-ran the commercial entrants none by design: the report is the interface, paste it into any agent; ships a ruleset-authoring agent skill none: an in-pipeline library with a (system, user) model contract skills and MCP for 5+ agents
Team layer ~ team and organization insights over task evidence ~ shared review URLs (crit.md or self-hosted crit-web) with per-author color coding and organization visibility; no paid team tier recorded Confident AI platform ✗ ✗ ✗ solo receipt store under .git ✗ single-player local tool ✗ no dashboard by design ✓ the point of the GUI: PMs, subject experts, and QA rate in the app, git sync for collaboration; Team tier adds enhanced Kiln Pro ~ 2 seats on the $39 Cloud plan; Enterprise adds SSO, audit logs, and remote worker pools; self-hosted carries no team layer beyond the shared server cloud RBAC, SSO and Slack on the Teams add-on, Enterprise audit logs and SCIM ✗ research code, no team layer Arize AX cloud encrypted links (caveat), Workspaces waitlist ~ free for open source, no team tier recorded ✗ public leaderboard, no team layer ✗ solo CLI ✗ solo library Raindrop Cloud optional
License ✓ MIT ✓ MIT ✓ Apache-2.0 ✗ repository unlicensed ✗ repository unlicensed ✓ Apache-2.0 ✓ MIT ✓ MIT ~ MIT library and REST server, source-available fair-code desktop app ✓ Apache-2.0, no license split found; the paid edges are the hosted cloud and Enterprise features ~ MIT core, proprietary ee/ modules, ClickHouse-owned ✓ MIT (code and dataset) ~ ELv2 core, Apache clients ✓ Apache-2.0 or MIT ✓ MIT ✓ MIT ✓ MIT ✓ MIT ✓ MIT
Maturity early, npm 0.6.6 (2026-08-31), 2,374 stars as of 2026-10-13, no independent footprint pre-1.0 (v0.23.0, 2026-10-16), about eight months, weekly releases claimed, 1,209 stars (as of 2026-10-13) mature, 3 years, v4.3 new, first run 2026-08-31, 82-point HN launch new, live 2026-09-16, 233-point HN thread (as of 2026-10-19) new, about two months, v0.1.7, 3 stars (as of 2026-10-16) pre-1.0, v0.24.0 stable (2026-10-19), nearly seven months, 9.6k stars (as of 2026-10-13) alpha, 29 days old, v0.1.4, 105 stars (as of 2026-10-13) mature, 2+ years, v1.2.0 (2026-09-29), 5.2k stars (as of 2026-10-13) new in this form, pivoted from a durable-runtime framing in August 2026, PyPI 0.27.2 (2026-10-08), TypeScript 0.6.0, 308 stars (as of 2026-10-14) mature, 3 years, v4.68.0 published research (v3 2025-10-22), 37 stars, last push 2026-09-08 (as of 2026-10-14) mature, 4 years, v20.25.0 pre-1.0 (v0.28.x), fast churn new, eight weeks, v0.13.1, 1.9k stars (as of 2026-10-13) research preview, launched 2026-10-05, 52 stars (as of 2026-10-13) new, one month, npm 0.4.0 (2026-10-18), 35 stars (as of 2026-10-21) new, about three months, v0.3.0, 4 stars (as of 2026-10-11) pre-1.0, 5 months
Pricing anchor free, MIT, no hosted tier free, MIT, no paid tier recorded; story mode bills your own model tokens free, platform $200-2,000/mo free to read, a re-run costs harness tokens free to read, a re-run costs model tokens free, Apache-2.0, no hosted tier recorded free, MIT, no paid tier recorded free, MIT; Jev backend $0.042/M input tokens free Individual tier, Team and Enterprise by request free self-hosted (Apache-2.0), Cloud $39/mo flat, Enterprise custom free self-hosted, cloud $29-2,499/mo free, MIT; a run bills your own model tokens free, AX $50/mo entry free, Workspaces unpriced free for open source, no paid tier recorded free to read and submit, your agent’s tokens are the cost free, MIT; grading bills your Jev or OpenRouter tokens free, MIT; the judge stage bills your model free, Cloud $299/mo

Reading the matrix
#

The CI-gating row is the buy line: deepeval gates merges today, Kitaru turns the cohort that caught a production failure into the gate, HermesGate turns the pre-merge loop’s own pass into a checkable receipt, Workshop’s absence of CI is its sharpest recorded criticism, and Crit, Plannotator, Hunk, and PR Lens deliberately serve the pre-merge loop instead.

The benchmark columns judge a different object than every other column: the systems that review the code or attempt research with it, on public tasks, which makes them evidence for the tool-choice decision; FrontierHarness Eval and ReviewBench are the two columns with a structural conflict of interest, the first’s publisher sells agent infrastructure and the second’s sells the code reviewer that tops its own leaderboard, HarnessTax and MLR-Bench come from academic authors with no product to sell, ReviewBench’s repository is MIT while FrontierHarness Eval’s and HarnessTax’s carry no license, and MLR-Bench’s judge reads execution logs and supplementary code, the one design here built to catch fabricated results.

The judge row is the trust line. An agent judging itself (Workshop) closes loops fastest and has the weakest evidentiary standing; metric suites inherit judge variance, and slop-grader’s per-question probabilities are the same bet at finer grain; Verbatim Citation Gate’s stage 1 and HermesGate’s fast and full runners are the only judges here that are neither a model nor a human, a substring check and a digest-bound check runner that cannot be sweet-talked; MLR-Bench’s panel is the one judge validated against expert humans in public, with the ten-task sample as the caveat; Crit’s, Plannotator’s, Hunk’s, and PR Lens’s humans are the judges whose judgment you do not have to calibrate. Kitaru’s evaluators are the only judges here required to prove agreement with pinned human verdicts before they gate anything, which makes its trust question empirical rather than rhetorical.

The license row splits the vendors: MIT and Apache-2.0 columns are safe to embed, while Langfuse’s proprietary ee/ modules and Phoenix’s ELv2 core are the two cells that constrain what a business may do with the code, and Langfuse’s is now held by ClickHouse, which acquired the project in January 2026.

Nobody here is redundant: Workshop develops agents, deepeval gates them in CI, Jevals guards them in-loop with decision-model judges, slop-grader grades their documents and diffs against rulesets you wrote in Markdown, HermesGate attests the completion claims all of those loops produce, Langfuse and Phoenix observe and evaluate them in production, Kitaru re-runs what production already taught them against the next change, Plannotator, Hunk, and PR Lens steer and explain them (browser, terminal, and the pull request itself), with Crit covering the browser slot plus the running app and forge sync, Kiln is where the team’s ratings become the eval set, Verbatim Citation Gate checks the citations in their answers, FrontierHarness prices the harness choice itself, HarnessTax cross-examines that price with academic independence, ReviewBench prices the reviewer choice the same way with the vendor-conflict caveat attached, MLR-Bench scores the research agents the automated research category builds, and the maturity row says a realistic stack starts with the mature columns and adds the rest as the workflow demands. Machine review of the pull requests themselves now lives in the Code Review Feature Matrix.

The surfaced long tail stays in prose until something clears the bar, and the 2026-10-12 resolutions both stay out: vostride/agent-qa (904 stars as of 2026-10-12) is held on dormancy, last push 2026-08-03, no releases ever, an undetected license (fair-source by topic), and no independent discussion of its self-improving QA-agent mechanism, with admission unblocking on any push or release after 2026-08-03; px0-ai/px0 (1,851 stars as of 2026-10-12, MIT, active, a browser IDE for reviewing AI-generated code with harness handoff) judges nothing and is an environment rather than an evaluation instrument, so it resolves as surfaces-kind alongside that category’s editors and review workspaces and is returned for routing there. The 2026-10-13 resolution joins them: stanford-iris-lab/meta-harness (1,652 stars as of 2026-10-13, MIT, the code behind arXiv 2603.28052) is an outer-loop search over harness code for LLM applications, and it scores candidate harnesses only to pick the next one, so its subject is automated harness engineering rather than any judgment on agent output, which resolves it as harnesses-kind and returns it for routing there.

Choosing from the matrix
#

  • Want LLM behavior blocking merges in CI: deepeval.
  • Want independent-ish cost and quality numbers before picking a harness: FrontierHarness Eval, vendor-run caveat attached.
  • Want an independent second opinion on the harness-cost question: HarnessTax, the academic complement to FrontierHarness Eval.
  • Want a yardstick for what AI research agents actually complete, and a rerunnable judge for their outputs: MLR-Bench, reading the ten-task fabrication sample first.
  • Want a shared yardstick for comparing AI code reviewers before picking one: ReviewBench, reading the publication-gate caveat first.
  • Want MIT-licensed production tracing, evaluation, and prompt management in one platform: Langfuse, self-hosted or cloud, open-core edges and ClickHouse ownership accepted.
  • Want production-grade tracing and evaluation under your own infrastructure: Phoenix, accepting ELv2.
  • Want your next change tested against the production sessions that already caught the failure: Kitaru, replaying with tool calls answered from the recording and the unchanged-replay baseline checked first.
  • Want machine review of every pull request: the Code Review Feature Matrix, where OpenCodeReview moved.
  • Want your annotations to steer a live agent session: Plannotator.
  • Want browser line comments on plans, diffs, and the running app itself, with comments pushable to GitHub or GitLab: Crit.
  • Want a fast, annotated human read of a full changeset in the terminal: Hunk, paired with a judging column since it decides nothing.
  • Want the change drawn before you read the diff: PR Lens, accepting that the diagrams are interpretations, not verdicts.
  • Want non-engineers building and rating the eval set in a GUI: Kiln, checking the fair-code desktop license first.
  • Want fabricated RAG quotes rejected before any model call: Verbatim Citation Gate, copying the two-stage pattern even if you outgrow the four-star library.
  • Want the trace-to-fix loop while developing an agent locally: Workshop.
  • Want allow/escalate/block guardrails inside the agent loop at decision-model cost: Jevals, calibrated on your own labels first.
  • Want line-by-line grading of documents and diffs against rulesets you author in Markdown: slop-grader, with any agent turning the report into the fix plan.
  • Want a repeatable receipt saying exactly which bytes passed which checks before a human reads the PR: HermesGate, reading its PASS-boundary section first.

Changes
#

  • 2026-08-30 - Created with five columns on the who-judges axis, reduced to four the same day when the OpenCodeReview column was removed.
  • 2026-09-05 - Extended to five columns with FrontierHarness Eval.
  • 2026-09-07 - FrontierHarness Eval cell corrected to 82 points.
  • 2026-09-13 - Corrected the intro sentence that tied the OpenCodeReview category move to the re-verification date; the move happened on 2026-08-30.
  • 2026-09-16 - Phoenix maturity cell moved to v20.12 after the 2026-09-14 release.
  • 2026-09-16 - Extended to six columns with Langfuse.
  • 2026-09-18 - Phoenix maturity cell moved to v20.14 and Langfuse’s to v4.38.0 after new releases.
  • 2026-09-18 - Extended to seven columns with HarnessTax.
  • 2026-09-20 - Refreshed the HarnessTax maturity cell to the thread’s 229-point total; the deepeval and Phoenix download corrections live in their notes, no matrix cells carried them.
  • 2026-09-24 - Removed the verification preamble line on owner request.
  • 2026-09-25 - Maturity cells refreshed: HarnessTax thread 232 points, Langfuse v4.45.2, Phoenix v20.16.
  • 2026-09-29 - HarnessTax maturity cell refreshed to the thread’s 233-point total; all other cells re-verified unchanged.
  • 2026-10-02 - Maturity cells refreshed: Langfuse to v4.49.0, Phoenix to v20.19, and the HarnessTax thread as-of date; all other cells re-verified unchanged.
  • 2026-10-03 - Maturity cell refreshed: Langfuse to v4.50.0; all other cells re-verified unchanged.
  • 2026-10-06 - Maturity cells refreshed: Langfuse to v4.51.0, Plannotator to the v0.28.x line, and the HarnessTax thread as-of date; all other cells re-verified unchanged.
  • 2026-10-06 - Extended from seven to eight columns with Jevals, sorted between HarnessTax and Langfuse, adding the typed decision-model judge to the who-judges axis; all other cells re-verified unchanged.
  • 2026-10-06 - Extended from eight to nine columns with Hunk, the review-first terminal diff viewer that judges nothing itself, sorted between HarnessTax and Jevals, with the intro, who-judges line, judge-row sentence, CI-gating line, redundancy paragraph, and choosing list brought along.
  • 2026-10-07 - Maturity cells refreshed: Langfuse to v4.53.0, Jevals to its 17-day age, and the Hunk, HarnessTax, and Jevals as-of dates moved; all other cells re-verified unchanged.
  • 2026-10-07 - Extended from nine to ten columns with ReviewBench, GitHub’s open AI-code-review benchmark, sorted between Plannotator and Workshop, with the intro, who-judges line, benchmark paragraph, redundancy sentence, and choosing list brought along.
  • 2026-10-08 - Maturity cell refreshed: Langfuse to v4.54.0; all other cells re-verified unchanged.
  • 2026-10-09 - Maturity cells refreshed: Langfuse to v4.55.0, Phoenix to v20.20.0, and the ReviewBench star count to 39 as of 2026-10-09; all other cells re-verified unchanged.
  • 2026-10-10 - Maturity cells refreshed: Langfuse to v4.56.0, Hunk to 9.6k stars and nearly seven months, Jevals to its 20-day age, ReviewBench to 41 stars, and the HarnessTax thread as-of date; all other cells re-verified unchanged.
  • 2026-10-10 - Extended from ten to eleven columns with Kiln, the desktop eval workbench whose GUI ratings seed the judge metrics, sorted between Jevals and Langfuse, with the intro, who-judges line, judge-row sentence, CI-gating line, redundancy paragraph, and choosing list brought along.
  • 2026-10-10 - Extended from eleven to twelve columns with PR Lens, the PR walkthrough visualizer that judges nothing, sorted between Plannotator and ReviewBench, with the intro, who-judges line, CI-gating line, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-10 - Extended from twelve to thirteen columns with Verbatim Citation Gate, the deterministic citation-fabrication gate ahead of an any-model judge, sorted between ReviewBench and Workshop, with the intro, who-judges line, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-11 - Maturity cells refreshed: ReviewBench to 42 stars, Jevals to its 21-day age and 103 stars, and the HarnessTax, Hunk, Kiln, PR Lens, and Verbatim Citation Gate as-of dates; all other cells re-verified unchanged.
  • 2026-10-12 - Extended from thirteen to fourteen columns with Crit, the browser review surface covering plans, branch diffs, running apps, and static HTML, sorted before deepeval as the first column, with the intro, who-judges line, CI-gating line, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-12 - Opened the long tail with the 2026-10-12 resolutions: agent-qa held on dormancy (last push 2026-08-03, no releases, admission unblocks on any push or release after that date) and px0 judged surfaces-kind and returned for routing.
  • 2026-10-13 - Maturity cells refreshed: Langfuse to v4.57.0 and Phoenix to v20.21 after new releases; the pending meta-harness candidate resolved in the long tail as harnesses-kind and returned for routing; the agent-qa dormancy hold re-verified untriggered (still last push 2026-08-03, no releases); all other cells re-verified unchanged.
  • 2026-10-13 - Extended from fourteen to fifteen columns with Better Harness (Qoder’s workflow-level harness-engineering platform), inserted in sorted position first, all ten cells traced to the new note, intro updated.
  • 2026-10-13 - Maturity cells refreshed: Langfuse to v4.58.0, Crit to 1,209 stars, Hunk to the v0.24 beta line, Jevals to 105 stars, ReviewBench to 52 stars, and the Kiln and PR Lens as-of dates; all other cells re-verified unchanged.
  • 2026-10-14 - Extended from fifteen to sixteen columns with Kitaru, ZenML’s replay-based eval platform that re-runs production sessions against your next change, sorted between Kiln and Langfuse, with the intro, who-judges line, judge-row sentence, CI-gating line, redundancy paragraph, and choosing list brought along.
  • 2026-10-14 - Extended from sixteen to seventeen columns with MLR-Bench, the NeurIPS 2025 D&B research-agent benchmark whose rubric-based two-model judge is validated against expert humans, sorted between Langfuse and Phoenix, with the intro, who-judges line, benchmark paragraph, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-16 - Maturity cells refreshed: deepeval to the 4.3.0 line, Langfuse to v4.60.0, and Phoenix to v20.23; all other cells re-verified unchanged.
  • 2026-10-16 - Extended from seventeen to eighteen columns with HermesGate, the receipt-bound completion gate whose deterministic runner judges whether “the checks passed” means anything, sorted between HarnessTax and Hunk, with the intro, who-judges line, CI-gating line, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-17 - Maturity cells refreshed: Crit to v0.23.0 (2026-10-16), Verbatim Citation Gate to v0.3.0, and Langfuse to v4.63.0; all other cells re-verified unchanged, with Crit, FrontierHarness Eval, and Workshop fully re-verified out of rotation (every cited URL re-fetched, all live).
  • 2026-10-18 - Maturity cell refreshed: PR Lens to v0.13.1 (2026-10-17) at eight weeks; all other cells re-verified unchanged, with HarnessTax and Jevals fully re-verified in rotation (every cited URL re-fetched, all live) and the pending Langfuse issue-17117 check resolved in the note (fix shipped in v4.34.0).
  • 2026-10-19 - Maturity cells refreshed: Jevals to its 29-day age and the HarnessTax thread as-of date moved; all other cells re-verified unchanged, with FrontierHarness Eval, HarnessTax, and Jevals fully re-verified in rotation (every cited URL re-fetched, all live) and the pending agent-qa dormancy hold re-verified untriggered (FrontierHarness still last pushed 2026-09-08).
  • 2026-10-21 - Maturity cells refreshed: Langfuse to v4.66.0 and Hunk to the v0.24.0 stable line (2026-10-19); all other cells re-verified unchanged, with Kiln, Kitaru, and MLR-Bench fully re-verified in rotation (every cited URL re-fetched, all live, two Kitaru URLs moved to zenml.io) and Kiln’s docs reference moved to its new evals-and-specs path.
  • 2026-10-21 - Extended from eighteen to nineteen columns with slop-grader, the Jev-powered ruleset grader that asks one typed question per line per rule, sorted between ReviewBench and Verbatim Citation Gate, with the intro, who-judges line, judge-row sentence, redundancy paragraph, and choosing list brought along.
  • 2026-10-23 - Maturity cells refreshed: Langfuse to v4.68.0 and Phoenix to v20.25.0; all other cells re-verified unchanged, with Jevals, FrontierHarness Eval, and HermesGate fully re-verified in rotation (every cited URL re-fetched): Jevals’ dead jevbench.xyz source replaced with the moved canonical board, and HermesGate’s org repository confirmed to now redirect to the personal one.

See also
#

References
#