Where quality control lives in the agent workflow: CI gates, dashboards, machine review, human annotation, and the academic studies that benchmark the harnesses themselves.
- Better Harness - Qoder’s MIT harness-engineering platform that judges the agent work loop itself (goals, steps, validation, safeguards, lessons) from project and session evidence, findings carrying acceptance checks, 2.4k stars and no independent footprint yet.
- Crit - Tomasz Tomczyk’s MIT local-first browser surface where agent plans, diffs, running apps, and static HTML get line comments that flow back through a review file any agent can read.
- deepeval - the pytest-style eval framework with roughly fifty judge metrics that gates merges in CI.
- FrontierHarness Eval - the public benchmark that held the model constant and varied nine harnesses in twelve configurations, cost spread 17.5x, vendor-run caveat attached.
- HarnessTax - the UC Berkeley study that held seven models constant across three harnesses on SWE-bench Lite and Terminal-Bench, finding harness swaps cost up to 5x while success barely moves.
- Hunk - Modem’s MIT review-first terminal diff viewer, one annotated stream for the whole agent changeset, judging nothing itself.
- Jevals - OpenLayer’s MIT evals-and-guardrails library that swaps the LLM judge for typed Jev decision models, one calibrated request per trace.
- Kiln - Chesterfield Laboratories’ local-first desktop eval workbench where non-engineers rate outputs and the ratings become judge metrics, fine-tuning sets, and optimization targets.
- Kitaru - ZenML’s Apache-2.0 replay-based eval platform: recorded or imported production sessions re-executed against your next change, human-pinned judgments calibrating the evaluators, the catching cohort becoming the CI gate, 308 stars as of 2026-10-14.
- Langfuse - the MIT-core tracing and eval platform, 35.8k stars, inside ClickHouse since January with the proprietary ee/ split as the caution.
- MLR-Bench - the NUS-led NeurIPS 2025 D&B benchmark scoring AI research agents over 201 workshop tasks with MLR-Judge, a rubric-based two-model LLM panel validated against expert reviewers, whose headline finding is fabricated results in 8 of 10 coding-agent tasks.
- Phoenix - Arize’s OTel-native observability and eval platform, self-hostable under an Elastic license.
- Plannotator - the local review surface that turns your annotations on agent plans and diffs into the agent’s next instruction.
- PR Lens - coldtea’s MIT visualizer that draws every pull request as animated architecture and data-flow diagrams in a PR comment, judging nothing itself.
- ReviewBench - GitHub’s open AI-code-review benchmark: 219 public PRs, a multi-source golden set, Claude Sonnet 5 as judge, and a leaderboard whose scores publish only on maintainer approval.
- Verbatim Citation Gate - the MIT two-stage citation auditor: a zero-token deterministic gate for fabricated quotes, then a skeptical judge that fails closed.
- Workshop - Raindrop’s local debugger where the coding agent reads traces, writes evals, and fixes what fails.
Its members are compared on shared rows in the Evaluation and Review Feature Matrix.
Changes #
- 2026-08-30 - Added deepeval.
- 2026-08-30 - Added Phoenix.
- 2026-08-30 - Added Plannotator.
- 2026-08-30 - Added Workshop.
- 2026-09-05 - Added FrontierHarness Eval.
- 2026-09-16 - Added Langfuse.
- 2026-09-18 - Added HarnessTax.
- 2026-10-06 - Added Jevals.
- 2026-10-06 - Added Hunk.
- 2026-10-07 - Added ReviewBench.
- 2026-10-10 - Added Kiln.
- 2026-10-10 - Added PR Lens.
- 2026-10-10 - Added Verbatim Citation Gate.
- 2026-10-12 - Added Crit.
- 2026-10-13 - Added Better Harness.
- 2026-10-14 - Added Kitaru.
- 2026-10-14 - Added MLR-Bench.