Where quality control lives in the agent workflow: CI gates, dashboards, machine review, human annotation, and the academic studies that benchmark the harnesses themselves.
- deepeval - the pytest-style eval framework with roughly fifty judge metrics that gates merges in CI.
- FrontierHarness Eval - the public benchmark that held the model constant and varied nine harnesses in twelve configurations, cost spread 17.5x, vendor-run caveat attached.
- HarnessTax - the UC Berkeley study that held seven models constant across three harnesses on SWE-bench Lite and Terminal-Bench, finding harness swaps cost up to 5x while success barely moves.
- Hunk - Modem’s MIT review-first terminal diff viewer, one annotated stream for the whole agent changeset, judging nothing itself.
- Jevals - OpenLayer’s MIT evals-and-guardrails library that swaps the LLM judge for typed Jev decision models, one calibrated request per trace.
- Langfuse - the MIT-core tracing and eval platform, 35.5k stars, inside ClickHouse since January with the proprietary ee/ split as the caution.
- Phoenix - Arize’s OTel-native observability and eval platform, self-hostable under an Elastic license.
- Plannotator - the local review surface that turns your annotations on agent plans and diffs into the agent’s next instruction.
- ReviewBench - GitHub’s open AI-code-review benchmark: 219 public PRs, a multi-source golden set, Claude Sonnet 5 as judge, and a leaderboard whose scores publish only on maintainer approval.
- Workshop - Raindrop’s local debugger where the coding agent reads traces, writes evals, and fixes what fails.
Its members are compared on shared rows in the Evaluation and Review Feature Matrix.
Changes #
- 2026-08-30 - Added deepeval.
- 2026-08-30 - Added Phoenix.
- 2026-08-30 - Added Plannotator.
- 2026-08-30 - Added Workshop.
- 2026-09-05 - Added FrontierHarness Eval.
- 2026-09-16 - Added Langfuse.
- 2026-09-18 - Added HarnessTax.
- 2026-10-06 - Added Jevals.
- 2026-10-06 - Added Hunk.
- 2026-10-07 - Added ReviewBench.