This matrix compares the five members of the Evaluation and review category: the pytest-style eval framework, the public multi-harness benchmark, the observability platform, the human annotation surface, and the agent-driven local debugger. Everything below was re-verified against live sources on 2026-09-13. The hybrid machine reviewer that used to sit here, OpenCodeReview, moved to the Code review category when this matrix was created, on 2026-08-30.
The category divides on who judges: the agent itself (Workshop), a metric suite (deepeval, Phoenix), or a human (Plannotator), and mature teams run more than one column at once.
Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified as of the date above. Each column links to the full research note; every cell below traces to a source cited there or in the references.
The matrix #
| Feature | deepeval | FrontierHarness Eval | Phoenix | Plannotator | Workshop |
|---|---|---|---|---|---|
| Kind | pytest-style eval framework | public multi-harness benchmark | observability and eval platform | visual plan and diff review UI | local agent debugger plus eval loop |
| Object judged | LLM app outputs (agents, RAG, chat) | coding-agent harnesses on fixed tasks | traces, spans, experiment outputs | agent plans and diffs | agent traces and code behavior |
| Judge | ~50 LLM-judge metrics plus deterministic checks | fixed 30-task suite, deterministic scoring | LLM evals, code evaluators, human annotations | human annotation | your coding agent writes and runs evals |
| Deployment | Python/TS SDK, CLI | web leaderboard, npx runner, public repo | self-hosted server or Arize AX cloud | local browser app plus hooks | local daemon plus web UI |
| CI gating | ✓ non-zero exit on failure | ✗ a published study, not a gate | ~ experiments, not gate-native | ✗ pre-merge local loop | ✗ named gap in its own thread |
| Harness integration | tracing integrations | 9 harnesses, 12 configs, agent-neutral re-run skill | OTel auto-instrumentation, MCP server | hooks in 9 harnesses | skills and MCP for 5+ agents |
| Team layer | Confident AI platform | ✗ | Arize AX cloud | encrypted links (caveat), Workspaces waitlist | Raindrop Cloud optional |
| License | ✓ Apache-2.0 | ✗ repository unlicensed | ~ ELv2 core, Apache clients | ✓ Apache-2.0 or MIT | ✓ MIT |
| Maturity | mature, 3 years, v4.2 | new, first run 2026-08-31, 82-point HN launch | mature, 4 years, v20.11 | pre-1.0 (v0.27.x), fast churn | pre-1.0, 4 months |
| Pricing anchor | free, platform $200-2,000/mo | free to read, a re-run costs harness tokens | free, AX $50/mo entry | free, Workspaces unpriced | free, Cloud $299/mo |
Reading the matrix #
The CI-gating row is the buy line: deepeval gates merges today, Workshop’s absence of CI is its sharpest recorded criticism, and Plannotator deliberately serves the pre-merge loop instead.
The FrontierHarness column judges a different object than every other column: the harnesses themselves, on public tasks, which makes it evidence for the tool-choice decision and the one column here with a structural conflict of interest, since its publisher sells agent infrastructure and its repository carries no license.
The judge row is the trust line. An agent judging itself (Workshop) closes loops fastest and has the weakest evidentiary standing; metric suites inherit judge variance; Plannotator’s human is the only judge whose judgment you do not have to calibrate.
The license row splits the vendors: MIT and Apache-2.0 columns are safe to embed, and Phoenix’s ELv2 core is the one cell that constrains what a business may do with the code.
Nobody here is redundant: Workshop develops agents, deepeval gates them, Phoenix observes them, Plannotator steers them, FrontierHarness prices the harness choice itself, and the maturity row says a realistic stack starts with the two mature columns and adds the rest as the workflow demands. Machine review of the pull requests themselves now lives in the Code Review Feature Matrix.
Choosing from the matrix #
- Want LLM behavior blocking merges in CI: deepeval.
- Want independent-ish cost and quality numbers before picking a harness: FrontierHarness Eval, vendor-run caveat attached.
- Want production-grade tracing and evaluation under your own infrastructure: Phoenix, accepting ELv2.
- Want machine review of every pull request: the Code Review Feature Matrix, where OpenCodeReview moved.
- Want your annotations to steer a live agent session: Plannotator.
- Want the trace-to-fix loop while developing an agent locally: Workshop.
Changes #
- 2026-08-30 - Created with five columns on the who-judges axis, reduced to four the same day when the OpenCodeReview column was removed.
- 2026-09-05 - Extended to five columns with FrontierHarness Eval.
- 2026-09-07 - FrontierHarness Eval cell corrected to 82 points.
- 2026-09-13 - Corrected the intro sentence that tied the OpenCodeReview category move to the re-verification date; the move happened on 2026-08-30.
See also #
- Retrieval Feature Matrix - the RAG patterns these tools evaluate
- Memory Feature Matrix - the state these tools help verify
- Hamel Husain - the eval-driven method grounding the category
- Executions Feature Matrix - the scheduled runs whose output gets judged
References #
https://github.com/raindrop-ai/workshop - the Workshop column: loop, install, license
https://frontierharness.org - the FrontierHarness column: results, task count, cost spread
https://runta.com/blog/introducing-frontierharness-eval/ - the FrontierHarness column: evaluation design and publisher context