↓ Skip to main content
tomrochette.com
  1. Agents/

deepeval

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 4 minutes read (~650 words)
agent-curated 173 evaluation 8 fully-ai-generated 263 llm-as-judge 1 llm=glm-5.3-flash 178 open-source 62 research-note 148 testing 2
Table of Contents

deepeval is an Apache-2.0, pytest-style unit-testing framework for LLM applications that scores outputs with LLM-as-a-judge metrics, G-Eval, task completion, faithfulness, and roughly fifty more, so eval failures gate code like ordinary test failures. Facts below verified as of 2026-09-13.

deepeval is the category’s default answer to the question where do evals live, and its answer is your existing test suite, which is exactly why the open-core line around the Confident AI platform is the decision that matters.

What it is
#

A Python SDK (3.9+) with a beta TypeScript SDK (47 of 49 metrics ported) where evals are pytest tests: deepeval test run exits non-zero on failure, so any CI blocks merges on regression. Metric families cover agentic evaluation (task completion, tool correctness, plan adherence), RAG (faithfulness, contextual precision and recall), multi-turn chat, MCP, multimodal, safety (bias, toxicity, PII), and deterministic non-LLM checks, plus synthetic dataset generation and prompt optimization (GEPA, MIPROv2). Any LLM can judge, including Ollama and vLLM local models, and tracing integrations cover OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, and Pydantic AI. Made by Confident AI, a seed-stage San Francisco company, Apache-2.0.

Status
#

The volume leader of Python eval frameworks: 18,247 stars, 1,922 forks, 583 open issues and PRs as of 2026-09-13. Created 2023-08-10, pushed 2026-09-13, Python v4.2.2 released 2026-09-06, about 3.5 million PyPI downloads a month as of 2026-09-13. Three years old and commercially backed, it is the most mature column in this category, and the 576-issue backlog reads as heavy usage rather than neglect.

Strengths
#

  • Native pytest and CI fit: evals are ordinary tests with a non-zero exit on failure, the integration every rival bolts on afterward.
  • The largest ready-made metric library in the category, each with a self-explanation for debugging.
  • Judge-model freedom including fully local judges, plus deterministic metrics for schema and exact-match checks.
  • Verified traction and sustained release cadence.

Cautions
#

  • Most flagship metrics are LLM-as-judge: non-deterministic, prompt-sensitive, and billed to your judge key on every run.
  • The open-core split is real: comparisons, regression tracking, datasets, and online evals live in the paid Confident AI platform, and deepeval login is “highly recommended” while auto-logging test cases to their cloud unless you opt out.
  • The TypeScript SDK is explicitly not score-parity-verified against Python.
  • Docs-quality criticism at launch has been addressed but 583 open issues signal friction.

Pricing
#

OSS is free, Apache-2.0, fully local with your own judge key. Confident AI platform: Free (2 seats, 5 test runs per week), Starter $200/month, Team $2,000/month, Enterprise custom with on-prem and HIPAA.

Compared to
#

  • Workshop: agent-authored evals inside a local debug loop; choose deepeval when the eval suite must gate CI.
  • Phoenix: observability-first with evals attached to traces; choose deepeval when test-first regression is the job, and note Phoenix’s core is ELv2 while deepeval is Apache-2.0.
  • promptfoo: config-driven declarative evals and red teaming, now part of OpenAI; choose it for YAML-driven cross-model comparison without Python.

Bottom line
#

Recommended for Python teams that want LLM behavior under the same merge gate as unit tests, with deliberate opt-outs configured for the cloud logging. Not for teams that cannot tolerate LLM-judge variance, or that need score parity across Python and TypeScript today.

Changes
#

  • 2026-08-30 - Created as an evaluation research note recording the open-core split caution.
  • 2026-09-08 - Updated the docs reference after deepeval moved its documentation from docs.confident-ai.com to deepeval.com.

See also
#

References
#