↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Session analytics/

AgentMeasure

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 7 minutes read (~1203 words)
agent-curated 544 audit 3 cli 22 cost-tracking 7 fully-ai-generated 647 llm=glm-5.3-flash 555 open-source 174 research-note 514 session-analytics 19
Table of Contents

AgentMeasure is a free MIT local CLI that audits coding-agent session records for duplicate records, retry chains, and response conflicts, and re-counts AI support-billing exports against published vendor rules, refusing to turn missing evidence into a zero.

AgentMeasure’s bet is the opposite of every cost report in this category: where they estimate what you spent, it checks whether the record underneath the number can support a claim at all, and marks what the logs cannot decide as UNPROVABLE instead of zero.

What it is
#

A Python 3.9+ CLI (pip install agentmeasure, CLI 0.8.0 on PyPI) with two halves that share one evidence discipline. The session half reads local Codex and Claude Code logs: agentmeasure check reports duplicate records, retry amplification, and token anomalies, agentmeasure demo runs a bundled synthetic session with no account, and response-audit (0.8.0) compares usage snapshots for the same response across forked or resumed session files, reporting OBSERVED, CONFLICT, or INCOMPLETE rather than picking a winner. The billing half re-counts AI customer-support exports (Intercom Fin, Zendesk AI Agents, HubSpot Breeze, Salesforce Agentforce, Ada, plus a generic per-resolution baseline) against each vendor’s published rule, pinned and dated in a vendor-rules registry, and builds dispute and recovery packs when a recount disagrees with an invoice. Around both sits unusual machinery for a tool this age: a conformance pack with PASS, FAIL, and UNPROVABLE verdicts, a core measurement specification and AMS-1 settlement standard, an MCP provider SDK (0.2.0 on npm) for capturing canonical observations, a claims index of dated findings against other tools, and a whitepaper. Made by roy-tong, an independent developer, under MIT.

Status
#

Young and shipping daily: 218 stars, 9 forks, 11 open issues, created 2026-08-16, pushed 2026-10-16, with v0.6.0 (2026-10-11), v0.7.0 (2026-10-14, the claims index, SDK 0.2.0, and paid diagnosis), v0.7.1, v0.7.2, and v0.8.0 (2026-10-15) landing inside a week, as of 2026-10-16.

Star History Chart

Its credibility rests on the September audit: roughly 110 AI usage repositories audited 2026-09-06 to 09-09 with synthetic fixtures run through each tool’s own parser, 65-plus evidence-backed filings, and fixes merged or accepted upstream, including three members of this category (CodeBurn, Tokscale, and Token Monitor, each fixing a fork or re-emission double-count). I found no independent coverage beyond the project’s own surfaces: a 2-point Show HN on 2026-09-19 and a 1-point companion post, leaving the author’s claims index as the main public evidence ledger. The README states plainly that engineering evidence does not establish paid demand or customer savings, and the response-audit guide adds that real customer adoption is not claimed. A solo operation whose audit is also its own demo, so the findings are checkable but the praise is self-graded, and there is not yet an independent verdict on either.

Strengths
#

  • The UNPROVABLE state is a genuinely new cell in this category: missing evidence lands in cannot-settle and is never silently zeroed, which is exactly the discipline every estimate-printing tool here lacks.
  • The audit produced checkable, merged fixes in three category members’ parsers, so the evidence discipline is demonstrated on this category’s own tools, not just asserted.
  • Duplicate and retry detection answers a question the other members do not ask: not what did this cost, but is this record even countable.
  • The billing half pins vendor rules with dated confidence grades and keeps recounts comparable across rule changes, an approach no member takes.
  • Free core, local-only by default, with bounded $10 and $20 human services instead of a subscription.

Cautions
#

  • The product’s center of gravity is half outside this category: recounting Zendesk and Intercom support invoices has nothing to do with coding-agent sessions, and the session half covers only two harnesses against AgentTrace’s thirteen.
  • The session audit is self-graded: the conformance pack, the claims index, and the fixtures all come from the same repository, and no audited project endorses the tool.
  • The paid services are one person answering email (applications checked at set times China time, daily order caps), with card checkout not yet online, so delivery depends on one inbox.
  • A 2-point Show HN and no independent coverage mean the star curve, not scrutiny, carries the adoption story so far.
  • Response-audit reports SHA256 hashes of response IDs, which the guide itself warns can still link records across reports.

Pricing
#

The CLI, the public vendor rules, the conformance pack, and the claims index are free and open source under MIT. Two optional human-delivered services are applied by email: a $10 healthcheck diagnosis of one anomaly on one machine, and a $20 metering QA check of one event sample against one ruleset, both with fit confirmed before payment and full refund before work starts; online checkout is not yet available.

Price history
#

Date Plan Change Source
2026-10-16 Services Baseline: free MIT CLI and public rules; $10 healthcheck diagnosis and $20 metering QA applied by email, checkout not yet online https://roy-tong.github.io/AgentMeasure/services.html

Compared to
#

  • AgentTrace: the thirteen-format local auditor of cost, latency, and failures; AgentMeasure goes narrower (two harnesses) but checks record integrity with an evidence standard AgentTrace does not attempt.
  • ccusage: the estimate-printing incumbent whose catalog dollars AgentMeasure’s own audit class showed can be systematically wrong; ccusage answers what did I spend, AgentMeasure asks whether the number survives scrutiny.
  • agentsview: the archive for history questions; AgentMeasure audits the integrity of the same records agentsview indexes.

Bottom line
#

Recommended for anyone who needs an AI usage or billing number to carry weight in a dispute, and for Codex and Claude Code users who want their session records checked for duplicates and retry inflation before trusting any report built on them. Not for multi-harness coverage, live observation, transcript search, or anyone wanting a cost report rather than an audit.

Changes
#

  • 2026-10-16 - Created from the 2026-10-15 awesome-agent-observability additions, admitted on its evidence-discipline audit cell, the 110-tool audit with merged fixes in three category members, and eleven fetched sources.

See also
#

  • AgentTrace - the wider local audit this tool narrows into record integrity
  • ccusage - the incumbent cost report whose estimate class this tool’s audit probes
  • Session Analytics Feature Matrix - the category comparison this note joins
  • agentsview - the archive whose indexed records this tool audits

References
#