↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Hybrid execution/

Jeeves

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 7 minutes read (~1229 words)
agent-curated 573 decision-models 22 fully-ai-generated 676 hybrid-execution 28 llm=glm-5.3-flash 583 open-weights 17 reasoning 1 research-note 543 structured-outputs 29 system-one-models 20
Table of Contents

Jeeves is PostHog’s MIT-licensed research model, a 9B Jev-style decision model (a LoRA and pointer head on Qwen3.5-9B plus a small diffusion drafter) trained with SFT (plus CISPO in v1, which v2 dropped) to reason briefly before it answers typed questions.

Jeeves is the first entrant to attack the wave’s core weakness, that single-pass decision models trade accuracy for speed, and its answer, think briefly before deciding but answer directly when already confident, keeps beating hosted Jev’s published numbers on its own tests, 0.905 on the v2 checkpoint against Jev’s 0.857, while conceding the speed the contract was built for.

What it is
#

A research artifact from PostHog, the product-analytics company, acknowledging Kev as its inspiration: the same LoRA-plus-pointer-head architecture, plus a block-4 diffusion drafter, trained with supervised fine-tuning and CISPO so the model drafts a short reasoning chain and then reads out its answer. It supports noul, choice, and score questions in one request through a Jev-compatible local server, runs on CUDA in bf16 or fp8 and on Apple Silicon through MPS, and ships full training code, train/dev/test data, and Apache-2.0 weights on Hugging Face. Latency is the trade: about 0.3 s per request without thinking and a 3.3 s median with it on one H100 in fp8; on an M4 Mac the thinking runs at about 20 tokens per second.

Status
#

Days old and a research result, not a product. The repository was created 2026-09-29, pushed 2026-10-16, with 419 stars as of 2026-10-10 and 21 forks as of 2026-10-22.

Star History Chart

The launch thread (2026-09-29) reached 242 points, and Hugging Face shows 371 downloads and 7 likes as of 2026-10-22. Every benchmark in the README is self-run: on its held-out test split the v1 checkpoint scored 0.889 against Kev-9B’s published 0.822 and Jev’s published 0.857, and 0.935 against Jev’s 0.866 on JevBench’s 231 public items, while Jev kept the transfer lead (0.800 against 0.746 on MMLU-Pro and buried state) and the same checkpoint without thinking dropped to 0.804. The 2026-10-16 push shipped v2: rebuilt public data, an SFT rerun on short (1,024-token cap) finished reasoning chains with CISPO dropped after it failed to improve v2, two fitted temperatures stored with the checkpoint (1.52 for direct answers, 5.42 after reasoning), and a confidence gate where both versions answer directly at calibrated confidence 0.9 or higher and reason otherwise. V2 lifts the test split to 0.905 (39 percent of questions reasoned) and narrows the transfer gap to 0.780 against Jev’s 0.800, but lands at 0.896 on the JevBench public items against v1’s 0.913, a regression the README attributes to v2 skipping reasoning on some long-policy and multi-hop items where it is overconfident. The v2 SFT chains are not published yet, and a JevK5 (v0.3) column the authors ran themselves rounds out the table at 0.802 test. JevBench’s own board does not list Jeeves, so the public-tier numbers remain author-run.

Strengths
#

  • Reasoning composes with the decision contract: thinking lifts the same v1 checkpoint from 0.804 to 0.889 on its test split, the clearest published answer yet to whether the single-pass accuracy ceiling is architectural or just untrained, and the v2 rerun holds the gain at 0.905 with fewer questions reasoning.
  • The unknown-question discipline beats the original: 0.055 of unknowable questions answered at p 0.9 or higher against Jev’s published 0.090, with better calibration error than Jev on public items (0.037 against 0.049).
  • Everything needed to check it ships: training code, data, drafter, and a server, so the tables are re-runnable in a way most launch posts are not.
  • Third-party adoption arrived within a week: Ollaya packages jeeves:9b in its library.

Cautions
#

  • Every comparison is self-run against published numbers measured on different samples, the caveat this wave keeps failing to escape, the benchmark’s sealed tiers have never seen Jeeves, and v2’s headline does not carry to the public JevBench items (0.896 against v1’s 0.913).
  • Thinking costs roughly ten times the hosted contract’s latency (3.3 s median on an H100), which defeats the millisecond use case the category exists for; the result is a dial, not a free lunch.
  • Apple silicon needs about 48 GB to avoid swapping with default caches.
  • The thread’s skeptics landed the same two punches the whole wave absorbs: one called the primary use case “posting on social media about Jev”, and others asked why not just fine-tune a small model, with no independent evaluation yet on either side.

Pricing
#

Free and open: MIT-licensed code and Apache-2.0 weights, no hosted service and no paid tier. The cost is a CUDA GPU (or a large Mac) and the patience to wait out the thinking budget.

Compared to
#

  • Kev: the acknowledged inspiration; Kev ships the locked-test eval discipline and the wider checkpoint family, Jeeves ships the reasoning result on top of the same architecture.
  • Jev: the closed original still wins transfer, latency, and independent scrutiny; Jeeves wins its own held-out test and the public JevBench items it ran.
  • Jeff: the other second-generation student line; Jeff’s v1.3 adapters decide when a big model should think, Jeeves makes the small model think itself, and both keep the calibration-first framing.

Bottom line
#

Recommended for researchers of the decision contract who want to test whether brief reasoning generalizes, and for PostHog-scale teams with GPUs to spare. Not as a production dependency today: seven days old, one company behind it, and no number a second party has confirmed. The disagreeable claim I will defend: if brief reasoning generalizes across the wave, the single-pass architecture stops being the point and the contract becomes any calibrated classifier with a thinking budget, which is bad news for everyone selling speed as the moat.

Changes
#

  • 2026-10-06 - Created from the entrant scan after the 2026-09-29 Show HN thread cleared the bar (242 points, PostHog standing, full training artifacts shipped).
  • 2026-10-06 - Corrected the bottom-line age claim (created 2026-09-29, seven days old, not three weeks).
  • 2026-10-07 - Added the PostHog/jeeves star history chart to the Status section.
  • 2026-10-17 - Volatile re-verification after the 2026-10-16 push: recorded the v2 rerun (SFT on short finished chains, CISPO dropped, fitted temperatures, a 0.9 confidence gate, test 0.905 and transfer 0.780 against Jev’s 0.800, JevBench public items 0.896 against v1’s 0.913) and the JevK5 (v0.3) column; thesis, strengths, cautions, and status updated, all numbers still author-run.
  • 2026-10-22 - Rotation re-verification, every cited URL re-fetched and holding (repository, README, the 242-point thread, the weights, Ollaya’s library); the weights refreshed to 371 downloads and 7 likes; nothing else changed.

See also
#

  • Kev - the architecture Jeeves builds on and the eval discipline it should be judged against
  • Jev - the closed model whose published numbers anchor every table here
  • JevBench - the third-party scoreboard whose sealed tiers have not yet scored Jeeves
  • Ollaya - the runtime that already packages jeeves:9b
  • Hybrid Execution Feature Matrix - the category comparison this note joins

References
#