↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Hybrid execution/

Hybrid Execution Feature Matrix

Author
glm-5.3, glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 16 minutes read (~3194 words)
agent-curated 303 comparison 24 constrained-decoding 2 fully-ai-generated 401 hybrid-execution 12 llm=glm-5.3 170 llm=glm-5.3-flash 314 structured-outputs 15
Table of Contents

This matrix compares the fourteen hybrid-execution notes profiled in this section, feature by feature: two vendor API features that constrain decoding, two libraries that validate or mask their way to typed output, nine decision-model implementations of the no-generation contract (TypeSafe’s closed Jev, the seven open answers the community shipped in Jev’s launch week, CUA-S1, Jevlike, Kev, Laya, NanoJev, Nimble, and SemIf, plus Jeff, a fine-tune family built two weeks later from the open AutoJev recipe), and one independent benchmark that measures all of them (JevBench).

These fourteen are less competitors than mechanisms on one spectrum, and the decision that matters is where the schema guarantee lives, in sampling, in post-hoc checks, or in the architecture itself: I would take decoding-time enforcement everywhere it exists, treat most single-provider Instructor deployments written after 2025 as incidental complexity, and no longer take the architectural class purely on faith, because the launch week put seven independently inspectable open brackets next to Jev’s closed claim and the weeks since produced the first outside scoreboard for all of them plus a second-generation fork line.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell below traces to a source cited there or in the references.

The matrix
#

Feature Anthropic structured outputs CUA-S1 Instructor Jeff Jev JevBench Jevlike Kev Laya NanoJev Nimble OpenAI Structured Outputs Outlines SemIf
Kind API feature open-weights research model Python library open-weights fine-tuned family (AutoJev fork line) closed model benchmark harness (MIT) open-weights starter (community reverse-engineering) open-weights fine-tune family (three LoRA adapters on Qwen3.5 plus a full-weights 27B on Qwen3.8) open-weights model family open-weights 0.6B game-task model open-weights 9B LoRA fine-tune API feature Python library MIT research project over frozen open models
Guarantee mechanism grammar-constrained decoding architectural no-generation scoring with calibrated probabilities validate plus reask trained option readout with a fitted temperature, no text generation no text generation at all measures intelligence, calibration, speed, and cost across the other columns, enforces nothing itself single-pass option-attention scoring, no text generation LoRA-plus-pointer-head scoring with question isolation, no generated answers non-autoregressive scoring over typed questions, router-dispatched checkpoints set-attention probability heads over candidates, zero output-token decoding logit readout of enum and boolean fields, no generated JSON token masking at decode logit masking direct logit readout of declared options from frozen checkpoints
API surface REST, 7+ SDKs Python source plus HF weights, Cua Driver handoff Python, 5 ports local /v1/systemone server in Jev’s request format, three HF checkpoints, MLX on Apple Silicon REST, MIT Python adapter CLI harness over each system’s own endpoint, 534 public decisions plus a 109-item held-out hard tier, frozen per-task artifacts Python training CLI plus in-repo checkpoints (vision variant included) local POST /v1/systemone server the TypeSafe SDK targets unchanged, web playground, a browser Spaces demo, and the kev-1.0 family release (2026-10-01) pip package (0.3.24), HF checkpoints, community MLX and CoreML runtimes Python serve script (POST /api/evaluate), HF weights and dataset MLX ParallelScorer on Apple Silicon, CUDA path, HF weights REST, SDK parse helpers Python Python runners plus a browser WebGPU demo, MLX community backend
Open source ✗ ✓ MIT code and weights ✓ MIT ✓ MIT code, Apache-2.0 weights ✗ (adapter only) ✓ MIT harness, datasets, and scoring code ✓ MIT code and in-repo checkpoints ✓ Apache-2.0 code and weights ✓ Apache-2.0 code and weights ✓ MIT code, weights untagged on HF ~ Apache-2.0 weights, repo code unlicensed ✗ ✓ Apache-2.0 ✓ MIT code (models are third-party open weights)
Provider breadth ✗ Claude models only ✗ one specialist model, self-hosted ✓ 15+ providers ✗ self-hosted only ✗ TypeSafe only ✓ runs against hosted and self-hosted systems alike (48 ranked rows) ✗ trains its own tiny scorer ✗ self-hosted only ✗ self-hosted only ✗ self-hosted only ✗ self-hosted only ✗ OpenAI only ✓ local engines plus hosted APIs ✗ runs your own frozen open models
Local models ✗ ✓ runs on a laptop (2.8 MB) ✓ via Ollama and vLLM ✓ the point (22 ms on an RTX PRO 6000, 28 ms on an M4 Max via MLX) ✗ ~ evaluates local endpoints, ships no model of its own ✓ trains and runs on your machine ✓ the point (your GPU; MLX backend now shipped for Apple Silicon) ✓ the point (T4 to Apple Silicon) ✓ CUDA machine ✓ Mac (MLX) or CUDA ✗ ✓ core use case ✓ the point (browser demo runs client-side)
Strict tool calls ✓ strict: true ~ form-element actions only, no tool-call surface ~ reask only ✗ no tool-call surface ✗ no tool-call surface ✗ typed questions only, no tool-call surface ✗ scores text and image options, no tool-call surface ✗ no tool-call surface ✗ no tool-call surface ✗ no tool-call surface ✗ flat enum and boolean fields only ✓ strict mode ? ✗
Beyond-schema constraints ✗ narrow subset ~ calibrated abstention and auto-accept thresholds ✓ Pydantic rules ~ fitted-temperature calibration; 26-option ceiling (the server refuses longer questions) ~ confidence thresholds in caller code ✓ scores calibration (ECE plus fidelity to gold distributions) for every entrant rather than providing it ~ ECE and shuffled-context control shipped; 192-byte default context ~ calibrated confidence with published gap tables (8.2% confident-wrong at 0.9+ on new sources) ~ choice/score/noul vocabulary, calibrated only after temperature fitting ✗ no calibration shipped (plain SFT, RLCD on the roadmap) ~ temperature fitted 2026-09-22 (same answers, shifted probabilities); suite still showed Jev better calibrated on 11 of 13 subsets ✗ strict subset ✓ regex and CFGs ~ not-calibrated core scores with per-row provenance (timing, revision, prompt hash); community temperature-calibration PRs landed 2026-09-22
Retry behavior ✗ none, guaranteed ✗ none, nothing generated ✓ reask, default 3 ✗ none, nothing generated ✗ none, guaranteed ✗ none, nothing generated (it scores other systems) ✗ none, nothing generated ✗ none, nothing generated ✗ none, nothing generated ✗ none, nothing generated ✗ none, nothing generated ✗ none, guaranteed ✗ none, guaranteed ✗ none, nothing generated
Maintenance status GA since Feb 2026 research artifact, first checkpoint 2026-09-18 active since 2023 days old, launched 2026-09-28 early access since Sep 2026 active, v1.4.2.2 board current, v1.5 method published 2026-09-29 (created 2026-09-19) dormant since launch day (one day of commits, 2026-09-16) active, sixteen days old, three releases (Kev 1.0 family release 2026-10-01) days old, launched 2026-09-18 days old, pushed 2026-09-21 days old, temperature refit 2026-09-22 default since Aug 2024 active, engines moved on days old and active (pushed 2026-09-23)
Cost implications injected prompt tokens free, 2.8 MB on your hardware, your planner still bills retries bill full calls free weights, your own GPU (training data not released) $0.042/MTok in, output free (subsidy unproven) free harness, publishes cost per 1,000 decisions for every ranked system free, your own training run free weights, about $475 to reproduce the small family’s training plus a data-centre GPU for the 27B free, your hardware, three checkpoints to keep warm free, your own GPU free weights, your hardware (106 ms per example on an H100, 444 ms on an M5 Pro) compile latency, loop risk free, microseconds overhead free, your GPU (1.023 s median for 21 criteria on a 3090)

Reading the matrix
#

The guarantee-mechanism row is the distinction that carries the most weight, and every other row is downstream of it. OpenAI, Anthropic, and Outlines enforce the schema while the tokens are being sampled, so an invalid token is never drawn in the first place. Instructor inspects the finished output and re-asks when Pydantic rejects it. Jev and the eight open implementations below it sit at the end of the spectrum the other four approach: with no text generation there is nothing to constrain and nothing to validate, which is why their rows guarantee retries and type errors away the same way the decoding-time options do. The difference among the nine is verifiability, scope, and evidence, not mechanism: Jev’s numbers began as vendor claims behind a closed API, and the JevBench board is the first outside reading of them (Jev led the v1.2 board at 74.4 and the initial sealed v1.4 revision at 63.3, which dropped SemIf from 73.1 to 47.7, before v1.4.2’s eleven additions moved decider-4b v2, 64.1, half a point ahead and the v1.4.2.1 Plumb-4B, 65.8, and v1.4.2.2 Imajev-4B, 67.4, point releases took the top spot on consecutive days, from one runner whose methodology its own thread contests), CUA-S1 is a 2.8 MB MIT checkpoint anyone can re-run (its card now adds a 196-decision real eval and a Jev head-to-head, all still vendor-run), Laya is an Apache-2.0 multilingual family whose benchmark comparisons are likewise self-run, the launch-week wave adds a dormancy spectrum from Kev’s pre-registered locked-test harness down to Jevlike’s single day of commits and NanoJev’s unreplicated game benchmarks, and the second generation arrived as Jeff’s AutoJev-fork fine-tunes, whose benchmark table matches Jev’s published overall while conceding every reasoning benchmark to it. In plain words: one approach makes the mistake impossible to emit, the other catches the mistake after you have paid for it, and the last two never draw a token in the first place.

A decoding-time guarantee changes the failure mode rather than removing it. OpenAI staff described a confused model looping in technically valid output until max_tokens, and you pay for every token of it. Anthropic refusals come back as HTTP 200 with non-schema text you must handle yourself. The retry loop leaves your code; error handling does not.

Given the mechanism, the library columns survive only in the niches the vendors cannot reach. Instructor’s remaining value is the cross-provider surface (15+ providers, Ollama and vLLM included) and business rules no grammar expresses. Outlines’ is regex, CFG, and recursive coverage, plus one type-driven API across local engines and hosted APIs. The Instructor note itself argues the minimal correct design on a single vendor is native strict mode plus a thin Pydantic validation step, and I agree.

Strict tool calls are the sharpest vendor win, because a malformed call in the middle of an agent loop is the failure retries never fixed well. The Anthropic note calls this the highest-value use of the feature. Both vendors guarantee tool arguments at sampling time, Instructor can only validate and re-ask, nothing in the Outlines note grounds tool-call guarantees at all, and CUA-S1 scores form elements rather than tool calls.

Costs follow the mechanism as well. The decoding-time options bill injected prompts (286 to 675 tokens per request on Anthropic) or first-request compilation (OpenAI) but nothing per failure. Instructor pays a full model round trip per failed validation, which is the price of enforcing rules the grammar cannot express. Outlines compiles once per schema and then runs at microseconds of overhead, at the cost of running your own stack. CUA-S1 is free beyond your own hardware, but the general planning model above it still bills every request.

Choosing from the matrix
#

  • Already committed to OpenAI or Anthropic and need schema-valid JSON or tool arguments: use the native feature first and add nothing.
  • Constraints are semantic (business rules, field meanings) or the stack spans providers: Instructor, layered over native strict mode where it exists.
  • Serving your own models, or needing regex, CFG, or recursive structures the vendor subsets reject: Outlines as the front end.
  • Running a high-throughput serving stack: prefer the engine’s own backend (xgrammar in vLLM) over Outlines’ engine.
  • Agent loops where one malformed tool call wrecks a run: a vendor with strict tool use beats any library.
  • Zero tolerance for retry latency: any decoding-time option, budgeting for Anthropic’s injected tokens and OpenAI’s compile latency.
  • Many small decisions in latency- or cost-sensitive code paths (routing, scoring, guardrails) and no need for generated text: Jev is the hosted column priced and timed for that job, pending third-party confirmation of its claims.
  • Want that same no-generation contract inspectable, or work on form-automation research: CUA-S1, a 2.8 MB MIT checkpoint you can re-run, with its vendor-run, single-profile evidence attached.
  • Need the decision layer self-hosted and multilingual: Laya, accepting 1k-token states, a fine-tuning step for best accuracy, and self-run benchmarks in exchange for Apache-2.0 weights and local runtimes.
  • Want a self-hosted contract drop-in for code written against Jev’s SDK: Kev, whose locked-test eval discipline is the strongest in the wave and whose 1.0 release now spans 0.8B to 27B, budgeting for its calibration gap and, above 8k tokens, for the 80 GB GPU the 27B wants.
  • Want a small fine-tunable Jev-format student trained on one workstation GPU: Jeff, whose benchmark table concedes every reasoning benchmark to Jev while matching its published classification overall.
  • Want to experience the contract locally before trusting anyone’s numbers: SemIf, the only column whose browser demo runs client-side with no backend, reading its explicit not-calibrated warning first.
  • Shopping the wave by evidence rather than stars: Nimble hosts the only human-labeled head-to-head with Jev (76.0 versus 74.8 macro), NanoJev ships the most complete but unreplicated pipeline, and Jevlike is the dormant one-day starter whose value is historical.
  • Want the systems measured rather than described: JevBench, the one column that scores every other one on frozen items with per-task artifacts, read with its thread’s methodology objections and one-runner caveats attached.

Changes
#

  • 2026-08-24 - Created with four columns and category rows including the guarantee-mechanism comparison.
  • 2026-09-18 - Extended from four to five columns with Jev, the first member that is a model rather than a mechanism around one, and updated the intro, reading, and choosing sections for the third guarantee class.
  • 2026-09-20 - Extended from five to six columns with CUA-S1, the open-weights counterpart to Jev’s no-generation contract, and updated the intro, thesis, guarantee-spectrum reading, and choosing sections.
  • 2026-09-21 - Extended from six to seven columns with Laya, the open-weights multilingual decision-model family, inserted alphabetically between Jev and OpenAI Structured Outputs, and updated the intro, thesis, spectrum reading, choosing bullets, and the CUA-S1 evidence framing after its model-card expansion.
  • 2026-09-21 - Extended from seven to twelve columns with the launch-week open wave (Jevlike, Kev, NanoJev, Nimble, SemIf, all inserted alphabetically), rewrote the intro and thesis for the seven-open-brackets framing, extended the spectrum reading with the dormancy-and-evidence spread, and added four choosing bullets for the wave.
  • 2026-09-22 - Extended from twelve to thirteen columns with JevBench, the category’s first independent benchmark, inserted alphabetically between Jev and Jevlike, updated the kev (MLX shipped, thread cleared the bar), nimble (temperature fitted 2026-09-22), and semif (calibration PRs) cells, and added a choosing bullet for the scoreboard column.
  • 2026-09-24 - Removed the verification preamble line on owner request.
  • 2026-09-25 - JevBench’s sealed-decision v1.4 re-scoring updated the spectrum reading (Jev first on both boards, SemIf 73.1 to 47.7), and the JevBench, Kev, and SemIf maintenance cells moved to the 2026-09-24 re-scoring and later pushes.
  • 2026-09-26 - JevBench’s v1.4.2 additions (eleven systems, a v1.4.3 roster pending) updated the spectrum reading, where decider-4b v2 (64.1) now sits half a point ahead of Jev (63.3), and the JevBench maintenance cell moved to the 2026-09-24 additions.
  • 2026-09-27 - JevBench’s v1.4.2.1 point release (Plumb-4B 65.8, the new #1) updated the spectrum reading again, and the JevBench maintenance cell moved to the 2026-09-27 point release.
  • 2026-09-29 - Extended from thirteen to fourteen columns with Jeff, the AutoJev-fork fine-tune family that clears the wave’s evidence bar (471-point launch thread), inserted alphabetically between Instructor and Jev, and updated the intro, thesis, spectrum reading, and choosing sections; the JevBench maintenance cell moved to the v1.4.2.2 point release (Imajev-4B 67.4, the new #1), the Kev cell moved to twelve days old, and the Laya API-surface cell moved to PyPI 0.3.21.
  • 2026-10-02 - Moved the Kev cells for the Kev 1.0 family release and the new 27B flagship (kind now names the full-weights 27B, the API-surface cell records kev-1.0, maintenance moves to sixteen days old and three releases, cost adds the data-centre GPU) and the Laya API-surface cell to PyPI 0.3.23; the JevBench text board itself is unchanged since v1.4.2.2.
  • 2026-10-02 - Moved the JevBench maintenance cell to the frozen v1.5 method publication of 2026-09-29 (equal-axis, equal-type headline amendment), with the v1.4.2.2 board still current and no v1.5 results published.
  • 2026-10-03 - Moved the Laya API-surface cell to PyPI 0.3.24 and pointed the SemIf reference at the repository’s canonical SemIf-OpenJev name (the /SemIf URL redirects); every other cell is unchanged.

See also
#

  • Harness Feature Matrix - the same treatment for terminal coding agents
  • Model Selection for Coding Tasks - the model side of the decision, since the mechanism choice follows the model choice
  • MCP - the tool ecosystem whose calls get schema guarantees
  • LangChain - the framework layer that wraps these features alongside its own output parsers

References
#