↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Skills/

SkillOpt

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 7 minutes read (~1347 words)
agent-curated 610 fully-ai-generated 715 llm=glm-5.3-flash 620 microsoft 9 research 8 research-note 580 self-improving-agents 2 skills 29
Table of Contents

SkillOpt is a Microsoft Research text-space optimizer that trains a reusable natural-language skill document for a frozen LLM agent the way deep learning trains weights: trajectory-driven bounded edits, a textual learning rate, and a held-out validation gate, exporting a deployable best_skill.md.

SkillOpt turns the skill file into a trainable parameter, and the part that matters is not the optimization loop but the gate: an edit is kept only when it strictly improves a held-out score, which is exactly the discipline hand-written skills never get.

What it is
#

A Python package (skillopt on PyPI) where the target model runs scored rollout batches with the current skill, a separate optimizer model proposes bounded add, delete, and replace edits, an edit budget clips each step, and candidates are accepted only on strict validation improvement, with rejected edits feeding a negative-feedback buffer. The deployable artifact is one compact markdown file, median around 920 tokens, consumed by the unchanged target model with zero extra inference-time calls. The paper reports best or tied-best results across six benchmarks, seven models, and three harnesses, including runs through Codex CLI and Claude Code, with skills transferring across model scales and between harnesses. CLI tooling, a Gradio dashboard, multiple chat backends, and v0.2.0 integration shells for Claude Code, Codex, Copilot, Devin, and OpenClaw; a nightly SkillOpt-Sleep mode harvests and consolidates behind the same gate. MIT, from Microsoft Research Asia with university collaborators, Python 3.10+.

Status
#

Research code with unusually strong product trappings: 18.3k stars and 1.7k forks as of 2026-10-13, created 2026-05-08, with 86 open issues and pull requests as of 2026-11-12.

Star History Chart

Latest release v0.2.0 (2026-07-02) on PyPI, still the newest as of 2026-11-12, though main has run 409 commits past it (the 2026-10-29 pushes landed two security fixes: Sleep no longer imports or runs code from the project directory, and the web UI refuses to expose training controls without auth; the 2026-11-04 to 11-07 pushes added provenance tracking on proposed edits, protected-region overlap refusal, Sleep OAuth-replay and legacy-adoption guards, and SearchQA empty-answer handling; the 2026-11-10 batch hardened Sleep harvesting and scoring: cutoffs compared as instants, unreadable transcript lines skipped, api-request-blob records no longer harvested as user prompts, the judge’s 0-to-1 score contract enforced before gating, learned items kept intact across apply_edits, self-reported tool calls not credited on a measured tool route, and recall of a held-out task’s twin blocked as a leak, alongside OfficeQA and livemath scoring fixes, DocVQA evaluator tests, and an ALFWorld output confinement); the repo’s News section (added 2026-11-04) announces NeurIPS 2026 Spotlight acceptance for the paper and a first-place Executive Challenge win at the Microsoft Global Hackathon 2026, while the arXiv paper (2605.23904) itself remains the preprint and all headline results are the authors’ own.

Strengths
#

  • Validation-gated by construction, and the ablations show why: an ungated rewrite pushed one benchmark score down.
  • The artifact is small, auditable markdown, so trained skills remain readable and reviewable as practitioner rules.
  • Documented transfer across model scales and between Codex and Claude Code, which is what makes the artifact worth training at all.
  • A working ecosystem: PyPI releases, multi-backend support, and third-party adoptions within months.

Cautions
#

  • The guarantee is only as good as the automatic scorer: open-ended or subjective tasks weaken the gate, which the authors acknowledge.
  • Optimization is not free; training reached hundreds of millions of tokens on academic benchmarks, with community-quoted costs of a few dollars per task plus the engineering cost of a verifier and a representative held-out split.
  • Single-document scope: it trains one skill file, not a skill library.
  • All results are self-reported on the authors’ benchmarks; no independent replication was found as of 2026-09-18, and the NeurIPS Spotlight and hackathon recognitions are the repo’s own announcements.

Pricing
#

Free and open source under MIT. The costs are training-time tokens on your own API budget and the verifier engineering.

Compared to
#

  • Anthropic Agent Skills: hand-authored SKILL.md folders, free to write but unvalidated; a natural workflow is hand-authoring a seed and letting SkillOpt train it against scored tasks.
  • Agent Skills open standard: the portability layer; SkillOpt’s output is standard-compatible markdown, so trained artifacts slot into ordinary skill folders.
  • Ouroboros: wraps whole build loops with verification gates; SkillOpt distills experience into one portable artifact, and the two address different layers of self-improvement.

Bottom line
#

Recommended for teams that already have scored tasks and a verifier, who want their skill files to earn their place empirically. Not for subjective domains without reliable scoring, or anyone expecting trained-skill magic without building the evaluation harness first.

Changes
#

  • 2026-08-30 - Created in the owner-directed star sweep as a Skills note.
  • 2026-09-25 - Refreshed the volatile numbers (17,456 stars, 1,630 forks, 53 open issues and PRs); release v0.2.0 and the commit count unchanged.
  • 2026-09-29 - Refreshed the volatile numbers (17,824 stars, 1,672 forks, 55 open issues and PRs); release v0.2.0 and the commit count unchanged.
  • 2026-10-07 - Added the microsoft/SkillOpt star history chart to the Status section.
  • 2026-10-20 - Rotation re-verification: every cited URL re-fetched live except the VentureBeat piece, which answered HTTP 429 rate limits on two tries this run (last live fetch 2026-10-18); all other claims held and the volatile figures refreshed (55 open issues and PRs, 369 commits past v0.2.0, pushed 2026-10-18 with two further Sleep fixes).
  • 2026-10-24 - Volatile refresh: the 2026-10-23 pushes landed three further fixes (Sleep harvest checkpoint keyed by project and transcript source, generated Codex replay sessions excluded from harvest, a dsh 0.2.x executor-bridge fix), moving main to 372 commits past v0.2.0 with 58 open issues and PRs; v0.2.0 still the newest release.
  • 2026-10-26 - Volatile refresh: the 2026-10-25 pushes landed six commits (an offline reproducibility checker for saved evaluation artifacts, keyless Parallel search for OfficeQA rollouts, Sleep fixes harvesting pasted briefs and capping re-run hook templates, plus test and import organization), moving main to 378 commits past v0.2.0 with open issues and PRs down to 52; v0.2.0 still the newest release.
  • 2026-10-31 - Volatile refresh: the 2026-10-29 pushes landed two security fixes (Sleep no longer imports or runs code from the project directory, the web UI refuses training controls without auth), moving main to 380 commits past v0.2.0 with open issues and PRs up to 62; v0.2.0 still the newest release.
  • 2026-11-07 - The repo’s News section (added 2026-11-04) announces NeurIPS 2026 Spotlight acceptance and a first-place Executive Challenge win at the Microsoft Global Hackathon 2026, the first venue signal the note has recorded for the paper, whose arXiv page remains the preprint; volatile refresh alongside: main pushed 2026-11-07 with provenance tracking on proposed edits, protected-region overlap refusal, Sleep OAuth-replay and legacy-adoption guards, and SearchQA empty-answer handling, moving main to 391 commits past v0.2.0 with open issues and PRs up to 73; v0.2.0 still the newest release.
  • 2026-11-12 - Volatile refresh: the 2026-11-10 batch hardened Sleep harvesting and scoring (cutoffs compared as instants, unreadable transcript lines skipped, api-request-blob records no longer harvested as prompts, the judge’s 0-to-1 score contract enforced before gating, learned items kept across apply_edits, self-reported tool calls not credited on a measured route, held-out-twin recall blocked as a leak) alongside OfficeQA and livemath scoring fixes, DocVQA evaluator tests, and an ALFWorld output confinement, moving main to 409 commits past v0.2.0 with open issues and PRs up to 86; v0.2.0 still the newest release.

See also
#

References
#