What Is Microsoft SkillOpt? A Beginner's Guide to Training Agent Skills
Learn what Microsoft SkillOpt is, how it trains a skill file with epochs and validation gates, what it costs to run, and when writing the skill by hand is still the right call.
SkillOpt is an open-source optimizer from Microsoft that treats an agent's skill file as the thing you train. The model weights never move. Instead, a second LLM reads scored transcripts of your agent attempting real tasks, proposes bounded add / delete / replace edits to a Markdown skill document, and an edit is kept only when it improves a held-out validation score. What you deploy at the end is a single file, typically 300 to 2,000 tokens, that costs nothing extra at inference time. Two practical facts up front: it is MIT-licensed and pip-installable, and the interesting workflow needs a git checkout rather than just the PyPI package.
System requirements
Python
3.10 or newer
Stated on the PyPI page and in the repository badges. The package installs with pip install skillopt.
Working directory
A git checkout, for the real workflow
The PyPI wheel gives you the library. Benchmark configs, data scripts and the Claude Code / Codex / Copilot / Devin integration shells live in the repository, not the wheel, so the documented quickstart starts with git clone.
Model access
Two roles, billed separately
You configure an optimizer model and a target model independently. Backends include Azure OpenAI (openai_chat), a provider-neutral openai_compatible backend, Qwen, MiniMax, and exec harnesses for Codex and Claude Code. Note that despite the name, claude_chat shells out to claude -p; the docs are explicit that it is not a direct Anthropic API client.
A scored task set
A benchmark or your own
Six benchmarks ship built in: DocVQA, ALFWorld, OfficeQA, SearchQA, LiveMathematicianBench and SpreadsheetBench. Optimizing against your own work means writing an adapter. The docs put a new benchmark at roughly 100 lines.
Token budget
Real, and not published
Training runs the target model across many rollouts and the optimizer model across every reflection step. The repository does not publish a cost-per-run figure, so treat your first run as a measurement exercise and start on the smallest split you can.
Install the package, or clone for the full workflow
The one-line install gives you the library and CLIs. The documented quickstart clones the repository, because the configs and data scripts are not in the wheel.
pip install skillopt gets you the skillopt, skillopt-train and skillopt-eval entry points.
The research path installs an editable checkout with the extra for whichever benchmark you plan to use. Extras published on PyPI include alfworld, claude, qwen, searchqa, webui, docs, dev and all.
pip install skilloptgit clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e ".[searchqa]"Configure credentials
Optimizer and target are separate roles with separate configuration, so you can optimize with one model and deploy against another.
Copy the example environment file and fill in one auth mode.
If your provider speaks the OpenAI Chat Completions protocol, the docs tell you to try the built-in openai_compatible backend before writing an integration.
cp .env.example .env
set -a; source .env; set +aTip
Because the two roles are billed separately, a cheap target and an expensive optimizer is a legitimate configuration, and usually the sensible one to start with.
Materialize the data split
Benchmarks ship as ID manifests rather than bundled datasets, so the runnable split is built locally.
For SearchQA, one script turns the checked-in manifest into the split the trainer reads.
python scripts/materialize_searchqa.pyTrain the skill
The loop runs rollout, reflect, aggregate, select, update and gate, then repeats over epochs.
Point the trainer at a benchmark config and an output directory. Everything the run produces, including the winning skill file, lands under that directory.
Hyperparameters are named after their deep-learning counterparts. learning_rate is the maximum number of edits allowed in a step, which is gradient clipping by another name. lr_scheduler accepts cosine, linear or constant.
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_quickstartTip
Start with the smallest split and one epoch. You are measuring what a run costs before you decide how much of it to buy.
Evaluate on a split the optimizer never saw
The validation gate is the entire point of the tool. Confirm the gain holds on held-out data.
Evaluate the produced best_skill.md against an unseen split rather than the one used during selection.
A gain that appears on the selection split and vanishes on the unseen split is the failure mode this tool exists to catch. Treat that outcome as the tool working, not as a wasted run.
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_quickstart/best_skill.md \
--split valid_unseenDeploy the file
The artifact is a Markdown file. Nothing else ships.
best_skill.md is typically 300 to 2,000 tokens and runs against the unchanged target model, adding zero inference-time model calls.
Microsoft reports that optimized skills transfer across model scales, between the Codex and Claude Code harnesses, and to nearby benchmarks without further optimization. That is a vendor claim about their own artifacts. Test it on yours before relying on it.
Optional: monitor SkillOpt training in the WebUI
A Gradio dashboard ships as an extra. Note the default bind address before you launch it.
The WebUI listens on 0.0.0.0 by default, which exposes it on every network interface. Bind it to localhost unless you have a specific reason not to.
pip install -e ".[webui]"
python -m skillopt_webui.app --host 127.0.0.1What problem does SkillOpt solve?
Most teams write a skill or instruction file once, read it back, decide it seems reasonable, and never measure it again. It gets edited when something breaks. Nobody can say whether the current version is better than the one from three months ago, because nothing was ever scored.
SkillOpt's framing is that this document is the trainable state of a frozen agent, and that it deserves the same discipline as weights. The repository is blunt about the gap it is closing. It describes today's agent skills as "usually hand-crafted, generated one-shot by a strong LLM, or evolved through loosely controlled self-revision", and argues that none of those behaves like a real optimizer for the skill itself.
The optimizer is a clever piece of engineering. The part worth stealing even if you never install the tool is the validation gate: an edit to your instructions is only an improvement if it beats the previous version on data the editor never saw.
How does SkillOpt train a skill file?
Six stages run per step, and the vocabulary is deliberately borrowed from deep learning.
- Rollout: the target model attempts tasks using the current skill file.
- Reflect: a separate optimizer model reads the scored trajectories and proposes edit patches.
- Aggregate: patches from across the batch are merged.
- Select: edits are ranked and clipped to a budget set by the learning_rate hyperparameter.
- Update: the surviving edits are applied to the skill document.
- Gate: the candidate is evaluated on a held-out selection split and accepted only if it strictly improves. Ordinary prompt tinkering has no equivalent, and this is the mechanism that makes the rest trustworthy.
- At epoch boundaries a slow update and a meta-skill memory run, which is where the multi-epoch stability comes from. A rejected-edit buffer stops the optimizer from re-proposing changes that already failed.
How does SkillOpt map onto neural network training?
| Deep learning | SkillOpt |
|---|---|
| Model weights | The skill document (Markdown) |
| Forward pass | Rollout: the target executes tasks |
| Loss and gradient | Reflect: the optimizer produces edit patches |
| Gradient clipping | Edit selection, capped by learning_rate |
| SGD step | Patch application to the skill |
| Validation set | Gated evaluation on the selection split |
| LR schedule | lr_scheduler: cosine, linear or constant |
| Epochs | Multi-epoch runs with slow update and meta-skill memory |
The mapping is not decoration. It is why the hyperparameters are named the way they are, and it is the fastest way to predict what a knob will do before you turn it.
What do SkillOpt's benchmark numbers actually show?
Microsoft reports results across six benchmarks, seven target models and three execution harnesses: direct chat, the Codex CLI and the Claude Code CLI. It states that SkillOpt is best or tied-best on all 52 evaluated (model, benchmark, harness) cells.
The headline lift is for GPT-5.5, and the baseline matters: these are gains over average no-skill accuracy, meaning an agent running with no skill file at all.
- GPT-5.5 in direct chat: +23.5 points over no-skill average accuracy.
- GPT-5.5 inside the Codex agentic loop: +24.8 points.
- GPT-5.5 inside Claude Code: +19.1 points.
- These are the authors' own figures from the paper (arXiv:2605.23904) and the repository README, not independently reproduced results.
- Read the baseline carefully. "Better than no skill file" is a much weaker claim than "better than the skill file your team already wrote", and the second is the comparison that decides whether this is worth your budget.
The trade-offs worth knowing
The project is unusually well documented for research code, and it is still research code with a production wrapper. The gaps are worth naming.
- Training costs real tokens and the repository does not publish a cost-per-run figure. Every rollout runs the target model and every reflection runs the optimizer model. Budget a measurement run before you plan a program of work.
- Optimizing against your own tasks means writing an adapter. The six built-in benchmarks are research benchmarks; your production workload is not one of them. The docs estimate about 100 lines for a new benchmark, and that estimate assumes you already have a reliable scorer, which is usually the harder half.
- You need a scoring function at all. If you cannot automatically score whether an agent run succeeded, there is nothing for the gate to gate on, and this tool has nothing to offer you yet.
- claude_chat is not an Anthropic API client. The docs say plainly that it launches claude -p. Plan your auth and rate limits accordingly.
- The integration shells for Claude Code, Codex, Copilot and Devin ship in the repository, not the PyPI wheel. If you pip install and go looking for them, they are not there.
- SkillOpt-Sleep, the nightly self-evolution companion, reviews real coding-agent session data. The docs tell you to read its data boundary and try the no-provider mock path first. That is good advice and it is there for a reason.
- The WebUI binds to 0.0.0.0 by default.
Who should use SkillOpt?
If you maintain skill or instruction files for agents that do repeatable, scoreable work such as support triage, document extraction, structured research or spreadsheet manipulation, this is worth a real evaluation. You have the two things it needs: a task set and a way to score it.
If your agent does open-ended work that a human judges case by case, you do not have a validation gate, and the honest move is to build the scoring first. That is useful on its own, with or without SkillOpt.
And if you have never measured your current skill file at all, the cheapest version of this idea costs nothing: hold out twenty real tasks, score your current instructions against them, change one thing, and score again. SkillOpt automates that loop well, but the discipline is what produces the gain.
The optimizer is clever; the validation gate is the part you should copy whether or not you install it. Strong fit if your agent's work is scoreable, premature if it isn't.
Frequently asked questions
What is SkillOpt in one sentence?+
It is an open-source Microsoft tool that treats an agent's Markdown skill file as a trainable parameter, editing it through a rollout-reflect-update loop and keeping only the changes that improve a held-out validation score. Model weights are never altered.
Is SkillOpt free?+
The software is free and MIT-licensed. Running it is not: training makes repeated calls to both a target model and a separate optimizer model. The repository does not publish a cost-per-run figure, so measure a small run before committing to a larger one.
Does SkillOpt fine-tune or change the model?+
No. That is the central design choice. Weights stay frozen and the only thing that changes is a Markdown document, which is why the result works against a hosted model you do not control and adds no inference-time calls.
Does it work with Claude Code?+
Yes. Claude Code is one of the three execution harnesses in the published results, and v0.2.0 added integration shells for Claude Code, Codex, Copilot and Devin. One caveat from the docs: the backend named claude_chat launches claude -p rather than calling the Anthropic API directly, and those integration files live in the repository rather than the PyPI wheel.
What do I actually deploy at the end?+
A single file called best_skill.md, typically between 300 and 2,000 tokens. It runs against the unchanged target model and requires zero additional inference calls.
Can I use it on my own tasks instead of a benchmark?+
Yes, by writing a benchmark adapter. That is a package with a data loader, a scored rollout helper and a YAML config, which the docs put at roughly 100 lines. The real prerequisite is an automatic way to score a run. Without a scorer there is nothing for the validation gate to check.
What is SkillOpt-Sleep?+
A companion shipped in v0.2.0 as the skillopt-sleep CLI that reviews past coding-agent sessions overnight and stages proposed skill updates behind the same held-out gate. It reads real session data, so the docs recommend reading its data-boundary notes and trying the mock path before pointing it at anything sensitive.
How much does it improve results?+
Microsoft reports best or tied-best performance on all 52 evaluated model-benchmark-harness cells, and for GPT-5.5 an average lift over no-skill accuracy of +23.5 points in direct chat, +24.8 in the Codex loop and +19.1 in Claude Code. Note the baseline is no skill file at all, not a well-written one, and these are the authors' own figures.
Is it production-ready?+
It is a versioned PyPI package with documentation, a changelog and an MIT license, which is further than most research code travels. It is still a research project: benchmarks are research benchmarks, some controls are marked experimental, and SkillOpt-Sleep is a preview.
What are the system requirements?+
Python 3.10 or newer, plus credentials for an optimizer model and a target model. There is no GPU requirement, because nothing is trained locally. The compute cost is API tokens rather than hardware.
