The Agent Skills That Pay Off Are Scripts With a Manual Attached

In August I packaged the way articles get written on this site into a skill. The SKILL.md is 118 lines, it points at three reference files on voice, SEO and a final checklist, and it ships a 460-line PHP audit script. A month later the lesson is lopsided: the part of that skill doing the most work is the part the model never reads. It runs the script, gets a list of errors about description length, duplicate closing headers and broken route names, and fixes them. The prose instructions matter; the script is what makes them stick.

The other lesson is less flattering. My CLAUDE.md and my memory index both contain a line telling the agent to use that skill — because without the reminder it doesn't always reach for it. Both lessons line up with what the published measurements now say about skills, and the measurements say a few things the "top 10 skills to install" posts leave out.

A skill costs almost nothing until it runs

A skill is a directory with a SKILL.md file, and its cost is deferred until the moment it is used. The Agent Skills specification, published by Anthropic in December 2025 and since adopted by Codex CLI, Gemini CLI, Cursor and a long list of other agents, defines three loading levels:

That last row is the one to design around. Anthropic's October 2025 engineering write-up gives the example of a PDF skill where Claude runs a bundled script "without loading either the script or the PDF into context". A 460-line audit costs my sessions the few dozen tokens of its output, not its source.

The spec keeps the rest short: name up to 64 lowercase characters matching the directory, description up to 1,024 characters saying what the skill does and when to use it, and SKILL.md under 500 lines with detail pushed into files one level deep.

Curated skills help most where the model lacks the expertise

Hand-written skills measurably raise agent success, but by far the least in the domain most developers write them for. SkillsBench is the largest paired evaluation so far. Its June 2026 revision runs 87 tasks across 8 domains under 18 model-harness configurations, and curated skills lift the average pass rate from 33.9% to 50.5% — 16.6 percentage points, with individual configurations gaining between 4.1 and 25.7.

The first version of the paper, from February, broke that down by domain, and the spread is the useful part. Software engineering gained 4.5 points. Healthcare gained 51.9. And 16 of 84 tasks got worse with a skill than without one.

That fits everything else we know about coding agents. A model trained on most of GitHub does not need a skill explaining how to write a Symfony controller, and a skill that tries is paying context for nothing — the same pattern the ETH Zurich AGENTS.md study found for context files, which raised cost without raising success rates. What it lacks is the knowledge that lives only in your organisation: this site's banned closing headers, the Search Console findings that decide what an article should optimise for, the route names that contain digits. Two more findings from the paper point the same way. Focused skills with at most three modules beat exhaustive documentation bundles, and small models with the right skill matched larger models without one.

Half of auto-triggered skills never fire

Automatic invocation is the weakest link in the whole mechanism, and the numbers are consistent across independent tests. In Vercel's evals published on 27 January 2026, the agent never invoked the available skill in 56% of cases, and adding explicit "use the skill" instructions made results swing on small wording changes.

Scott Spence measured Claude Code directly in February 2026, running the real binary with Sonnet 4.5 in sandboxes against 22 prompts that should each trigger a specific skill:

The forced-evaluation hook has limits of its own: on a harder set of 24 ambiguous prompts its overall accuracy dropped to 75%. It does prove the failure is in deciding to look, not in following the skill once loaded.

My fixes are boring and they work. The skills I care about get invoked by name — "use the write-article skill" costs four words. The one-line pointer lives in CLAUDE.md, which is exactly the kind of fact an instruction file exists for: small, and impossible for the agent to grep, as I argued in what belongs in CLAUDE.md. Anything with side effects — a deploy, a commit, a message to Slack — gets disable-model-invocation: true, so the model cannot decide on its own that the code looks ready. And when I do want automatic triggering, the description names the situations, not the topic: the spec's own poor example is literally Helps with PDFs.

Agent-written skills don't help

Asking the agent to write its own skills is the most popular shortcut, and the one measurement of it found no benefit. SkillsBench's first version tested a third condition next to "no skills" and "curated skills": skills the model generated itself before attempting the task. They provided no benefit on average, and the authors' summary is blunt — "models cannot reliably author the procedural knowledge they benefit from consuming."

It makes sense once you ask what the model would put in the file. It can only write down what it already knows, and what it already knows is what it would have done anyway. The value of a skill is the delta between the model's defaults and your process.

The article skill here came from the other direction. The banned-phrase list in its voice reference is a record of phrasing I rejected in real drafts. The rule about never reusing a closing header exists because I noticed every piece ending with the same one. The agent typed most of the file; the content came from corrections. If you use an agent to draft a skill, feed it the corrections — past review comments, the diff between its draft and what you shipped — not the topic.

Put the checks in a script, not in the prose

An instruction can be skimmed past at hour three of a session; strlen($description) > 160 cannot. Every rule in a skill that can be checked mechanically belongs in a script the skill runs, with the prose reduced to "run it and fix every error". That is how my audit is wired, and the same shape works for anything with a definition of done:

---
name: release-notes
description: Draft release notes from commits since the last tag. Use when the user asks for release notes, a changelog entry or what shipped.
disable-model-invocation: true
allowed-tools: Bash(${CLAUDE_SKILL_DIR}/scripts/*)
---

## Commits since the last tag

!`${CLAUDE_SKILL_DIR}/scripts/commits-since-tag.sh`

## Steps

1. Group the commits above into Added, Changed and Fixed.
2. Write the result to `notes.md`.
3. Run `${CLAUDE_SKILL_DIR}/scripts/check-notes.sh notes.md` and fix every line it reports.

#!/usr/bin/env bash
# scripts/commits-since-tag.sh - fall back to the whole history when there is no tag yet
if tag=$(git describe --tags --abbrev=0 2>/dev/null); then
  git log "$tag..HEAD" --oneline --no-merges
else
  git log --oneline --no-merges
fi

The !`command` line is Claude Code's dynamic context injection: the command runs before the skill reaches the model, so the instructions arrive with the real commit list already inlined instead of a request to go and fetch it. It is unforgiving in two ways, and both shaped the example. An injected command never prompts for permission, so anything not pre-approved in allowed-tools aborts the whole invocation — and a non-zero exit aborts it too: a bare git describe in a repository without tags would kill the skill before the model saw a word. So the fallback lives in a script, ${CLAUDE_SKILL_DIR} resolves to the skill's own directory wherever the session started, and the check at the end is where "every entry references an issue" or "no internal ticket IDs" gets enforced.

If you are still deciding what lives where, the Custom Instructions templates are a lean starting point for the always-on part, and everything procedural can then move into skills like this one.

Third-party skills are code with your shell access

A skill you install runs with the same permissions as the agent that loads it, and the public marketplaces already contain malicious ones. The largest measurement so far, published on arXiv in February 2026, collected 98,380 skills from skills.rest and skillsmp.com and confirmed 157 as malicious — 0.16%, carrying 632 vulnerabilities between them. Remote script execution was the most common technique at 25.2%, followed by behaviour manipulation at 18.8% and credential harvesting at 17.7%. A single actor accounted for 54.1% of the confirmed cases, mostly through templated brand impersonation.

Snyk's ToxicSkills audit the same month scanned 3,984 skills from ClawHub and skills.sh: 36.82% had at least one security flaw, 13.4% had a critical one, and human review confirmed 76 malicious payloads — password-protected ZIPs with an unzip-and-execute line, and base64-encoded curl commands passed straight to eval.

Two techniques make a quick read of SKILL.md insufficient. Embrace the Red demonstrated instructions hidden in Unicode Tag characters (U+E0000 to U+E007F), which render as nothing in most editors and on GitHub while the model reads them fine. Claude Code has refused content containing tag characters since its 10 February 2026 release, but a skills directory is often shared with other agents, so scan it anyway. And the Cloud Security Alliance's May 2026 research note describes persistence payloads that tell the agent to write modified instructions into CLAUDE.md or AGENTS.md, so the skill's influence survives uninstalling the skill. This is the same authority-boundary problem I described for MCP servers wired in without an audit, with a lower barrier to entry: no server to run, just a Markdown file.

How companies vet skills before anyone installs one

In the companies I have worked with, a third-party skill goes through a security review before it is allowed anywhere near a developer machine — the same gate a new dependency passes. A review that holds up covers four things:

1. Read every file, not just SKILL.md. Scripts, references and assets, plus every URL they fetch.
2. Scan for what a human can't see or shouldn't miss. The first command below finds invisible characters; the second finds the usual payload shapes. Every hit needs a human look — a legitimate skill can mention CLAUDE.md — but zero hits on a skill that claims to format dates is a good start.
3. Pin it. Install from a commit SHA you reviewed, not from a branch that can change under you.
4. Allowlist the source centrally, so "reviewed" is enforced rather than remembered.

# Invisible Unicode: tag characters, zero-width and bidi control characters
grep -rnP '[\x{E0000}-\x{E007F}\x{200B}-\x{200F}\x{202A}-\x{202E}\x{2066}-\x{2069}]' .claude/skills/

# Downloads piped into a shell, decoded payloads, writes to agent context files
grep -rnE 'curl[^|]*\|\s*(ba)?sh|base64 (-d|--decode)|(CLAUDE|AGENTS|MEMORY)\.md' .claude/skills/

The platform side has caught up with this in 2026. Claude Enterprise plans can scan third-party skills and plugins on upload or edit, returning pass, warn or fail within a minute or two — but by Anthropic's own description it only looks for skills "built to misuse the access they're given", and it skips skills that were already installed or created by Claude. In Claude Code, managed settings give security teams strictKnownMarketplaces to allowlist marketplace sources, strictPluginOnlyCustomization to block skills, agents, hooks and MCP servers from user and project directories, and allowManagedHooksOnly to run only the hooks the organisation deploys. Scanning is a filter in front of review, not a replacement for it.

Where a Skill Is the Wrong Tool

A skill is the right home for a procedure you trigger by name or for domain knowledge the model lacks, and the wrong home for almost everything else:

Before you write one, do the task by hand enough times to have corrected the agent at least twice; those corrections are the skill. Keep SKILL.md short, move every mechanical rule into a script, and invoke it by name. Before you install someone else's, treat it as a dependency with shell access: read all of it, scan it, pin it, and let a central allowlist decide what "reviewed" means.