skill-creator — How It Works

The Flow

End to end

Interview

Capture what the skill does, when it triggers, expected output format

Write SKILL.md

Frontmatter (name, description) + instructions + bundled scripts/references

Write test prompts

2-3 realistic prompts saved to evals/evals.json

repeat until happy

Spawn parallel subagents

with_skill + baseline subagent for each prompt, launched simultaneously

Grade outputs

Grader agent checks assertions, verifies claims, saves grading.json

Aggregate benchmark

Pass rates, timing, tokens aggregated into benchmark.json with variance

Launch eval viewer

Browser-based UI: inspect outputs, see grades, leave feedback per case

Read feedback, improve skill

Generalize from feedback, keep prompts lean, bundle repeated scripts

after skill is solid

Optimize description

Generate 20 trigger eval queries, train/test split, auto-optimize via run_loop.py

Package

Zip into .skill file for distribution (excludes evals, cache)

The Engine

claude -p + structured JSON

Two primitives: claude -p subprocesses and structured JSON between every stage.

# Spawn a test run as subprocess
claude -p "Extract contacts from cards.pdf" \
  --output-format stream-json \
  --verbose

# Stream events arrive as JSON lines:
{"type": "stream_event", "event": {
  "type": "content_block_start",
  "content_block": {
    "type": "tool_use",
    "name": "Skill"
  }
}}

# On completion, capture:
{"total_tokens": 84852,
 "duration_ms": 23332}

Subprocess, not API

Each test spawns a real claude -p process — identical to how users invoke skills. Stream events detect triggering in real-time.

JSON all the way down

Every stage emits JSON: eval_metadata, grading, timing, benchmark. Scripts aggregate, viewer renders. No ambiguity.

Grading & Benchmarks

Quantitative rigor

Grade against assertions, aggregate into benchmarks with variance.

Grade

Each assertion checked against outputs with cited evidence. PASS or FAIL.

Verify

Extracts implicit claims from outputs and verifies them independently.

Aggregate

Pass rates, timing, tokens per config with mean ± stddev.

Analyze

Non-discriminating assertions, flaky evals, cost tradeoffs.

with_skill

100% ± 0%

40.5 ± 2.3s

4,650 ± 450

without_skill

50% ± 0%

27.2 ± 1.9s

2,350 ± 450

delta

+50%

+13.3s

+2,300

Specialized Agents

The crew

Three subagent roles handle distinct parts of evaluation.

⚖

agents/grader.md

Grader

Checks assertions with cited evidence. Verifies implicit claims. Flags weak assertions.

👁

agents/comparator.md

Comparator

Blind A/B comparison. Sees outputs as A/B with no origin info. Scores on rubric, declares winner.

🔍

agents/analyzer.md

Analyzer

Post-hoc analysis of why one version won. Reads transcripts, surfaces gaps and priorities.

Description Improver

Rewrite to trigger better

The description field controls triggering. This pipeline generates eval queries, curates them, then rewrites until it triggers correctly.

Generate 20 eval queries

Should-trigger + should-not-trigger mix. Realistic edge cases, not obvious keyword matches

Human reviews via HTML UI

Edit text, toggle should-trigger, add/remove queries. Exports to eval_set.json

run_loop.py: train/test split + optimize

60/40 split. Each query runs 3x. Rewrites description based on failures. Up to 5 iterations.

Select best by test score

Picked by held-out test performance, not train — prevents overfitting to the training queries

eval query review — edit, toggle, add/remove before optimization open in new tab

# What the improver sees each iteration:
current_description: "Extract and format data..."

FAILED TO TRIGGER (should have, didn't):
  "Convert meeting notes into summary"
  triggered 1/3 times

FALSE TRIGGERS (shouldn't have, did):
  "Add a column to my xlsx file"
  triggered 2/3 times

PREVIOUS ATTEMPTS: (don't repeat these)
  # full history with scores per query

# Rewrites description via claude -p
# Hard limit: 1024 chars

Generalize, don't overfit

No expanding keyword lists. Generalize failures to broader intent categories. Stay under ~200 words.

History-aware

Each iteration sees all prior attempts + scores. Avoids repeats, encourages structural variation.

optimization results — iterations with per-query pass/fail open in new tab

Improving a Skill

Already have a skill?
Make it better

Got a skill that's almost right? Same eval loop, but the old version becomes the baseline.

# Improve an existing skill
"use /skill-creator to improve ~/.claude/skills/code-review — it's too shallow on security issues"

Key difference

Baseline = old version

Creating: baseline is no skill. Improving: baseline is the old version. Isolates what you fixed (or broke).

Snapshot first

Preserve before editing

Old skill is snapshotted before edits. Baseline runs point at the frozen copy, keeping the comparison clean.

# 1. Snapshot the current version
cp -r ~/.claude/skills/code-review \
     workspace/skill-snapshot/

# 2. Edit the skill
#    (make your improvements to SKILL.md)

# 3. Run test cases with both versions
#    new = the edited skill
#    old = workspace/skill-snapshot/

# Workspace structure for improvement:
iteration-2/
  review-react-hooks/
    with_skill/    # new (improved)
    old_skill/     # baseline snapshot
    eval_metadata.json
  review-api-errors/
    with_skill/
    old_skill/
    eval_metadata.json
  benchmark.json
  feedback.json

Improvement Philosophy

Generalize

The skill runs a million times. Don't overfit to test cases — fix the pattern, not the specific failure.

Keep it lean

Read transcripts, not just outputs. If the model wastes time, cut the instructions causing it.

Explain the why

Skip the ALL CAPS ALWAYS/NEVER. Explain why it matters. Reasoning beats rigid rules.

Bundle repeated work

Every test run wrote the same helper? Write it once, put it in scripts/.

The Iteration Loop

Read feedback

from previous iteration

→

Improve skill

edit SKILL.md / scripts

→

Rerun evals

iteration-N+1/

→

Launch viewer

--previous-workspace

→

Human review

compare & leave feedback

Viewer shows previous output and previous feedback per test case. Iterate until feedback is empty or the user says stop.

Demo: Improvement Viewer

Real viewer with improvement data. Baseline labeled "old skill," previous output/feedback visible, benchmark compares with_skill vs old_skill.

eval-viewer — code-review · iteration 2 (improving) open in new tab

Build skillsthat actually work

End to end

claude -p + structured JSON

Subprocess, not API

JSON all the way down

Side-by-side runs

Skill-Augmented Run

Baseline Run

Skill + workspace