Skill Optimizer — Analyzer + Auto Research for Gemini CLI Skills

MODE 1: ANALYZE

Run when the user says analyze, or as the first step before optimization. This mode reads the skill, scores it, and produces a comprehensive report. No changes are made to any files.

A1: Read the Skill

1. Read ~/.gemini/skills/<skill-name>/SKILL.md
2. Read all files in ~/.gemini/skills/<skill-name>/references/ if the directory exists
3. Count total lines, sections, and phases

A2: Structural Analysis

Score the skill across 7 dimensions. For each dimension, assign a rating:

• Strong — the skill handles this well with concrete detail
• Adequate — present but could be improved
• Weak — missing or too vague to be useful
• Absent — not present at all

Dimension 1: YAML Frontmatter Completeness

Check for these fields:

• name — present and matches directory name?
• description — present, detailed, includes trigger phrases?
• allowed-tools — present, lists appropriate tools for what the skill does?
• argument-hint — present if the skill accepts arguments?

Findings to report:

• Missing fields
• Description trigger coverage — does it capture enough phrases to reliably activate?
• Tool declarations — are any tools used in the body but not declared? Any declared but unused?

Dimension 2: Phase Architecture

Does the skill decompose work into discrete phases with clear boundaries?

Quality signals (strong):

• Named phases with numbered steps (Phase 1, Phase 2, etc.)
• Each phase has a clear input, action, and output
• Phases have explicit actor roles (who does what — orchestrator, researcher, reviewer, etc.)
• Dependencies between phases are stated

Warning signals (weak/absent):

• Single block of instructions with no decomposition
• Vague flow like "do X then Y then Z" without structure
• No clear handoff between steps
• Reads like a one-liner or rough notes rather than a production workflow

Dimension 3: Output Specification

Does the skill define what "done" looks like?

Quality signals (strong):

• Full output template with markdown structure shown
• Example output included
• Format constraints stated (headers, bullets, length, sections)
• Output location/filename specified

Warning signals (weak/absent):

• No template or example
• Vague description like "produce findings" or "generate a report"
• No format constraints — relies on the model's defaults

Dimension 4: Constraint Density & Specificity

Are instructions concrete enough to produce consistent output?

Quality signals (strong):

• Specific constraints: "maximum 5 insights", "under 25 words per bullet"
• Concrete examples of good vs. bad output
• Named tools, formats, and conventions referenced
• Decision criteria for branching logic ("if X, do Y; otherwise, do Z")

Warning signals (weak/absent):

• Subjective language: "make it good", "be thorough", "high quality"
• Open-ended instructions: "explore the codebase", "find relevant information"
• No examples of what good output looks like
• No decision criteria — model has to guess at every branch

Calibration: Too few constraints = unreliable output. Too many = brittle, gameable. The sweet spot is 3-6 concrete constraints per phase, not per skill.

Dimension 5: Error Handling & Edge Cases

Does the skill anticipate what can go wrong?

Quality signals (strong):

• Error table mapping failure types to recovery actions
• Fallback strategies for common failures
• Graceful degradation when sources are incomplete
• Bounded retries with escalation

Warning signals (weak/absent):

• No mention of what happens when things fail
• No fallback for missing inputs or empty results
• Assumes everything works perfectly

Dimension 6: Examples & Demonstration

Does the skill show rather than just tell?

Quality signals (strong):

• 3+ concrete input/output examples
• Good vs. bad examples side by side
• Edge case examples
• Code blocks showing exact format

Warning signals (weak/absent):

• Zero examples
• Only abstract descriptions of what the output should be
• "Use your judgment" without showing what good judgment looks like

Dimension 7: Scaling & Complexity Management

Does the skill adapt to different input sizes or complexities?

Quality signals (strong):

• Scaling table (simple/medium/complex with different approaches)
• Adaptive depth based on input size
• Explicit scope limits ("maximum N items", "stop after N rounds")

Warning signals (weak/absent):

• One-size-fits-all approach regardless of input complexity
• No scope limits — could run forever on large inputs
• No mention of how to handle trivial vs. complex cases differently

A3: Quality Score Card

Present findings as a visual scorecard:

## Skill Analysis: <skill-name>

| Dimension                    | Rating   | Key Finding                           |
|------------------------------|----------|---------------------------------------|
| YAML Frontmatter             | Strong   | All fields present, rich triggers      |
| Phase Architecture           | Adequate | 3 phases but no actor roles defined    |
| Output Specification         | Strong   | Full template with markdown example    |
| Constraint Density           | Weak     | Subjective language, no decision trees |
| Error Handling               | Absent   | No failure modes addressed             |
| Examples & Demonstration     | Weak     | 1 example, no good-vs-bad comparison   |
| Scaling & Complexity Mgmt    | Absent   | Same approach for all input sizes      |

**Overall Tier:** <tier>

Tier Classification

• Tier 1 — Production Ready (5+ Strong, 0 Absent): Reliable, well-structured, handles edge cases. Examples: deep-research, youtube
• Tier 2 — Usable (3+ Strong/Adequate, 1-2 Absent): Works most of the time but has blind spots. Examples: connect, emerge
• Tier 3 — Needs Work (2+ Weak/Absent in critical dimensions): Unreliable output, vague instructions. Examples: skills that read like rough notes or one-liners

A4: Eval Recommendations

Based on the structural analysis, recommend 4-6 binary eval criteria that would meaningfully test this skill's output quality.

For each recommended eval:

### Recommended Eval <N>

**Question:** <binary yes/no question>
**Tests dimension:** <which of the 7 dimensions this eval validates>
**Why this matters:** <1-2 sentences on what failure here would indicate>
**Risk of gaming:** <low/medium/high — can the model trivially satisfy this without quality?>
**Priority:** <critical / important / nice-to-have>

PASS: <what "yes" looks like concretely>
FAIL: <what "no" looks like concretely>

Eval Selection Principles

1. Cover different dimensions — don't cluster all evals on one aspect (e.g., all format checks)
2. At least one structural eval — tests the output's architecture (sections, ordering, completeness)
3. At least one content eval — tests whether the substance is correct/useful
4. At least one constraint eval — tests adherence to a specific rule in the skill
5. Avoid redundancy — each eval should catch a different failure mode
6. Prefer evals where failure is currently likely — target the skill's weakest dimensions

Eval Anti-Patterns to Flag

If the user provides custom evals, check for and warn about:

• Subjective criteria: "Is it well-written?" — can't be judged consistently
• Overly narrow criteria: "Exactly 47 words" — encourages gaming
• Compound criteria: "Is it clear AND concise AND well-structured?" — test one thing at a time
• Unobservable criteria: "Does the model understand the topic?" — can't verify from output alone
• Correlated criteria: Two evals that always pass/fail together add noise, not signal

A5: Optimization Recommendations

Produce a prioritized list of specific changes that would improve the skill, organized by effort and impact.

Priority Matrix

### Optimization Recommendations

#### Quick Wins (high impact, low effort)
1. <specific change> — addresses <dimension>
   WHY: <what problem this solves>
   HOW: <1-2 sentence description of the change>

#### Structural Improvements (high impact, medium effort)
1. <specific change> — addresses <dimension>
   WHY: <what problem this solves>
   HOW: <description of the change>

#### Deep Investments (high impact, high effort)
1. <specific change> — addresses <dimension>
   WHY: <what problem this solves>
   HOW: <description, possibly involving references/ files>

Common Recommendation Patterns

Apply these based on what the analysis found:

If Phase Architecture is Weak/Absent:

• Decompose the single-block instruction into 3-5 named phases
• Define input → action → output for each phase
• Add explicit handoff points between phases

If Output Specification is Weak/Absent:

• Add a full markdown output template
• Show a concrete example of completed output
• Specify filename/location for generated artifacts

If Constraint Density is Weak:

• Replace subjective language with measurable constraints
• Add decision trees for branching logic
• Define "done" criteria for open-ended instructions
• Add "good vs. bad" examples for ambiguous instructions

If Error Handling is Absent:

• Add an error table: failure type → detection → recovery action
• Define fallback for missing/empty inputs
• Add scope limits to prevent runaway execution

If Examples are Weak/Absent:

• Add 3+ input/output examples covering typical, edge, and complex cases
• Add "good vs. bad" comparison for the most critical output aspect

If Scaling is Absent:

• Add a complexity table: simple/medium/complex → approach for each
• Define scope limits (max items, max depth, max time)

If Description Triggers are Insufficient:

• Audit when the skill fails to activate — add missing trigger phrases
• Cross-reference with similar skills to avoid trigger conflicts

A6: Analysis Report

Save the complete analysis to ~/.gemini/skills/<skill-name>/analysis-report.md:

# Skill Analysis Report: <skill-name>

**Analyzed:** YYYY-MM-DD
**Lines:** <N>
**Tier:** <1/2/3>
**References:** <N files / none>

## Score Card

<the table from A3>

## Detailed Findings

<expanded findings per dimension — 2-4 sentences each with specific line references>

## Recommended Evals

<the eval recommendations from A4>

## Optimization Recommendations

<the prioritized list from A5>

## Next Steps

- [ ] Apply quick wins manually or run `/skill-optimizer optimize <skill-name>`
- [ ] Review eval recommendations and approve/modify before optimization
- [ ] Consider adding references/ directory for complex logic (if applicable)

After saving, print a summary to the user and ask:

Analysis complete for <skill-name> (Tier <N>).
<1-sentence summary of biggest finding>

Full report saved to ~/.gemini/skills/<skill-name>/analysis-report.md

Would you like to proceed to optimization with the recommended evals?

MODE 2: OPTIMIZE

The auto research loop. Run when the user says optimize, or after analysis when the user confirms they want to proceed.

Core Concept

The auto research loop (inspired by Andrej Karpathy's auto research repo):

1. Run the target skill with sample inputs
2. Evaluate outputs against binary (yes/no) criteria
3. Score the run (passes / total checks)
4. Mutate the skill prompt to address failures
5. Keep the winning version, discard losers
6. Repeat until score plateaus or hits target

O1: Read & Backup

1. Read ~/.gemini/skills/<skill-name>/SKILL.md
2. Copy to ~/.gemini/skills/<skill-name>/SKILL.md.backup (preserve the original)
3. If an analysis report exists, read it — use its findings to inform the optimization strategy

O2: Establish Eval Criteria

If coming from analyze mode: use the recommended evals from A4 (user already approved them).

If user provided --evals: parse them into binary yes/no questions. Check for anti-patterns (subjective, compound, unobservable, correlated) and warn if found.

If neither: auto-generate 4-6 binary eval criteria by analyzing the skill instructions.

Eval Format

For each criterion:

EVAL_<N>: <yes/no question>
PASS: <what "yes" looks like>
FAIL: <what "no" looks like>

Eval Quality Rules

• Binary — answerable with yes or no, nothing subjective
• Observable — can be verified by reading the output
• Independent — each criterion tests one thing
• Non-gameable — the model can't trivially satisfy them without genuine quality
• 4-6 max — more than 6 causes the skill to game the eval rather than genuinely improve

Examples of Good vs Bad Evals

Good:

• "Does the output contain section headers formatted with bold markdown (**Header**)?"
• "Are all bullet points concise (under 25 words each)?"
• "Does the output avoid timestamps in the summary section?"

Bad:

• "Is the output high quality?" (subjective)
• "Does it look professional?" (subjective)
• "Is it under 500 words?" (too narrow, encourages truncation)

Present criteria and wait for user confirmation before proceeding.

O3: Prepare Sample Inputs

Generate 3-5 diverse sample inputs the skill would realistically receive:

• A typical/common use case
• An edge case (minimal input, unusual request)
• A complex case (long input, multiple requirements)

For skills that require external data (URLs, files, etc.), ask the user to provide sample inputs or point to existing test data.

Print the sample inputs for user review.

O4: Run the Optimization Loop

For each round (1 to --rounds):

O4a. Run the skill

For each sample input, execute the skill:

• If the skill produces files (markdown, images, code), run it and capture output
• If the skill requires interactive conversation, simulate expected output

For skills with Bash scripts, actually execute them. For prompt-only skills, use an Agent to role-play as the skill and generate output.

O4b. Evaluate outputs

For each output, check every eval criterion. Score as PASS (1) or FAIL (0).

Create a scorecard:

Round <N> Results:
┌─────────────┬────────┬────────┬────────┬────────┬───────┐
│ Sample Input │ Eval 1 │ Eval 2 │ Eval 3 │ Eval 4 │ Score │
├─────────────┼────────┼────────┼────────┼────────┼───────┤
│ Input 1      │ PASS   │ PASS   │ FAIL   │ PASS   │ 3/4   │
│ Input 2      │ PASS   │ FAIL   │ PASS   │ PASS   │ 3/4   │
│ Input 3      │ PASS   │ PASS   │ PASS   │ PASS   │ 4/4   │
├─────────────┼────────┼────────┼────────┼────────┼───────┤
│ TOTAL        │        │        │        │        │ 10/12 │
└─────────────┴────────┴────────┴────────┴────────┴───────┘
Pass rate: 83.3%

O4c. Analyze failures

For each FAIL, identify:

• Which part of the skill prompt caused the failure (cite line numbers)
• Whether the instruction was missing, ambiguous, or contradictory
• Which analysis dimension (from A2) the failure maps to
• What specific change would fix it

O4d. Mutate the prompt

Apply targeted edits to the skill's SKILL.md to address failures:

• Add missing constraints
• Clarify ambiguous instructions
• Add examples for common failure patterns
• Reorder instructions to prioritize frequently-failed criteria

Mutation rules:

• Make the minimum change needed to fix the failure — don't rewrite the whole skill
• Never remove instructions that are currently passing
• Add constraints as close as possible to the relevant section
• Preserve the skill's original structure and voice
• Keep a changelog comment at the bottom of each mutation

O4e. Check for improvement

Compare this round's score to the previous round:

• If improved: keep the mutation, continue
• If same: keep the mutation (might help with variance), continue
• If worse: revert the mutation, try a different approach
• If target reached: stop early

O4f. Print round summary

Round <N> complete:
  Score: <X>/<total> (<percentage>%)
  Change from last round: +<N> / -<N> / same
  Mutation applied: <brief description of what changed>
  Status: improved / plateau / regressed (reverted)

O5: Final Report

After all rounds complete (or target reached), produce:

## Optimization Report: <skill-name>

**Rounds completed:** <N>
**Starting score:** <X>/<total> (<percentage>%)
**Final score:** <X>/<total> (<percentage>%)
**Improvement:** +<percentage points>

### Eval Criteria Used
1. <criterion> — pass rate: <X>%
2. <criterion> — pass rate: <X>%
...

### Changes Applied (cumulative)
1. Round <N>: <description of mutation>
2. Round <N>: <description of mutation>
...

### Per-Eval Breakdown
| Eval | Start Pass Rate | Final Pass Rate | Trend     |
|------|----------------|-----------------|-----------|
| 1    | 60%            | 100%            | Fixed     |
| 2    | 80%            | 80%             | Unchanged |
| 3    | 40%            | 80%             | Improved  |
| 4    | 100%           | 100%            | Stable    |

### Remaining Failures
- <description of any persistent failures and why they're hard to fix>

### Recommendations for Further Improvement
- <suggestions beyond prompt changes — references/ files, tool additions, structural rewrites>
- <link back to analysis report if one exists>

Save to ~/.gemini/skills/<skill-name>/optimization-report.md

O6: Save to Vault

Save a condensed research note to ~/Vault/research/YYMMDD-optimize-<skill-name>.md:

---