Skill Tester

This skill is essential for maintaining ecosystem consistency, enabling automated CI/CD integration, and supporting both manual and automated quality assurance workflows. It serves as the foundation for pre-commit hooks, pull request validation, and continuous integration processes that maintain the high-quality standards of the claude-skills repository.

Core Features

Comprehensive Skill Validation

• Structure Compliance: Validates directory structure, required files (SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/)
• Documentation Standards: Checks SKILL.md frontmatter, section completeness, minimum line counts per tier
• File Format Validation: Ensures proper Markdown formatting, YAML frontmatter syntax, and file naming conventions

Advanced Script Testing

• Syntax Validation: Compiles Python scripts to detect syntax errors before execution
• Import Analysis: Enforces standard library only policy, identifies external dependencies
• Runtime Testing: Executes scripts with sample data, validates argparse implementation, tests --help functionality
• Output Format Compliance: Verifies dual output support (JSON + human-readable), proper error handling

Multi-Dimensional Quality Scoring

• Documentation Quality (25%): SKILL.md depth and completeness, README clarity, reference documentation quality
• Code Quality (25%): Script complexity, error handling robustness, output format consistency, maintainability
• Completeness (25%): Required directory presence, sample data adequacy, expected output verification
• Usability (25%): Example clarity, argparse help text quality, installation simplicity, user experience

Tier Classification System

Automatically classifies skills based on complexity and functionality:

BASIC Tier Requirements

• Minimum 100 lines in SKILL.md
• At least 1 Python script (100-300 LOC)
• Basic argparse implementation
• Simple input/output handling
• Essential documentation coverage

STANDARD Tier Requirements

• Minimum 200 lines in SKILL.md
• 1-2 Python scripts (300-500 LOC each)
• Advanced argparse with subcommands
• JSON + text output formats
• Comprehensive examples and references
• Error handling and edge case management

POWERFUL Tier Requirements

• Minimum 300 lines in SKILL.md
• 2-3 Python scripts (500-800 LOC each)
• Complex argparse with multiple modes
• Sophisticated output formatting and validation
• Extensive documentation and reference materials
• Advanced error handling and recovery mechanisms
• CI/CD integration capabilities

Architecture & Design

Modular Design Philosophy

The skill-tester follows a modular architecture where each component serves a specific validation purpose:

• skill_validator.py: Core structural and documentation validation engine
• script_tester.py: Runtime testing and execution validation framework
• quality_scorer.py: Multi-dimensional quality assessment and scoring system

Standards Enforcement

All validation is performed against well-defined standards documented in the references/ directory:

• Skill Structure Specification: Defines mandatory and optional components
• Tier Requirements Matrix: Detailed requirements for each skill tier
• Quality Scoring Rubric: Comprehensive scoring methodology and weightings

Integration Capabilities

Designed for seamless integration into existing development workflows:

• Pre-commit Hooks: Prevents substandard skills from being committed
• CI/CD Pipelines: Automated quality gates in pull request workflows
• Manual Validation: Interactive command-line tools for development-time validation
• Batch Processing: Bulk validation and scoring of existing skill repositories

Implementation Details

skill_validator.py Core Functions

# Primary validation workflow
validate_skill_structure() -> ValidationReport
check_skill_md_compliance() -> DocumentationReport  
validate_python_scripts() -> ScriptReport
generate_compliance_score() -> float

Key validation checks include:

• SKILL.md frontmatter parsing and validation
• Required section presence (Description, Features, Usage, etc.)
• Minimum line count enforcement per tier
• Python script argparse implementation verification
• Standard library import enforcement
• Directory structure compliance
• README.md quality assessment

script_tester.py Testing Framework

# Core testing functions
syntax_validation() -> SyntaxReport
import_validation() -> ImportReport
runtime_testing() -> RuntimeReport
output_format_validation() -> OutputReport

Testing capabilities encompass:

• Python AST-based syntax validation
• Import statement analysis and external dependency detection
• Controlled script execution with timeout protection
• Argparse --help functionality verification
• Sample data processing and output validation
• Expected output comparison and difference reporting

quality_scorer.py Scoring System

# Multi-dimensional scoring
score_documentation() -> float  # 25% weight
score_code_quality() -> float   # 25% weight
score_completeness() -> float   # 25% weight
score_usability() -> float      # 25% weight
calculate_overall_grade() -> str # A-F grade

Scoring dimensions include:

• Documentation: Completeness, clarity, examples, reference quality
• Code Quality: Complexity, maintainability, error handling, output consistency
• Completeness: Required files, sample data, expected outputs, test coverage
• Usability: Help text quality, example clarity, installation simplicity

Usage Scenarios

Development Workflow Integration

# Pre-commit hook validation
skill_validator.py path/to/skill --tier POWERFUL --json

# Comprehensive skill testing
script_tester.py path/to/skill --timeout 30 --sample-data

# Quality assessment and scoring
quality_scorer.py path/to/skill --detailed --recommendations

CI/CD Pipeline Integration

# GitHub Actions workflow example
- name: "validate-skill-quality"
  run: |
    python skill_validator.py engineering/${{ matrix.skill }} --json | tee validation.json
    python script_tester.py engineering/${{ matrix.skill }} | tee testing.json
    python quality_scorer.py engineering/${{ matrix.skill }} --json | tee scoring.json

Batch Repository Analysis

# Validate all skills in repository
find engineering/ -type d -maxdepth 1 | xargs -I {} skill_validator.py {}

# Generate repository quality report
quality_scorer.py engineering/ --batch --output-format json > repo_quality.json

Output Formats & Reporting

Dual Output Support

All tools provide both human-readable and machine-parseable output:

Human-Readable Format

=== SKILL VALIDATION REPORT ===
Skill: engineering/example-skill
Tier: STANDARD
Overall Score: 85/100 (B)

Structure Validation: ✓ PASS
├─ SKILL.md: ✓ EXISTS (247 lines)
├─ README.md: ✓ EXISTS  
├─ scripts/: ✓ EXISTS (2 files)
└─ references/: ⚠ MISSING (recommended)

Documentation Quality: 22/25 (88%)
Code Quality: 20/25 (80%)
Completeness: 18/25 (72%)
Usability: 21/25 (84%)

Recommendations:
• Add references/ directory with documentation
• Improve error handling in main.py
• Include more comprehensive examples

JSON Format

{
  "skill_path": "engineering/example-skill",
  "timestamp": "2026-02-16T16:41:00Z",
  "validation_results": {
    "structure_compliance": {
      "score": 0.95,
      "checks": {
        "skill_md_exists": true,
        "readme_exists": true,
        "scripts_directory": true,
        "references_directory": false
      }
    },
    "overall_score": 85,
    "letter_grade": "B",
    "tier_recommendation": "STANDARD",
    "improvement_suggestions": [
      "Add references/ directory",
      "Improve error handling",
      "Include comprehensive examples"
    ]
  }
}

Quality Assurance Standards

Code Quality Requirements

• Standard Library Only: No external dependencies (pip packages)
• Error Handling: Comprehensive exception handling with meaningful error messages
• Output Consistency: Standardized JSON schema and human-readable formatting
• Performance: Efficient validation algorithms with reasonable execution time
• Maintainability: Clear code structure, comprehensive docstrings, type hints where appropriate

Testing Standards

• Self-Testing: The skill-tester validates itself (meta-validation)
• Sample Data Coverage: Comprehensive test cases covering edge cases and error conditions
• Expected Output Verification: All sample runs produce verifiable, reproducible outputs
• Timeout Protection: Safe execution of potentially problematic scripts with timeout limits

Documentation Standards

• Comprehensive Coverage: All functions, classes, and modules documented
• Usage Examples: Clear, practical examples for all use cases
• Integration Guides: Step-by-step CI/CD and workflow integration instructions
• Reference Materials: Complete specification documents for standards and requirements

Integration Examples

Pre-Commit Hook Setup

#!/bin/bash
# .git/hooks/pre-commit
echo "Running skill validation..."
python engineering/skill-tester/scripts/skill_validator.py engineering/new-skill --tier STANDARD
if [ $? -ne 0 ]; then
    echo "Skill validation failed. Commit blocked."
    exit 1
fi
echo "Validation passed. Proceeding with commit."

GitHub Actions Workflow