Real Env Test

Key distinction: This Skill runs inside the current session. It is NOT a separate process. The sub-agent test-runner IS a separate context window.

Four-Phase Workflow

Phase 1: Information Collection (this session)
    ↓
Phase 2: Framework Generation (this session)
    ↓
Phase 3: Test Execution (delegated to test-runner sub-agent)
    ↓
Phase 4: Result Analysis (back in this session)

Phase 1: Information Collection

Goal: Understand the project and gather everything needed to generate the testing framework.

Detailed instructions: Read phase1-collection.md in this directory.

Summary

1. Auto-scan the project: Read build files (Makefile, Cargo.toml, go.mod), deployment docs (README, INSTALL.md), systemd units, config templates.

2. Evaluate completeness and select mode:

   Project docs	User intent	Mode
   Complete	Clear	Fast — ≤3 questions
   Partial	Clear	Semi — ask only about gaps
   Complete	Vague	Semi — ask only about test specifics
   Missing	Vague	Full — 7-category questionnaire

3. Collect missing information via multi-turn dialogue.

4. Present a summary of all inferred + confirmed values for user approval.

Phase 2: Framework Generation

Goal: Generate a complete testing/ directory with all files needed for automated testing.

Detailed instructions: Read phase2-generation.md in this directory.

Summary

Generate the following directory structure:

<project>/testing/
├── Taskfile.yml          # All task definitions
├── env.example.yml       # Configuration template (committed)
├── env.yml               # User's local config (git-ignored)
├── .gitignore            # Ignore env.yml and results/
├── scripts/
│   ├── remote-setup.sh   # Install + configure on remote
│   ├── collect-metrics.sh # System metrics sampling
│   └── collect-logs.sh   # Service log capture
├── results/              # Test results (git-ignored)
└── README.md             # Agent-friendly workflow documentation

Templates are in the templates/ subdirectory of this skill. Use them as the skeleton and fill in project-specific values.

Generation Rules

1. Use templates/Taskfile.yml.tpl — replace {{PLACEHOLDER}} markers
2. Use templates/README.md.tpl — fill in project-specific sections
3. Use templates/env.example.yml — adjust fields for the project
4. Generate scripts/ based on project's deploy/install requirements
5. Create .gitignore with env.yml and results/

Phase 3: Test Execution

Goal: Execute the generated testing workflows on the remote host.

This phase is delegated to the test-runner sub-agent (defined in .claude/agents/test-runner.md).

When to Delegate

• After Phase 2 is complete AND user has filled in env.yml
• User explicitly requests execution (e.g., "run the test now")
• User uses /run-test command directly

How to Delegate

Spawn the test-runner sub-agent with the following message:

Execute testing workflow for project at <path>.

Working directory: <project>/testing/
Workflow: <1|2|3|4> (or specific task commands)
Parameter overrides: <if any, e.g., rate=20, duration=120>

Read the README.md in the testing directory for complete workflow instructions.

What test-runner Does

1. Reads testing/README.md for available workflows
2. Reads testing/env.yml for current configuration
3. Runs task validate to confirm setup
4. Executes the specified workflow step by step
5. Uses async protocol: bench:start → poll bench:status → bench:stop
6. Collects results via task collect
7. Returns a structured summary

What test-runner CANNOT Do

• Spawn sub-agents (it is a leaf executor)
• Modify Taskfile.yml or scripts
• Ask the user questions
• Make autonomous decisions about failures (must escalate)

Phase 4: Result Analysis

Goal: Interpret the test results and report findings to the user.

After test-runner completes:

1. Read result files in testing/results/run_<timestamp>/:

   • benchmark/*.json — benchmark reports
   • metrics/ — system resource usage (CPU, memory, disk I/O)
   • logs/ — service logs

2. Extract key metrics:

   • Total containers created / failed
   • Average pull time, create time, start time
   • p50 / p95 / p99 latencies
   • Throughput (containers/second actual vs target)
   • Error rate and error categories
   • Peak CPU / memory / disk I/O

3. Identify issues:

   • Performance bottlenecks
   • Error patterns
   • Resource saturation points

4. Report to user with:

   • Summary table of key metrics
   • Comparison with previous runs (if available)
   • Specific recommendations

Agent-Friendly Design Requirements

All generated testing frameworks MUST follow these principles:

SSH Constraints

• Every SSH command must complete within 30 seconds
• No interactive prompts — all params come from env.yml
• Use async pattern for long-running operations (nohup + PID + polling)

Async Benchmark Protocol