Test Gap Analyzer

Usage

# Analyze existing coverage report
python scripts/coverage_analyzer.py coverage.xml -o human

# Detect coverage gaps
python scripts/gap_detector.py --src lib/ --tests tests/ --threshold 80

# Generate test stubs for uncovered code
python scripts/gap_detector.py --src lib/ --tests tests/ -o json > gaps.json
python scripts/test_stub_generator.py gaps.json --output tests/generated/

Test Structure

Shell Tests (BATS)

tests/
├── unit/           # Function-level tests
│   └── feature.bats
├── integration/    # Command workflow tests
│   └── workflow.bats
├── golden/         # Output format tests
│   └── output.bats
└── fixtures/       # Test data
    └── setup.bash

Python Tests (pytest)

tests/
├── test_layer0/    # Foundation tests
│   ├── test_exit_codes.py
│   ├── test_colors.py
│   └── test_constants.py
├── test_layer1/    # Basic helper tests
│   ├── test_logging.py
│   ├── test_error_json.py
│   ├── test_config.py
│   └── test_file_ops.py
├── test_layer2/    # Business logic tests
│   ├── test_validation.py
│   └── test_task_ops.py
├── test_layer3/    # Orchestration tests
│   ├── test_migrate.py
│   ├── test_hierarchy_unified.py
│   └── test_doctor.py
├── conftest.py     # Shared fixtures
└── pytest.ini      # pytest configuration

Coverage Analysis Methodology

Phase 1: Catalog Functions

Shell Functions

# Extract all function definitions from lib/
grep -rn '^[[:space:]]*[a-z_][a-z0-9_]*()' lib/*.sh | \
  sed 's/().*//' | \
  sed 's/.*://' | \
  sort -u

Python Functions

# Extract function definitions from lib/layer*
grep -rn "^def \|^    def " lib/layer*/*.py | \
  sed 's/def //' | \
  sed 's/(.*$//' | \
  sort -u

# Extract class definitions
grep -rn "^class " lib/layer*/*.py | \
  sed 's/class //' | \
  sed 's/(.*$//' | \
  sort -u

Phase 2: Map to Tests

Shell Functions

For each function:

1. Search tests/**/*.bats for function name
2. Check if function is called in test context
3. Record test file and test name

Python Functions

For each function/class:

1. Search tests/test_layer*/*.py for function/class name
2. Check for test functions (test_*) that exercise the code
3. Run pytest --cov=lib --cov-report=term-missing for precise coverage

Phase 3: Identify Gaps

Functions without test coverage:

• No test file references function
• Function called but not asserted
• Only happy path tested

Phase 4: Analyze Quality

Output Format

Coverage Report

# Test Coverage Report

## Summary

- **Total Functions**: {N}
- **Tested Functions**: {N}
- **Coverage**: {X}%
- **Critical Gaps**: {N}

## Coverage by File

| File | Functions | Tested | Coverage |
|------|-----------|--------|----------|
| lib/validation.sh | 15 | 12 | 80% |
| lib/task-ops.sh | 22 | 18 | 82% |
| lib/file-ops.sh | 8 | 3 | 38% |

## Untested Functions (Critical)

| Function | File | Risk | Recommended Test |
|----------|------|------|------------------|
| `atomic_write` | file-ops.sh | High | Unit test with failure scenarios |
| `validate_schema` | validation.sh | High | Integration test with invalid data |

## Partially Tested (Needs Improvement)

| Function | File | Missing |
|----------|------|---------|
| `add_task` | task-ops.sh | Error paths, edge cases |
| `update_status` | task-ops.sh | Invalid status values |

## Test Quality Issues

### Missing Negative Tests
- `validate_task_id` - No test for invalid format
- `parse_json` - No test for malformed input

### Missing Edge Cases
- `get_task_by_id` - No test for empty todo.json
- `list_tasks` - No test for >1000 tasks

## Generated Test Stubs

See: `{{OUTPUT_DIR}}/generated-tests/`

Test Stub Template

#!/usr/bin/env bash
# Tests for {{FUNCTION_NAME}}
# Generated: {{DATE}}

load '../fixtures/setup.bash'

setup() {
    setup_test_environment
}

teardown() {
    teardown_test_environment
}

# Happy path
@test "{{FUNCTION_NAME}} should {{EXPECTED_BEHAVIOR}}" {
    # Arrange
    {{SETUP_CODE}}

    # Act
    run {{FUNCTION_NAME}} {{ARGS}}

    # Assert
    assert_success
    assert_output --partial "{{EXPECTED_OUTPUT}}"
}

# Error case
@test "{{FUNCTION_NAME}} should fail with {{ERROR_CONDITION}}" {
    # Arrange
    {{ERROR_SETUP}}

    # Act
    run {{FUNCTION_NAME}} {{ERROR_ARGS}}

    # Assert
    assert_failure
    assert_output --partial "{{ERROR_MESSAGE}}"
}

# Edge case
@test "{{FUNCTION_NAME}} should handle {{EDGE_CASE}}" {
    # Arrange
    {{EDGE_SETUP}}

    # Act
    run {{FUNCTION_NAME}} {{EDGE_ARGS}}

    # Assert
    {{EDGE_ASSERTIONS}}
}

Python Test Stub Template (pytest)

"""Tests for {{MODULE_NAME}}.{{FUNCTION_NAME}}
Generated: {{DATE}}
"""

import pytest
from lib.layer{{N}}.{{MODULE_NAME}} import {{FUNCTION_NAME}}


class Test{{FUNCTION_NAME_CAMEL}}:
    """Tests for {{FUNCTION_NAME}} function."""

    # Happy path
    def test_{{FUNCTION_NAME}}_success(self):
        """{{FUNCTION_NAME}} should {{EXPECTED_BEHAVIOR}}."""
        # Arrange
        {{SETUP_CODE}}

        # Act
        result = {{FUNCTION_NAME}}({{ARGS}})

        # Assert
        assert result == {{EXPECTED_OUTPUT}}

    # Error case
    def test_{{FUNCTION_NAME}}_fails_with_{{ERROR_CONDITION}}(self):
        """{{FUNCTION_NAME}} should fail with {{ERROR_CONDITION}}."""
        # Arrange
        {{ERROR_SETUP}}

        # Act & Assert
        with pytest.raises({{EXCEPTION_TYPE}}):
            {{FUNCTION_NAME}}({{ERROR_ARGS}})

    # Edge case
    def test_{{FUNCTION_NAME}}_handles_{{EDGE_CASE}}(self):
        """{{FUNCTION_NAME}} should handle {{EDGE_CASE}}."""
        # Arrange
        {{EDGE_SETUP}}

        # Act
        result = {{FUNCTION_NAME}}({{EDGE_ARGS}})

        # Assert
        {{EDGE_ASSERTIONS}}


# Fixtures (if needed)
@pytest.fixture
def {{FIXTURE_NAME}}():
    """{{FIXTURE_DESCRIPTION}}"""
    return {{FIXTURE_VALUE}}

Python Coverage Commands

# Run pytest with coverage for all lib layers
pytest --cov=lib --cov-report=term-missing tests/

# Coverage for specific layer
pytest --cov=lib.layer1 --cov-report=html tests/test_layer1/

# Generate coverage report by layer
pytest --cov=lib.layer0 --cov=lib.layer1 --cov=lib.layer2 --cov=lib.layer3 \
       --cov-report=term-missing --cov-report=html:coverage_html tests/

Task System Integration

@_shared/templates/skill-boilerplate.md#task-integration

Skill-Specific Steps

1. Catalog all functions in lib/.sh (shell) and lib/layer/*.py (Python)
2. Map functions to test files (BATS or pytest)
3. Identify coverage gaps
4. Analyze test quality
5. Generate test stubs for gaps (BATS for shell, pytest for Python)
6. Write report to {{OUTPUT_DIR}}/{{DATE}}_{{SLUG}}.md
7. Write test stubs to {{OUTPUT_DIR}}/generated-tests/

Subagent Protocol

@_shared/templates/skill-boilerplate.md#subagent-protocol

Additional Output

Test stubs MUST be written to: {{OUTPUT_DIR}}/generated-tests/*.bats

Manifest Entry

@_shared/templates/skill-boilerplate.md#manifest-entry

Context Variables

Priority Matrix

Anti-Patterns

@_shared/templates/anti-patterns.md#testing-anti-patterns

Coverage-Specific Anti-Patterns

Error Handling

@_shared/templates/skill-boilerplate.md#error-handling

Completion Checklist

@_shared/templates/skill-boilerplate.md#completion-checklist

Skill-Specific Checklist

• All lib/*.sh functions cataloged (shell)
• All lib/layer*/*.py functions cataloged (Python)
• Functions mapped to test files (BATS for shell, pytest for Python)
• Coverage percentages calculated (pytest --cov for Python)
• Gaps identified and prioritized
• Test quality analyzed
• Test stubs generated for critical gaps
• Test files written to generated-tests/

Skill Chaining

@_shared/protocols/skill-chain-contracts.md

Produces

Consumes

Chain Relationships

The test-gap-analyzer identifies what tests are needed; test-writer-pytest creates the actual tests.