Agent Testing

Key rules:

• Unit tests MUST run WITHOUT calling an LLM (mock the model)
• Scenario tests MAY call an LLM but SHOULD use a cheap/fast model
• Evaluation tests call production model and use LLM-as-Judge
• Push testing DOWN the pyramid

Unit Testing Checklist

Scenario Testing Checklist

Evaluation (LLM-as-Judge)

Judge pattern: Send query + response + criteria to strong model, get JSON scores (1-5), normalize to 0-1, compare against threshold.

Prompt Versioning

Store prompts as YAML files with metadata (version, author, status, changelog). Use PromptManager for loading, rollback, and diffing.

CI/CD Integration

Anti-Patterns

1. Testing exact strings — LLMs are non-deterministic. Test semantics and structure.
2. Skipping eval tests — "Works on my machine" for AI. Automate quality regression.
3. Hardcoding outputs — Test properties, not specific text.
4. Ignoring cost — Tier execution: critical cases on PR, full suite nightly.
5. Testing formatting over substance — Content matters more than bullet format.

@reference references/code-examples.md — Load when writing actual test implementations (Python unit/scenario/eval tests, TypeScript Vitest setup, LLM-as-Judge class, CI workflow YAML, prompt versioning system)

Project Structure

tests/
├── conftest.py          # Shared config, markers, cost control
├── unit/                # Fast, no LLM
├── scenario/            # Medium cost, fixtures/
│   └── fixtures/*.json  # Recorded conversations
└── evaluation/          # Expensive, nightly
    ├── baselines/       # Known-good scores
    ├── judge.py         # LLM-as-Judge
    └── reporter.py      # Result collection

Commands

pytest tests/unit/ -v -x                              # Unit (always)
pytest tests/scenario/ -v                              # Scenario (on PR)
RUN_EVAL_TESTS=true pytest tests/evaluation/ -v -m eval  # Eval (nightly)