Eval

/eval — Run Agent Quality Evals

Execute eval scenarios to measure agent output quality mechanically.

What It Does

1. Reads scenario definitions from evals/scenarios/
2. For each scenario: sets up workspace, runs agent, checks output
3. Scores: files exist, content matches, forbidden patterns absent, commands pass
4. Saves results to evals/results/ as timestamped JSON
5. Prints summary scorecard

Usage

Run all scenarios:

./scripts/eval.sh

Run a specific scenario:

./scripts/eval.sh evals/scenarios/$ARGUMENTS

After Running