Test Priority Engine

Input Contract

Hard preconditions — refuse rather than emit a plan the user should not trust:

1. At least ONE priority signal must exist. If ALL of these are absent — no changed files, no baseline, no flakiness history, no scenario-set — the skill refuses with remediation "not enough signal to prioritize; run a full regression first to establish a baseline".
2. --mode must be one of the three canonical values. Typos (quik) block — silent defaulting is how modes silently shift.
3. If --time-budget is set, it must be positive and ≥ 10 seconds. A 5-second budget is a footgun — the skill would emit an empty plan.

Algorithm

Step 1 — Resolve the mode + budgets

Default mode by trigger:

See references/mode-budgets.md for the concrete time/count budgets per mode. User overrides (--time-budget / --count-budget) only TIGHTEN, never loosen — a mode is a floor, not a ceiling.

Step 2 — Derive the affected set

For every changed source file:

1. Call codebase-intel MCP's ci_dependency_graph to get transitive dependents.
2. For each dependent, find test files that import it (either directly or via the project's module resolver).
3. Add those test files to the affected set.
4. If codebase-intel is unavailable, fall back to directory proximity: every *.test.* / *.spec.* file under the same directory subtree as a changed file. The fallback is less precise but never misses a test; record "degraded affected-set derivation" in the run report so consumers know.

A test can be in the affected set for multiple changed files — that's a signal; it becomes part of the risk score (see risk-model.md).

Step 3 — Score every candidate

The candidate set is the union of:

• The affected set from Step 2
• Every P0 test in the baseline (P0 tests are always candidates)
• Every test in regression-baseline.json.flakyKnown

For each candidate, compute a risk score using the formula in references/risk-model.md. The score is a pure function of the inputs; two runs with identical inputs produce identical scores.

Step 4 — Enforce the P0 mandatory set

Every affected P0 test appears in the plan regardless of mode or budget. This is the gate contract. The budget squeezes non-P0 tests first, then P1, then P2, then P3 — but a P0 test that maps to a changed file is pinned in the plan as a non-negotiable inclusion.

If the P0 mandatory set alone exceeds the mode budget, the skill emits the full P0 set and records the budget overflow in the report as budgetExceeded: true, reason: "P0 mandatory set". The run continues — exceeding the budget for P0 coverage is the right trade-off and the operator sees exactly why.

Step 5 — Budget-fit the non-P0 tail

Sort the non-P0 candidates by risk score descending. Pack them into the remaining budget (time budget preferred; falls back to count budget when per-test duration is unknown — see mode-budgets.md §3 for the cold-start path).

Packing is first-fit by risk score. Tests that don't fit are NOT silently dropped — they go into the plan's "spill list" with their score and the reason they didn't fit. regression-test-runner + humans can decide to run them in a follow-up pass.

Step 6 — Write the plan

Emit .vibeflow/reports/priority-plan.md. Every test entry includes:

• Scheduled position (1..N)
• Test id + file
• Risk score + contributing factors (so the human can tell WHY)
• Priority tier
• Expected duration (from baseline)
• Cumulative budget used (running total)

The report is deterministic — same inputs, byte-identical output. Downstream tools can cache the plan and re-use it when inputs haven't changed.

Output Contract

priority-plan.md

# Priority Plan — <runId>

## Header
- Mode: quick | smart | full
- Time budget: 300s
- Count budget: 100
- Since: <base sha>
- Candidate pool: N tests (affected=a, p0=b, flakyKnown=c)
- Planned: P
- Spilled: S
- Budget exceeded: no | yes (reason: <reason>)

## Plan
| # | Test id | File | Prio | Risk | Duration | Cumulative | Reason |
|---|---------|------|------|------|----------|------------|--------|
| 1 | auth.test.ts::login >... | tests/unit/auth.test.ts | P0 | 0.92 | 17ms | 17ms | affected + baseline-fail×2 + P0 weight |
| 2 | ... |

## Spill (over budget)
| Test id | Prio | Risk | Reason it didn't fit |
|---------|------|------|----------------------|
| ... | P2 | 0.34 | time budget exhausted at 290/300s |

## Degraded signals (if any)
- codebase-intel unavailable — used directory-proximity fallback
- flakiness.json missing — risk score omits flake component

Machine-readable counterpart

.vibeflow/artifacts/priority/plan-<runId>.json — same data in a stable JSON shape consumed by regression-test-runner and learning-loop-engine. Fields match the table columns 1:1.

Gate Contract

Every affected P0 test appears in the plan, regardless of mode or budget. No other condition can push a P0 out. Three ways to misinterpret this and their responses:

1. P0 mandatory set exceeds the budget → plan emits the full P0 set with budgetExceeded: true. The operator sees the reason and can decide to raise the budget or split the run.
2. A P0 test has no baseline entry (cold start, new test file) → it still lands in the plan, with risk score computed from priority weight alone and degradedSignals: ["no baseline"] noted on its row.
3. A P0 test is tagged @quarantined → it is NOT in the plan. Quarantine is the human's explicit decision to ignore a test; the priority engine respects it. The plan's header notes the quarantined count so humans can audit.

Non-Goals

• Does NOT run tests.
• Does NOT modify regression-baseline.json.
• Does NOT invent priority tiers; the priority comes from scenario-set.md / baseline.tags / the test file's own tags.
• Does NOT try to predict which tests will fail. Risk is a historical + structural signal, not a failure oracle.
• Does NOT retry or re-order on failure. Order is decided up front and written once.

Downstream Dependencies

• regression-test-runner — consumes the plan as an ordering hint when it dispatches tests. incremental scope can use the plan's top-K directly.
• learning-loop-engine — ingests plans + outcomes over time to learn which risk factors predict real failures.
• release-decision-engine — reads the plan's budgetExceeded flag as a gate signal (a release run should never be budget- exceeded; the pre-release pipeline has no time ceiling).
• observability-analyzer — correlates plan position with actual failure outcomes to feed a feedback loop into the risk model.