Judge Rubric

Required inputs

• The task contract or judging objective.
• The domain context, expected outputs, and any non-negotiable constraints.
• The artifacts that will later be judged, such as final outputs, process traces, validation logs, or benchmark summaries.
• Any scoring mode, reference answer, criteria list, benchmark notes, or existing rubric fragments already available.

Rubric authoring workflow

1. Clarify the judging target first. State what is being judged, what success looks like, what evidence is allowed, and what the ruling must decide.
2. Classify the evidence shape. Decide whether the upcoming judgment is outcome-focused, process-aware, or hybrid so the rubric fits the actual artifacts rather than a generic template.
3. Lock 3 to 7 rubric dimensions before scoring starts. Prefer concrete dimensions such as correctness, constraint compliance, evidence quality, tool reliability, completeness, safety, format fidelity, or domain-specific policy adherence.
4. Define explicit pass, partial, and fail boundaries for each dimension. Behavioral anchors should say what observable evidence earns each band and what failure modes force a downgrade.
5. State the evidence ledger requirements for every dimension. Name which artifacts may support a score, what counts as corroboration, and what evidence is too weak to rely on.
6. Add aggregation and tie-break rules. Specify whether dimensions are equal-weighted or prioritized, then lock tie-breakers such as stronger rubric compliance, clearer evidence anchoring, lower evaluator risk, better benchmark fit, or clearer writing.
7. Add robustness controls before the rubric is used. Include order-bias checks, benchmark-overfitting skepticism, confidence guidance, and a rule that unsupported chain-of-thought or self-explanations are low-trust evidence unless confirmed by artifacts.
8. Return a decision-ready rubric package that another judge can apply without reinterpreting the task halfway through scoring.

Output package

• Judging target and evidence mode.
• Locked rubric table with dimensions, rationale, pass-partial-fail boundaries, and allowed evidence.
• Aggregation and tie-break rules.
• Robustness checks and confidence guidance.
• Missing inputs or blockers that must be resolved before the rubric is safe to use.

Structured helper

Use scripts/render_rubric.py when the rubric inputs are already structured and you want a deterministic first draft.

Use scripts/render_rubric.py --validate-only when you need to verify that a structured contract is complete and consistent before rendering or handing it to another judging workflow.

Accepted contract fields:

• domain
• task
• evidence_mode
• decision
• dimensions: array of objects with name, why_it_matters, pass_boundary, partial_boundary, fail_boundary, and allowed_evidence
• aggregation_rules: object with weighting_or_priority, non_negotiable_failures, and tie_breakers
• robustness_checks: object with order_bias_check, evidence_quality_check, benchmark_overfitting_check, and confidence_guidance
• blockers: object with missing_inputs, weak_evidence_areas, and clarifications_needed

Example:

python scripts/render_rubric.py --input-file contract.json --output-file rubric.md

Validation example:

python scripts/render_rubric.py --input-file contract.json --validate-only

Assets

• assets/rubric-template.md provides a reusable template for the final rubric package.

Notes

• Keep the rubric task-adaptive, but only adapt it before scoring begins.
• Prefer observable evidence over polished narration.
• If the domain is under-specified, stop and list the missing inputs instead of improvising score boundaries.