Benchmark Verification

Benchmark claims are only useful when their context is verified.

Core objective

Decide whether a benchmark claim is directly comparable, partially comparable, or not comparable.

Verification workflow

1. Identify the exact claim to verify.
2. Trace the claim back to the highest-quality available source.
3. Capture the comparison context:
   • model and version
   • dataset and split
   • metric
   • evaluation harness / script
   • prompting or few-shot setup
   • filtering or post-processing
   • hardware / runtime context when relevant
   • date / release / commit context
4. Compare candidates only after the contexts are aligned.
5. Return a verdict:
   • comparable
   • partially comparable
   • not comparable
6. If the benchmark interpretation itself is a critical claim, hand off to verification-protocol for a fuller verification trail.

Required cautions