Create a comparison fairness matrix assessing whether alternatives are evaluated under symmetric constraints. Fairness lens answering "Are alternatives compared under symmetric constraints?"
Philosophical Mode: Fairness Primary Question: "Are alternatives compared under symmetric constraints?" Focus: Compute Budget Symmetry, Tuning Protocol Parity, Data Access Equality, Engineering Effort Balance, Winner's Curse
/exp-lens-fair-comparison or /make-experiment-diag fairnessNEVER:
ALWAYS:
/mermaid skill using the Skill tool - this is MANDATORYSpawn Explore subagents to investigate:
Compute & Resource Allocation
Tuning Protocol per Method
Data Access & Preprocessing
Engineering Effort Indicators
Reporting Completeness
Build the symmetry matrix: rows = methods compared, columns = resource dimensions (compute, tuning, data, engineering, disclosure).
For each cell:
CRITICAL — Analyze Effort Attribution: For every claimed improvement:
Use the mermaid skill conventions to create a symmetry diagram with:
Direction: LR (methods flow through resource allocation to evaluation)
Subgraphs:
Node Styling:
cli class: Proposed methodphase class: Comparator methodshandler class: Shared resourcesgap class: Asymmetric resourcesdetector class: Symmetry checksoutput class: ResultsWrite the diagram to: temp/exp-lens-fair-comparison/exp_diag_fair_comparison_{YYYY-MM-DD_HHMMSS}.md
# Fair Comparison Analysis: {Experiment Name}
**Lens:** Fair Comparison (Fairness)
**Question:** Are alternatives compared under symmetric constraints?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Symmetry Matrix
| Method | Compute | Tuning Budget | Data Access | Engineering | Disclosure |
|--------|---------|---------------|-------------|-------------|------------|
| {proposed method} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |
| {comparator} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |
## Resource Disclosure
| Resource Type | Proposed Method | Comparators | Symmetric? |
|---------------|-----------------|-------------|------------|
| {GPU hours} | {value} | {value} | {Yes/No} |
| {Tuning trials} | {value} | {value} | {Yes/No} |
| {Extra data} | {value} | {value} | {Yes/No} |
## Symmetry Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Methods ["METHODS"]
direction TB
PROP["Proposed Method<br/>━━━━━━━━━━<br/>Method under<br/>evaluation"]
COMP["Comparator Methods<br/>━━━━━━━━━━<br/>Baseline and<br/>prior work"]
end
subgraph Resources ["RESOURCE ALLOCATION"]
direction TB
SHARED["Shared Resources<br/>━━━━━━━━━━<br/>Same data, same<br/>evaluation protocol"]
ASYM["Asymmetric Resources<br/>━━━━━━━━━━<br/>Differential compute<br/>or tuning budget"]
CHECK["Symmetry Check<br/>━━━━━━━━━━<br/>Verify parity<br/>across methods"]
end
subgraph Eval ["EVALUATION"]
direction TB
RESULT["Results<br/>━━━━━━━━━━<br/>Reported<br/>performance"]
end
PROP --> SHARED
COMP --> SHARED
PROP --> ASYM
SHARED --> CHECK
ASYM --> CHECK
CHECK --> RESULT
%% CLASS ASSIGNMENTS %%
class PROP cli;
class COMP phase;
class SHARED handler;
class ASYM gap;
class CHECK detector;
class RESULT output;
| Factor | Proposed Method Advantage | Impact on Claimed Improvement |
|---|---|---|
| {tuning trials} | {advantage} | {estimated impact} |
| {engineering tricks} | {advantage} | {estimated impact} |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-comparator-construction` - For baseline selection and construction adequacy
- `/exp-lens-sensitivity-robustness` - For sensitivity analysis across conditions