Create Comparator Construction experimental design analysis assessing whether baselines and controls are fair and relevant. Counterfactual lens answering "Is the comparator fair and relevant?"
Philosophical Mode: Counterfactual Primary Question: "Is the comparator fair and relevant?" Focus: Baseline Choice, Control Realism, Version Matching, Effort Symmetry, Baseline Drift
/exp-lens-comparator-construction or /make-experiment-diag comparatorNEVER:
/mermaid skill using the Skill tool - this is MANDATORYSpawn Explore subagents to investigate:
Baseline/Control Definitions
Implementation Parity
Version & Environment Match
Tuning Protocol Symmetry
Temporal Baseline Drift
For each comparator, assess:
CRITICAL — Analyze Counterfactual Quality: For each treatment-vs-comparator pair:
Build a fairness matrix with rows = comparators, columns = fairness dimensions.
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: LR (treatment and comparator flow in parallel toward evaluation)
Subgraphs: "PROPOSED METHOD", "COMPARATOR(S)", "SHARED EVALUATION"
Node Styling:
cli class: proposed method nodesphase class: comparator method nodeshandler class: shared evaluation pipeline nodesoutput class: results nodesgap class: asymmetries flaggeddetector class: parity checksWrite the analysis to: temp/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md
# Comparator Construction Analysis: {Experiment Name}
**Lens:** Comparator Construction (Counterfactual)
**Question:** Is the comparator fair and relevant?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Comparator Inventory
| Comparator | Source | Reimplemented? | Same Environment? | Same Tuning Budget? |
|------------|--------|---------------|-------------------|---------------------|
| {name} | {paper/repo} | Yes / No / Partial | Yes / No | Yes / No / Unknown |
## Fairness Matrix
| Comparator | Best Available? | Equal Effort? | Same Env? | Symmetric Tuning? | Temporally Current? |
|------------|----------------|--------------|-----------|-------------------|---------------------|
| {name} | Yes / No | Yes / No | Yes / No | Yes / No | Yes / No |
## Comparison Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Proposed ["PROPOSED METHOD"]
METHOD["Proposed Method<br/>━━━━━━━━━━<br/>{method name}"]
end
subgraph Comparators ["COMPARATOR(S)"]
COMP1["Comparator 1<br/>━━━━━━━━━━<br/>{name}"]
COMP2["Comparator 2<br/>━━━━━━━━━━<br/>{name}"]
end
subgraph Evaluation ["SHARED EVALUATION"]
EVAL["Evaluation Pipeline<br/>━━━━━━━━━━<br/>{dataset/benchmark}"]
RESULTS["Results<br/>━━━━━━━━━━<br/>{metrics reported}"]
PARITY["Parity Check<br/>━━━━━━━━━━<br/>{asymmetry found}"]
ASYM["Asymmetry<br/>━━━━━━━━━━<br/>{description}"]
end
METHOD -->|"evaluated on"| EVAL
COMP1 -->|"evaluated on"| EVAL
COMP2 -->|"evaluated on"| EVAL
EVAL --> RESULTS
RESULTS --> PARITY
PARITY -.->|"flagged"| ASYM
class METHOD cli;
class COMP1,COMP2 phase;
class EVAL handler;
class RESULTS output;
class PARITY detector;
class ASYM gap;
Color Legend:
| Color | Category | Description |
|---|---|---|
| Dark Blue | Proposed Method | The method being evaluated |
| Purple | Comparators | Baselines and controls |
| Orange | Evaluation | Shared evaluation pipeline |
| Dark Teal | Results | Reported outcomes |
| Red | Parity Checks | Fairness verification points |
| Yellow | Asymmetries | Flagged unfair differences |
| # | Asymmetry | Affects | Impact Assessment | Remediation |
|---|---|---|---|---|
| 1 | {description} | {comparator(s)} | High / Medium / Low | {how to fix} |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-estimand-clarity` - For clarifying what the comparison is measuring
- `/exp-lens-fair-comparison` - For deeper analysis of evaluation protocol fairness