Create Peer Review

Shadow Mode for Review Quality

Every local review runs in shadow mode alongside the teacher:

1. Tier 0.5 classifier predicts pass/fail per rubric dimension
2. Tier 1.5 GPT generates full review with scores and rationale
3. Teacher generates authoritative review (always runs in shadow)
4. Agreement logged to shadow.jsonl per reviewer persona
5. When reviewer GPT reaches 90% score agreement with teacher → promotion

Exemplar Comparison

Papers are compared against top-cited arxiv exemplars:

• Structural metrics: section word count ratios, citation density, figure-to-text ratio
• Quality signals: abstract information density, introduction flow, evaluation rigor
• Gap analysis: what the exemplars do that the candidate paper doesn't

Reviewer Personas

4 fictional reviewers in personas/reviewers.yaml:

• Alpha (Dr. Yuki Tanaka): Soundness focus, skeptical, demands ablations
• Beta (Dr. Marcus Webb): Novelty focus, constructive, encyclopedic related work knowledge
• Gamma (Dr. Priya Krishnamurthy): Significance focus, pragmatic, production deployment lens
• Delta (Dr. Soren Lindqvist): Adversarial focus, terse, finds security holes

Scoring System

Each reviewer scores on 6 dimensions (1-4 scale, NeurIPS/ICLR convention):

• Soundness: Are claims correct? Methods valid? (1=major flaws, 4=rigorous)
• Technical Novelty: Are the IDEAS new? (ICLR split)
• Empirical Novelty: Are the EXPERIMENTS/RESULTS new? (ICLR split)
• Significance: Does this matter? Who benefits?
• Clarity: Can it be understood and reproduced?
• Presentation: Figures, tables, formatting, code examples

Overall recommendation: decoupled 1-10 holistic judgment (ICLR convention). The weighted signal from dimension scores is advisory — reviewers make holistic calls (e.g., moderate novelty + immense significance = accept).

Accept/reject thresholds:

• Strong Accept: ≥8.0 (top 15%)
• Accept: ≥6.0 (marginally above threshold)
• Borderline: 5.0-6.0
• Reject: <5.0

Built-in Eval: Arxiv Benchmark Corpus

The skill embeds an arxiv benchmark corpus (data/arxiv_benchmarks.json) that provides empirical distributions for structural metrics. Every review command automatically runs a benchmark eval and attaches it to the meta-review, so the Tier 2 reviewer (project agent) has percentile rankings for scoring decisions.

Benchmark Metrics

Papers are scored by percentile rank against the corpus: weak (<p25), ok (p25-p75), strong (>p75).

Growing the Corpus

# Add a paper from local .tex file
bash run.sh ingest-benchmark 2305.05176 --tex /path/to/paper.tex

# Score your paper against the corpus
bash run.sh benchmark paper_output/sensai_cascade/draft.tex

# The corpus auto-grows as /dogpile discovers new arxiv papers

Commands

# Full peer review of a paper
bash run.sh review artifacts/neophyte_paper_v5/

# Benchmark eval: score paper against arxiv corpus
bash run.sh benchmark paper_output/draft.tex

# Add paper to benchmark corpus
bash run.sh ingest-benchmark 2305.05176 --tex path/to/paper.tex

# Compare against arxiv exemplars
bash run.sh compare-exemplars artifacts/neophyte_paper_v5/ --papers 2305.05176,2411.05844

# Score a single section
bash run.sh score-section artifacts/neophyte_paper_v5/sections/eval.tex --reviewer alpha

# Train reviewer GPTs from shadow data
bash run.sh train --reviewer alpha --data ~/.pi/assistant/shadow.jsonl

# Shadow agreement report
bash run.sh shadow-report

# Generate visual review summary (via /create-figure + /analytics)
bash run.sh visualize artifacts/neophyte_paper_v5/ --output review_dashboard.png

Output

Reviews are written to {paper_dir}/reviews/:

reviews/
├── alpha_review.json      # Per-reviewer structured review
├── beta_review.json
├── gamma_review.json
├── delta_review.json
├── meta_review.json       # Aggregated scores + decision
                           # Also includes benchmark_eval percentiles and pen_name_violations
├── exemplar_comparison.json
├── shadow.jsonl           # Shadow observation log
└── figures/
    ├── rubric_radar.png   # Radar chart of scores per reviewer
    ├── exemplar_gap.png   # Gap analysis vs exemplars
    └── score_history.png  # Score convergence across drafts

Self-Improvement Loop

Draft N → 4 reviewer personas score (shadow mode)
        → Teacher confirms/overrides
        → Disagreements logged to shadow.jsonl
        → When agreement ≥90%: reviewer GPT promoted
        → Next draft: reviewer GPT scores autonomously (free, on-device)
        → /create-figure generates visual diff between drafts

Pen Name Enforcement (Non-Negotiable)

Papers reviewed by /create-peer-review MUST use pen names for all persona references. Real people never appear in papers.

The reviewer personas (Alpha/Tanaka, Beta/Webb, Gamma/Krishnamurthy, Delta/Lindqvist) are already fictional and need no mapping.

Enforcement

• /create-peer-review flags any real persona name found in paper text as a HIGH severity finding
• The check runs before structural scoring — pen name violations block the review

Integration with /create-paper

The /create-paper skill can invoke /create-peer-review as a quality gate:

# In create-paper's multi-draft loop:
for draft_num in range(max_drafts):
    generate_draft(section)
    review = peer_review(section, reviewers=["alpha", "beta", "gamma", "delta"])
    if review.decision == "accept":
        break
    revision_notes = review.aggregate_feedback()
    # Feed revision notes back into next draft

/paper-lab Integration

/create-peer-review serves as the final quality gate in the /paper-lab convergence loop:

/paper-lab Phase 2 (headless):
  Round N: /review-paper → fixes → /review-paper (converge on 8.5+)

/paper-lab Phase 3 (final gate):
  /create-peer-review → 4 persona reviews + benchmark eval
  Target: 8+/10 average across all 4 reviewers

Score Interpretation for /paper-lab

• 8+/10: Paper passes — ready for submission
• 6-8/10: Needs another /paper-lab round focused on reviewer feedback
• <6/10: Fundamental issues — return to Phase 1 (/interview with author)

What /create-peer-review Adds Beyond /review-paper

• Arxiv benchmark comparison (structural percentiles)
• Cross-reviewer consensus analysis
• Author-as-reviewer qualitative feedback (senior corpus authors)
• Publication venue readiness assessment

Integration