Plan Research Paper Review

Step 0: Paper Context + Mode Selection

0A. Paper Identification

Read the paper (PDF or markdown). Extract:

• Title, authors, venue/target
• Abstract (full text)
• Key claims (numbered list)
• Methodology summary (1-2 sentences)
• Main results (key numbers)

0B. Mode Selection

Present three options:

1. AUTHOR: You wrote this paper and want it torn apart before submission. Maximum rigor on clarity, methodology, missing experiments, and anticipating reviewer objections. The question: "What will Reviewer 2 complain about?"
2. REVIEWER: You're evaluating an external paper. Balanced assessment of strengths, weaknesses, novelty, and significance. Produce a structured review suitable for a conference/journal. The question: "Should this be accepted?"
3. IMPLEMENT: You want to extract engineering value from this paper. Focus on the practical algorithm, key hyperparameters, implementation tricks, and what to ignore. The question: "How do I build this?"

Context-dependent defaults:

• Paper found in repo's docs/ or drafts/ → default AUTHOR
• Paper provided as external PDF/URL → default REVIEWER
• User says "implement" or "build" → default IMPLEMENT

STOP. AskUserQuestion with mode recommendation.

Review Sections

Section 1: Novelty & Related Work Assessment

Evaluate:

• What is the claimed contribution? (List each one)
• For each contribution — is it genuinely novel or incremental?
• Related work coverage — are obvious comparisons missing?
• Positioning — does the paper fairly represent prior work?
• Concurrent work — any recent papers doing something similar?

  CLAIMED CONTRIBUTION      | NOVEL? | CLOSEST PRIOR WORK | DELTA
  --------------------------|--------|-------------------|------
  1. [contribution]         | Y/N/△  | [citation]        | [what's different]
  2. [contribution]         | Y/N/△  | [citation]        | [what's different]

△ = incremental improvement over prior work (not truly novel, but valid if significant)

AUTHOR mode: List 3-5 papers the reviewers will ask "how is this different from X?" Preemptively address each.

STOP. AskUserQuestion once per issue. Recommend + WHY.

Section 2: Methodology Review

Evaluate:

• Is the method clearly described? Could you implement it from the paper alone?
• Mathematical notation — consistent, well-defined, standard?
• Assumptions — stated explicitly? Are they reasonable?
• Algorithm pseudocode — present and correct?
• Complexity analysis — time and space complexity stated?
• Limitations — honestly discussed?

For each methodological component:

  COMPONENT               | CLEAR? | CORRECT? | REPRODUCIBLE? | ISSUE
  ------------------------|--------|----------|--------------|-------
  Problem formulation     | Y/N    | Y/N/?    | Y/N          |
  Model architecture      | Y/N    | Y/N/?    | Y/N          |
  Training procedure      | Y/N    | Y/N/?    | Y/N          |
  Loss function           | Y/N    | Y/N/?    | Y/N          |
  Inference procedure     | Y/N    | Y/N/?    | Y/N          |

STOP. AskUserQuestion once per issue. Recommend + WHY.

Section 3: Experimental Evaluation Review

Evaluate:

• Datasets — appropriate? Sufficient size? Standard benchmarks?
• Baselines — strong and fair? State-of-the-art included?
• Metrics — appropriate for the task? Multiple metrics reported?
• Statistical rigor — confidence intervals? Multiple runs? Significance tests?
• Ablation studies — which components contribute how much?
• Hyperparameter sensitivity — how fragile are the results?
• Compute budget — training time, hardware, energy cost reported?

  EXPERIMENT              | DATASET  | BASELINES | STATS  | ABLATION | VERDICT
  ------------------------|---------|-----------|--------|----------|--------
  Main result (Table 1)   | ___     | Strong/Weak| Y/N   | Y/N      | ✓/✗/?
  Ablation (Table 2)      | ___     | N/A       | Y/N   | YES      | ✓/✗/?
  Scaling (Figure 3)      | ___     | Y/N       | Y/N   | N/A      | ✓/✗/?

Red flags:

• CRITICAL: Results reported without variance/std → unreliable
• CRITICAL: No comparison to strong baselines → unverifiable improvement
• WARNING: Single dataset → unclear generalization
• WARNING: No ablation → unclear which component matters
• WARNING: Compute budget not reported → unreproducible

STOP. AskUserQuestion once per issue. Recommend + WHY.

Section 4: Clarity & Presentation Review

Evaluate:

• Abstract — accurately represents the paper? All claims supported?
• Introduction — motivates the problem clearly? States contributions explicitly?
• Figures — informative? Self-contained? High quality?
• Tables — well-formatted? Key results highlighted?
• Writing quality — clear, concise, no jargon without definition?
• Paper length — appropriate for the contribution size?
• References — complete? Properly formatted?

AUTHOR mode additions:

• Rewrite suggestions for unclear paragraphs
• Figure improvement suggestions
• Title alternatives if current is weak

STOP. AskUserQuestion once per issue. Recommend + WHY.

Section 5: Reproducibility Assessment

Evaluate:

• Code availability — released? Link provided?
• Data availability — public dataset? Or proprietary?
• Hyperparameters — all listed? Search strategy described?
• Hardware requirements — specified?
• Random seeds — reported?
• Pre-trained models — available?
• Training details — batch size, learning rate schedule, epochs, early stopping?

Reproducibility score:

  CRITERION                | STATUS | NOTES
  -------------------------|--------|------
  Code released            | Y/N/P  |
  Data accessible          | Y/N/P  |
  Hyperparameters listed   | Y/N/P  |
  Hardware specified       | Y/N    |
  Random seeds reported    | Y/N    |
  Training time reported   | Y/N    |
  Pre-trained models       | Y/N    |
  SCORE: ___/7             |        |

(P = partial)

STOP. AskUserQuestion once per issue. Recommend + WHY.

Section 6: Implementation Guide (IMPLEMENT mode only)

Extract:

• Core algorithm in pseudocode
• Key hyperparameters with recommended values
• Implementation tricks mentioned in the paper
• Common pitfalls (from methodology/ablation sections)
• What to skip (paper components that don't matter for practical use)
• Estimated implementation effort (days/weeks)
• Dependencies and frameworks needed

  IMPLEMENTATION PLAN:

  1. Data preparation:     [what format, preprocessing]
  2. Model architecture:   [layers, dimensions, key design choices]
  3. Training loop:        [loss, optimizer, schedule, batch size]
  4. Evaluation:           [metrics, baselines to compare against]
  5. Key tricks:           [things that make it actually work]
  6. Skip:                 [paper components not needed for practical use]

STOP. AskUserQuestion once per issue. Recommend + WHY.

CRITICAL RULE — How to ask questions

Every AskUserQuestion MUST: (1) present 2-3 concrete lettered options, (2) state which option you recommend FIRST, (3) explain in 1-2 sentences WHY. One issue per question.

Required Outputs

REVIEWER mode — Structured Review

  SUMMARY: [2-3 sentence summary]

  RECOMMENDATION: Strong Accept / Accept / Weak Accept / Borderline / Weak Reject / Reject / Strong Reject

  CONFIDENCE: 1-5 (1 = guess, 5 = expert in this exact area)

  STRENGTHS:
  1. [specific strength with evidence]
  2. [specific strength with evidence]
  3. [specific strength with evidence]

  WEAKNESSES:
  1. [specific weakness with evidence]
  2. [specific weakness with evidence]
  3. [specific weakness with evidence]

  QUESTIONS FOR AUTHORS:
  1. [specific question]
  2. [specific question]

  MISSING REFERENCES:
  - [citation that should be discussed]

  MINOR ISSUES:
  - [typos, formatting, clarity issues]

AUTHOR mode — Revision Checklist

Prioritized list of changes before submission, with effort estimates.

IMPLEMENT mode — Engineering Spec

Pseudocode, hyperparameters, architecture diagram, and implementation plan.

Completion Summary

  +====================================================================+
  |            RESEARCH PAPER REVIEW — SUMMARY                         |
  +====================================================================+
  | Mode selected        | AUTHOR / REVIEWER / IMPLEMENT               |
  | Paper                | [title]                                      |
  | Novelty              | High / Medium / Low / Incremental            |
  | Methodology          | ___ issues found                             |
  | Experiments          | ___ red flags, ___ CRITICAL                  |
  | Reproducibility      | ___/7                                        |
  | Recommendation       | [accept/reject/revise]                       |
  | Critical decisions   | ___ unresolved                               |
  +====================================================================+