Paper Critique Framework

Pass 2: Detailed Read (60-90 minutes)

Read the full paper. Annotate as you go:

Annotation symbols:
  ? = I don't understand this
  ! = This is interesting / strong point
  X = I disagree / see a problem
  → = This needs more evidence or justification
  ≈ = This is similar to [existing work] — check novelty

Focus on:

• Claims vs. evidence: Is every major claim supported by data?
• Methodology: Are the methods appropriate for the research question?
• Experimental design: Are baselines fair? Are ablations sufficient?
• Figures and tables: Do they support the narrative? Are they readable?
• Writing quality: Is it clear, concise, and well-organized?

Pass 3: Verification (30-60 minutes)

For papers you're seriously evaluating:

• Check key references — do they say what the authors claim?
• Verify mathematical derivations (spot-check, not exhaustive)
• Examine statistical claims (p-values, confidence intervals, effect sizes)
• Check for cherry-picking in results (only best runs? selected metrics?)
• Look for missing baselines that should have been compared

Review Report Structure

## Summary (3-5 sentences)
[Describe what the paper does, the approach, and the main finding.
 Demonstrate that you understood the paper.]

## Strengths (bulleted list)
- S1: [Specific strength with evidence from the paper]
- S2: [Another strength]
- S3: [Another strength]

## Weaknesses (bulleted list, ordered by severity)
- W1 (Major): [Specific weakness + why it matters + suggestion to fix]
- W2 (Major): [Another major weakness]
- W3 (Minor): [A less critical issue]
- W4 (Minor): [Another minor issue]

## Questions for Authors
- Q1: [Something you'd like clarified]
- Q2: [A concern that the authors might be able to address]

## Detailed Comments
[Page/line-specific comments, typos, suggestions]

## Overall Assessment
Recommendation: [Strong Accept / Accept / Weak Accept / Borderline /
                  Weak Reject / Reject / Strong Reject]
Confidence: [High / Medium / Low]

Assessment Criteria by Dimension

Calibration Guide

Strong Accept: Significant contribution, technically sound, well-written.
  Would be a highlight of the venue.

Accept: Solid contribution with minor issues. Advances the field.
  Worth publishing as-is or with minor revisions.

Weak Accept: Has merit but notable weaknesses. Contribution is real but modest.
  Borderline for this venue; would be accepted at a less selective venue.

Borderline: Equal arguments for and against. Significant weaknesses offset
  by some novelty. Depends on other reviews.

Weak Reject: Interesting direction but fundamental issues not addressed.
  Major revisions needed that likely require a new submission cycle.

Reject: Significant problems in novelty, soundness, or relevance.
  Not suitable for this venue even with revisions.

Strong Reject: Fundamental flaws. Clearly below threshold.

Common Review Pitfalls to Avoid

Reviewing Different Paper Types

Empirical Papers

• Are datasets described completely? (Size, source, splits, preprocessing)
• Are baselines appropriate and fairly tuned?
• Statistical significance: error bars, multiple runs, significance tests
• Ablation studies: which components contribute to the gain?

Systems Papers

• Is the system actually built and evaluated (not just designed)?
• Performance at scale: latency, throughput, resource usage
• Comparison with existing systems (not just algorithms)
• Deployment experience or case studies

Survey/Position Papers

• Coverage: are major relevant works included?
• Taxonomy: is the organizational framework useful and complete?
• Analysis: does the survey add value beyond listing papers?
• Recency: are the most recent advances covered?

Ethics Considerations

During review, check:
□ Are human subjects involved? Was IRB/ethics approval obtained?
□ Are there potential harms from the technology described?
□ Is the data collection ethical? (Consent, privacy, bias)
□ Are dual-use concerns addressed? (Misuse potential)
□ Are limitations and societal implications discussed?

References

• Smith, R. (2006). "Peer review: a flawed process at the heart of science and journals." Journal of the Royal Society of Medicine, 99(4), 178-182.
• NeurIPS Reviewer Guidelines
• ACL Reviewer Tutorial
• Cormode, G. (2009). "How NOT to review a paper." ACM SIGMOD Record, 37(4), 100-104.