Critique Paper

Axis 1 — Motivation strength

The paper exists because of some claimed gap, failure, or opportunity. Audit it:

• Is the gap real? Could a knowledgeable reader name 2–3 prior works that already addressed it? (If you don't know, flag for find-paper-context.)
• Is the gap important? Would solving it actually matter, or is it a corner of a corner?
• Is the framing honest? Some papers manufacture a strawman by misrepresenting prior work. Re-read the related-work section with this in mind.
• Why now? Is there something new (data, compute, technique) that makes this attempt newly tractable, and is the paper explicit about it?

Verdict: strong / moderate / weak / manufactured, with one paragraph explaining.

Axis 2 — Hypothesis quality

For methods/empirical papers:

• Is there a clearly stated hypothesis? Many papers don't actually state one; they propose a method and then show it works. That's fine for engineering papers but should be noted.
• Is the hypothesis falsifiable? What experimental result would have made the authors say "our hypothesis is wrong"? If no such result exists, the hypothesis isn't doing scientific work.
• Is the hypothesis specific or load-bearing? "Our method helps" is not a hypothesis; "our method helps because of mechanism X, predicting that it should fail when X is absent" is.

For theoretical papers:

• Are the assumptions stated? Are they realistic or pathological?
• What does the theorem buy you in practice, vs. what's left implicit?

Axis 3 — Experimental design / data

• Datasets: Are they appropriate? Standard? Cherry-picked? Are they evaluated on enough data points / seeds for the claimed effect size?
• Baselines: Are the comparisons fair (same compute budget, same data, same hyperparam search effort)? Are strong baselines included, or only easy ones? A paper that beats a 2018 baseline in 2024 is suspicious.
• Ablations: Does each design choice get its own ablation, or are choices bundled? Are the ablations designed to isolate the mechanism the paper claims is responsible for the gains?
• Metrics: Are they appropriate for the claim? Are there metrics where the method might not look good that are conspicuously missing?
• Statistical reporting: Error bars? Multiple seeds? Significance tests? Or single-run point estimates?

For each subpoint where you have a finding, cite the table/figure/section.

Axis 4 — Claims-↔-evidence audit

This is the centerpiece. Walk through 03_claims.md and for each claim, ask:

Does the evidence cited actually support a claim of this scope?

Categorize each claim:

• supported — evidence matches the claim.
• overclaimed — evidence supports a narrower claim than the paper makes. (Most common failure mode.) Rewrite the claim narrower.
• underclaimed — evidence supports a stronger claim than the paper makes. (Rare and good for the authors.)
• assumed — claim is made but no evidence is provided. May still be true; just note it.
• contradicted — evidence in the paper itself doesn't support the claim, or some other table/figure undercuts it.

For overclaimed/contradicted/assumed cases, write specifically what would have made the claim valid. This is the most useful output of the whole skill.

Axis 5 — Limitations & failure modes

• What does the paper acknowledge as a limitation? Take the acknowledgment at face value and check whether the experimental section actually probes it.
• What does the paper not acknowledge that you can spot? E.g.:
  • Setting where the method clearly wouldn't work
  • Distribution shifts not tested
  • Compute or memory regimes not explored
  • Failure modes implied by the method but not surfaced
• Are there obvious adversarial cases the paper sidesteps?

Axis 6 — Reproducibility & rigor (informed by NeurIPS reproducibility checklist)

• Is code released? Data? Trained checkpoints?
• Are hyperparameters fully specified?
• Is the compute budget reported (GPU type, hours)?
• Are random seeds reported?
• Could a competent grad student reproduce the headline number from the paper alone?

If the answer to most of these is no, the paper's empirical claims should be discounted accordingly.

Step 3 — Write 04_critique.md

# Critique: {Title}

**Tag:** `{tag}`
**Critiqued on:** {YYYY-MM-DD}

## TL;DR critique
2–4 sentences. The single most important strength and the single most important weakness. This is the line you'd lead a journal club with.

## Axis 1 — Motivation strength
**Verdict:** strong / moderate / weak / manufactured
...

## Axis 2 — Hypothesis quality
...

## Axis 3 — Experimental design / data
...

## Axis 4 — Claims-↔-evidence audit
| Claim | Status | Notes |
|---|---|---|
| C1 | supported | ... |
| C2 | overclaimed | Evidence in Table 3 only covers setting X; claim is about settings X *and* Y. |
| ... | ... | ... |

## Axis 5 — Limitations & failure modes
**Acknowledged:** ...
**Unacknowledged:** ...

## Axis 6 — Reproducibility & rigor
...

## What I'd want to see in a follow-up
Concrete experiment / analysis that would resolve the biggest open question above.

## Most interesting question for journal club discussion
The single thing this paper should provoke an argument about. Not the most damning criticism — the most *interesting* one.

Step 4 — Update 07_open_questions.md

Append any new open questions surfaced during the audit. Mark which ones could be answered by find-paper-reviews (community has probably already debated them) vs find-paper-context (resolved by reading a follow-up) vs "needs an experiment".

Output

Report to the user:

• Path to 04_critique.md
• The TL;DR critique line
• The single most interesting question for journal club discussion
• Whether you recommend running find-paper-reviews and find-paper-context next (almost always yes — the audit will have generated questions only the community can answer)