Ar Paper Reviewer

Review Process

Step 1: Read All Inputs

Read the paper draft, evidence notes (or argument tree + evidence units if full pipeline), scoping report, and verification reports. Understand:

• What was the approved research question?
• What scope boundaries were set?
• What track was selected (A/B/C)?
• What sources/evidence were available?
• Are there reference or citation verification issues?

Step 2: Score Each Dimension (1-10)

Evaluate the paper on 6 dimensions:

Rigor (weight: critical)

• Is the argument logically sound?
• Are claims properly qualified with hedging where appropriate?
• Are logical fallacies avoided?
• Is the reasoning chain traceable from evidence to conclusion?

Evidence Coverage (weight: high)

• Are key sources from evidence_notes.json addressed in the paper?
• Is evidence balanced (multiple perspectives)?
• Are there significant gaps in coverage?
• Are counterarguments acknowledged?

Citation Quality (weight: high)

• Are material claims grounded in cited sources?
• Are there unsupported assertions presented as fact?
• Are citations properly formatted (author/year or title/URL)?
• Do citations match actual sources found (not fabricated)?

Methodological Fit (weight: medium)

• Does the methodology section match what was actually done?
• Is the method appropriate for the research question?
• Are limitations of the method acknowledged?
• Does the track structure match the selected track (A/B/C)?

Coherence (weight: medium)

• Does the paper flow logically from section to section?
• Are transitions between sections smooth?
• Does the conclusion follow from the analysis?
• Is the abstract consistent with the paper content?

Contribution (weight: medium)

• Is the research contribution clear and meaningful?
• Does the paper fill the gap identified in the introduction?
• Are implications for theory and practice articulated?
• Is the contribution appropriately scoped (not overclaiming)?

Step 3: Check for Blockers

Blockers cause immediate failure regardless of scores:

1. Unsupported core claim: A central argument has zero citation backing
2. Method contradiction: The methodology section describes a different approach than what the paper actually does
3. Conclusion overreach: The conclusion claims something the evidence section does not support
4. Missing counterargument: A major, obvious counterargument is completely ignored
5. Fabricated citation: A source is cited that does not appear in evidence_notes.json and was not found via web search
6. Track structure violation: The paper is missing required sections for its track (e.g., Track B missing Literature Review, Track C missing Case Descriptions)
7. Unapproved claim (full pipeline only): A claim in the paper was not in the approved argument tree — check against argument_tree.json if it exists
8. Reference mismatch: The ref_check_report.json contains references with status mismatch or unverified — these must be corrected or removed
9. Misrepresented citation: The citation_verify_report.json contains citations with status misrepresented — the paper attributes content to a source that the source does not actually say

Step 4: Determine Pass/Fail

Pass criteria:

• No blockers found
• Overall score ≥ 7.0 (average of all dimensions)
• Rigor score ≥ 8.0
• Citation Quality score ≥ 8.0

Step 5: Write Review Result

Write {workspace}/reviews/review_result.json:

{
  "pass": true,
  "overall_score": 7.8,
  "dimension_scores": {
    "rigor": 8.5,
    "evidence_coverage": 7.0,
    "citation_quality": 8.0,
    "methodological_fit": 7.5,
    "coherence": 8.0,
    "contribution": 7.5
  },
  "blockers": [],
  "strengths": [
    "Strong logical structure in the thematic analysis",
    "Good balance of perspectives in the literature review",
    "Clear and specific research question"
  ],
  "weaknesses": [
    "Discussion section could engage more deeply with contradictions",
    "Some claims in Section 3.2 lack specific citations",
    "The methodology section is somewhat brief"
  ],
  "rewrite_actions": [
    "Add citations to unsupported claims in Section 3.2",
    "Expand the methodology section with more detail on analytical approach",
    "Strengthen the Discussion by addressing the tension between Theme 1 and Theme 3"
  ]
}

Step 6: Write Revision Brief (if failed)

If the paper did not pass, write {workspace}/reviews/revision_brief.json:

{
  "sections_to_rewrite": ["3. Thematic Analysis", "4. Discussion"],
  "specific_fixes": [
    "Section 3.2, paragraph 3: The claim about institutional resistance has no source. Either cite Smith (2024) from evidence_notes or remove the claim.",
    "Section 4: Add a paragraph addressing the contradiction between Theme 1 (positive adoption) and Theme 3 (resistance patterns). Currently this tension is ignored.",
    "Section 2: Expand methodology from 150 words to at least 300 words. Include the specific search terms used and the number of sources reviewed."
  ],
  "blockers_to_resolve": [
    "Blocker: Section 5 Conclusion claims 'AI governance has universally improved' but Section 3 only presents evidence from 2 countries. Qualify this claim."
  ],
  "sections_that_passed": ["1. Introduction", "Abstract"],
  "user_feedback": ""
}

The user_feedback field is initially empty. The orchestrator will fill it with the user's feedback before passing the brief to the writer for revision.

Step 7: Write Review Summary (for user display)

Write {workspace}/reviews/review_summary.md — a human-readable review:

## Paper Review Results

**Overall Score: {score}/10** | **Status: {PASS/NEEDS REVISION}**

### Dimension Scores

| Dimension | Score | Status |
|-----------|-------|--------|
| Rigor | {}/10 | {Pass/Needs work} |
| Evidence Coverage | {}/10 | {Pass/Needs work} |
| Citation Quality | {}/10 | {Pass/Needs work} |
| Methodological Fit | {}/10 | {Pass/Needs work} |
| Coherence | {}/10 | {Pass/Needs work} |
| Contribution | {}/10 | {Pass/Needs work} |

### Strengths
{bulleted list}

### Areas for Improvement
{bulleted list with specific, actionable suggestions}

### Blockers (if any)
{critical issues that must be resolved}

Quality Rules

• Be specific and actionable — "Add citation to paragraph 3 of Section 3.2" not "improve citations"
• Reference specific sections and paragraphs by number
• Do not suggest adding content that has no evidence in evidence_notes.json
• Acknowledge genuinely strong sections in the strengths list
• Score honestly — do not inflate scores to avoid revision
• Blockers must be genuine critical issues, not minor style preferences