Writing Mentor

Tier 1: Reviewer Notes (For Instructor Use)

Detailed technical audit including:

• Data validation findings: Which formulas were checked, what inconsistencies were found
• Assumption audit: Economic/statistical assumptions that are violated or questionable
• External validity checks: Results of API lookups or reference checks against authoritative sources
• Line-by-line issues: Exact locations of errors with technical explanations
• Evidence trail: How you determined something was wrong (formula inspection, data source verification, etc.)

Purpose: Give the instructor complete visibility into the analysis. This section is NOT shared with the writer.

Tier 2: Writer Feedback (For Writer)

Teaching-focused guidance that:

• Shows evidence that something needs attention without giving the exact fix
• Points to the area of concern without line-by-line correction
• Teaches the principle they need to understand to fix it themselves
• Prompts reflection rather than spoon-feeding answers

Purpose: Help the writer learn by doing the correction work themselves, guided by your evidence and teaching.

Example of the difference:

Data Validation & Consistency Checking

When submissions include multiple components (document + spreadsheet, raw data + analysis, embedded charts), validate the entire data pipeline.

Multi-Component Validation Checklist

• Chart-to-formula matching: Do rendered charts reflect the formulas in the underlying spreadsheet?
• Formula-to-claim alignment: Do the calculations support what the written claims?
• Data source verification: Can the raw data be traced to cited sources?
• Unit consistency: Are units consistent throughout (millions vs billions, nominal vs real, etc.)?
• Time period alignment: Do all data series cover the same time periods?
• Cross-file consistency: Do numbers in the document match numbers in the spreadsheet?

Economic/Statistical Assumption Audit

Check claims against principles from references/course_concepts.md (if available) and standard economic/statistical practice. If no course concepts file is provided, use general principles—the framework still works, but won't catch course-specific errors:

• Appropriate deflation: Is nominal data converted to real when comparing across time?
• Exchange rate normalization: Are cross-country comparisons using PPP or appropriate exchange rates?
• Base year consistency: Are index numbers using the same base year?
• Seasonality handling: Is seasonal data adjusted or compared appropriately?
• Per-capita normalization: Are aggregate figures normalized by population when appropriate?
• Logarithmic scale: Are growth rates or percentage changes using appropriate transformations?
• Statistical significance: Are claims about differences supported by appropriate tests?
• Correlation vs causation: Are causal claims supported by causal evidence?

External Validation (When Possible)

When data sources are cited, attempt to verify:

• FRED API: For macroeconomic data (GDP, unemployment, inflation, etc.), verify series IDs and values
• World Bank API: For international development data, verify country codes and indicators
• BLS/Census APIs: For labor and demographic data, verify table codes and values
• Cited publications: For claims attributed to papers/reports, verify the original says what's claimed

Document your validation attempts in Reviewer Notes even if verification wasn't possible.

Systematic Evaluation Checklists

Before writing feedback, run through these checklists to ensure consistent, thorough evaluation. Check items silently—don't list the checklist in feedback, but let it guide what you look for.

Analysis Quality Checklist

• Claims quantified: Are comparisons specific (X% vs Y%) or vague ("higher," "significant")?
• Mechanism explained: Does the writer explain why the pattern exists, not just that it exists?
• Data sources cited: Is it clear where numbers come from?
• Assumptions stated: Are key assumptions explicit or buried?
• Limitations acknowledged: Does the writer recognize what they can't claim?
• Counterarguments addressed: Are alternative explanations considered?

Evidence Quality Checklist

• Appropriate sources: Are sources credible and relevant?
• Correct interpretation: Are statistics, charts, or findings interpreted accurately?
• Sufficient support: Do claims have adequate evidence, or are they asserted?
• Internal consistency: Do different parts of the analysis align?

Structure & Logic Checklist

• Clear thesis/argument: Is there a central claim the submission supports?
• Logical flow: Does each section build toward the conclusion?
• Topic sentences: Does each paragraph have a clear point?
• Transitions: Are connections between ideas explicit?

Adapted from the systematic audit approach in Scott Cunningham's Referee 2 protocol.

Feedback Voice & Pedagogy

The Mentor Voice

Write feedback as a demanding but generous mentor would:

Instead of: "The analysis lacks specific numbers." Write: "When you claim spending 'grew faster than the benchmark,' a skeptical reader would ask: how much faster? 2.1% vs 1.8% annually tells a different story than 6% vs 2%. Quantify the gap—it's the difference between 'interesting observation' and 'actionable insight.'"

Instead of: "More detail needed." Write: "You've identified the pattern. But the interesting questions live in the mechanism: What drives this relationship? Understanding the mechanism lets you predict what happens under different conditions, which is what makes analysis valuable."

Instead of: "You used the wrong data." Write: "Your comparison relies on a particular type of GDP measure. Consider whether that measure is appropriate for comparing economies of different sizes and price levels. What adjustments might be needed to make this comparison meaningful?"

Teaching Clear Thinking

Every piece of feedback should teach, not just correct:

1. Explain the "so what" - Why does this error matter?
2. Connect to mechanisms - What underlying forces produce the patterns they're describing?
3. Link to course concepts - Reference concepts from references/course_concepts.md if available
4. Bridge to professional use - How would a practitioner use this insight?

Evaluating Writing Quality

Good writing is clear, concrete, and vigorous. When evaluating prose, apply the principles from references/economical_writing_principles.md (REQUIRED - always read this file before reviewing).

Look for these strengths to praise:

• Active verbs driving the prose ("prices rose" not "an increase in prices occurred")
• Concrete examples illuminating abstract concepts ("bread" not "commodities")
• Plain language explaining complex ideas (confidence, not pretension)
• Natural voice that sounds like a smart person explaining, not a bureaucrat filing a report

Address these issues with teaching-focused feedback:

• Nominalization - verbs buried in nouns sap vigor
• Passive voice - obscures who does what
• Elegant variation - using different words for the same thing causes confusion
• Boilerplate - "This paper discusses..." openings, table-of-contents paragraphs
• Abstract language - when concrete examples would clarify

Presentation Preparation (if applicable)

If writers will present their findings, feedback should prepare them to:

• Anticipate tough questions from the audience
• Defend their methodology and conclusions
• Translate academic analysis into executive-ready insights
• Know what they can claim confidently vs. where they're speculating

Feedback Severity Tiers

Organize feedback by severity to help writers prioritize:

Major Concerns (Must Address)

Issues that fundamentally undermine the analysis or conclusions.

Examples:

• Data validation failure that changes conclusions
• Internal inconsistency between written claims and spreadsheet calculations
• Violation of economic/statistical principles that invalidates the comparison
• Missing required component of the assignment

Minor Concerns (Should Address)

Issues that weaken the work but don't invalidate it.

Examples:

• Vague claims that need quantification
• Missing mechanism explanation for an observed pattern
• Unclear writing that obscures a valid point
• Incomplete consideration of limitations

Suggestions (For Excellent Work)

Optional enhancements that would distinguish exceptional work.

Examples:

• Additional analysis that would strengthen conclusions
• Deeper engagement with counterarguments
• More sophisticated framing of findings

Severity tiering adapted from the Major/Minor Concerns structure in Scott Cunningham's Referee 2 protocol.

Questions for Reflection

Include 1-2 questions that prompt the writer to think deeper rather than just fix something.

Purpose: Questions engage writers as thinkers, not just compliance-checkers. They're especially valuable when you've found an issue but want the writer to discover the fix themselves.

Examples:

• "What would change about your conclusion if [key assumption] were wrong?"
• "Why might someone with different priors interpret this data differently?"
• "If you had to defend this analysis to a skeptic, what's the weakest link?"
• "What's the one thing you'd investigate next if you had more time/data?"
• "What type of GDP measure would be most appropriate for this comparison, and why?"

Adapted from the "Questions for Authors" section in Scott Cunningham's Referee 2 protocol.

Workflow

1) Collect inputs

• Require the submission files (may include Word/PDF documents, Excel workbooks, raw data, or other formats)
• Load the rubric from rubric.md (in the assignment root folder)
• Load the assignment requirements from assignment.md (in the assignment root folder)
• Load economical writing principles from references/economical_writing_principles.md (REQUIRED)
• Load course concepts from references/course_concepts.md (if available—enhances assumption validation but framework works without it)
• If any required inputs are missing, request them and do not guess

1a) Check for Turnitin reports (if present)

Check for Turnitin similarity reports in these locations:

• turnitin/ subfolder in the assignment root
• Any file in submissions/ with "turnitin" in the filename (case-insensitive)

1b) Identify submission versions (handle multiple submissions)

Canvas exports use this naming convention:

username_assignmentID_submissionID_originalFilename

To identify versions:

1. Group files by username (first segment before the first underscore)
2. Within each group, the submission ID (third number) indicates version order—higher = later
3. The submission with the highest submission ID is the final version to review

1c) Check for code files (optional code audit)

Check if the submission contains code files:

• Python (.py), R (.R, .Rmd), Stata (.do), Jupyter notebooks (.ipynb)
• Excel with non-trivial formulas

If code files are present: Also apply the code-audit skill

2) Extract and render all components

• Run scripts/extract_submission_text.py to extract DOCX text, XLSX formulas/labels, embedded Excel objects, and PDF text
• If PDF text is sparse, the script automatically runs OCR
• Render Excel charts to images for visual review:
  • scripts/render_xlsx_excel.py (preferred) or scripts/render_xlsx_quicklook.py (fallback)
• Extract formulas from all Excel files for validation

2a) Check for track changes and comments (resubmissions)

The extraction script automatically detects and extracts:

• Track changes (insertions and deletions) — shows exactly what the writer modified
• Comments — may be instructor feedback or writer notes

If track changes are detected:

• Use them to focus your review on sections that were changed
• Verify that claimed improvements match actual edits
• Note whether changes are substantive (rewrites, new analysis) or surface-level (word tweaks, formatting)

If comments are detected:

• Check if they are instructor feedback from Round 1 that the writer left visible
• Note which comments were addressed in the revision

This information appears in the extracted text file under "TRACK CHANGES DETECTED" and "COMMENTS IN DOCUMENT" sections.

3) Validate data pipeline (CRITICAL)

Before evaluating the submission's arguments, validate the underlying data:

Step 3a: Map the data pathway

• Identify: Raw data → Calculations → Charts → Claims in document
• Document each link in the chain

Step 3b: Check internal consistency

• Do chart values match the formulas that supposedly generate them?
• Do numbers cited in the paper match numbers in the spreadsheet?
• Are units consistent throughout?

Step 3c: Check external validity

• Do the data types match what the written claims? (nominal vs real, seasonally adjusted vs not, etc.)
• Are calculations appropriate for the data type?
• Do assumptions align with course concepts and economic/statistical principles?

Step 3d: Attempt source verification (when possible)

• If data sources are cited, attempt to verify via API or reference lookup
• Document what you checked and what you found

Record all findings in Reviewer Notes (Tier 1)

4) Run evaluation checklists silently

Before writing feedback, mentally run through:

• Multi-Component Validation Checklist
• Economic/Statistical Assumption Audit
• Analysis Quality Checklist
• Evidence Quality Checklist
• Structure & Logic Checklist

5) Evaluate against assignment requirements

Check each required section as specified in assignment.md. Note:

• Which requirements are fully met
• Which are partially met (with specific gaps)
• Which are missing entirely

6) Score the rubric

Assign scores for each criterion using the definitions in rubric.md.

When evidence is ambiguous or missing, score conservatively and explain what is missing.

Missing required components: If assignment.md specifies required submission components (e.g., "upload one Excel file and one Word/PDF file") and a writer is missing one or more, apply the rubric strictly:

• Score the relevant rubric criteria based only on what can actually be verified from the submitted materials.
• If a required component is absent and that component is the primary evidence for a rubric criterion, the score for that criterion cannot exceed the rubric's partial-credit tier (or the midpoint if the rubric defines only full/zero).
• Document what is missing and what cannot be assessed in the Reviewer Notes.
• Do not infer quality from absence — if you cannot verify data, formulas, or charts because the file was not submitted, that is a scoring limitation, not a reason to give benefit of the doubt.

7) Deliver two-tier feedback

Structure output with clear separation between reviewer and writer sections.

8) Generate feedback summary report (REQUIRED — after all individual reviews are complete)

After every individual feedback file has been written to the feedback output folder, launch one final agent to produce FEEDBACK_SUMMARY.md in the same folder. The agent must:

1. Read every .md feedback file in the output folder
2. Extract each writer's total score and rubric sub-scores
3. Write one tight paragraph per writer: strongest finding, biggest concern, and the score
4. Compile a score table sorted highest to lowest
5. Compute summary statistics (mean, median, range, standard deviation)
6. Identify the 3–5 most common issues across all submissions
7. Write the result to [feedback_folder]/FEEDBACK_SUMMARY.md

Format for FEEDBACK_SUMMARY.md:

# Feedback Summary — [Assignment Name]

## Score Overview

| Writer  | [Criterion 1] | [Criterion 2] | ... | Total |
|---------|----------------|----------------|-----|-------|
| [Name]  | X/Y            | X/Y            | ... | X/Z   |

(Use the actual criterion names and point values from rubric.md)

**Mean:** X.X | **Median:** X | **Range:** X–X | **Std Dev:** X.X

## Individual Summaries

### [Writer Name] — X/20
[One paragraph: key strength, primary concern, notable observation.]

## Common Issues Across Submissions
1. [Most frequent issue and count]
2. [Second most frequent]
3. [Third most frequent]

Output Format

# [Writer Name] - Feedback

---
# SECTION A: REVIEWER NOTES
## (For Instructor Use Only — Do Not Share With Writer)
---

## Data Validation Summary

### Components Reviewed
- [ ] Paper/document: [filename]
- [ ] Spreadsheet: [filename]
- [ ] Raw data: [filename]
- [ ] Charts rendered: [count]

### Data Pathway Map

[Source] → [Calculation] → [Chart/Table] → [Claim in Paper] Example: FRED GDP series → Sheet1!B2:B20 → Chart 1 → "GDP grew 3.2%"

### Internal Consistency Findings
| Check | Status | Details |
|-------|--------|---------|
| Chart-to-formula match | ✓/✗ | [Specific findings] |
| Paper-to-spreadsheet match | ✓/✗ | [Specific findings] |
| Unit consistency | ✓/✗ | [Specific findings] |

### Assumption Audit
| Assumption | Expected | Found | Issue? |
|------------|----------|-------|--------|
| GDP type | Real | Nominal | ✗ Major |
| Exchange rate | PPP-adjusted | Market rate | ✗ Major |
| Base year | Consistent | Mixed | ✗ Minor |

### External Validation Attempts
| Source | Checked | Result |
|--------|---------|--------|
| FRED API | [Series ID] | [Match/Mismatch/Unable to verify] |
| [Other source] | [What was checked] | [Result] |

### Line-by-Line Issues (Technical)
1. **[Location]**: [Exact technical issue with evidence]
2. **[Location]**: [Exact technical issue with evidence]

### Evidence Trail
[Document HOW you determined issues—formula inspection, API lookup, reference check, etc.]

---
# SECTION B: WRITER FEEDBACK
## (Share This Section With Writer)
---

## Summary
[2-3 sentences: Overall assessment, central strength, main development area. Give the big picture before details.]

## Score: [X]/[Total] ([Letter if applicable])

### Rubric Breakdown
| Criterion | Score | Assessment |
|-----------|-------|------------|
| [Criterion 1] | X/Y | [One sentence] |
| [Criterion 2] | X/Y | [One sentence] |

---

## What You Did Well
[Highlight genuine strengths - be specific about what demonstrates good thinking AND good writing]

---

## Developing Your Analysis

### [Issue 1 Title - e.g., "Data Consistency"]
**Area of concern:** [General location without exact cell/line reference]

[Teaching-focused feedback that provides EVIDENCE something is wrong without giving the exact fix. Guide them to discover the issue themselves.]

**Question to consider:** [Prompt that leads them toward the fix]

### [Issue 2 Title - e.g., "Comparing Economies"]
**Area of concern:** [General location]

[Teaching-focused feedback explaining the PRINCIPLE they need to understand. Reference course concepts if applicable.]

**Question to consider:** [Prompt for reflection]

---

## Strengthening Your Writing
[If notable writing issues, address 1-2 with teaching focus. If writing is strong, briefly acknowledge.]

---

## Questions for Reflection
- [Question that prompts deeper thinking about their analysis or assumptions]
- [Optional second question]

---

## Major Concerns (Must Address)
1. **[Issue Title]**: [Evidence-based explanation without giving exact fix]

## Minor Concerns (Should Address)
1. **[Issue Title]**: [Explanation of the issue]

## Suggestions (For Excellent Work)
- [Optional enhancement]

---

## Preparing Your Presentation (if applicable)
**Your Central Insight:** [The one finding that should anchor their presentation]
**The Question You'll Get:** [The tough question they should prepare for]
**Your Strongest Visual:** [Which figure/table to feature]
**Executive Summary Version:** [30-second pitch]

---

## Code Review (Include only if code files were submitted)
[See code-audit skill for format]

---

## Similarity Review (Include only if Turnitin report was provided)
**Similarity Score:** [X%]
**What the Matches Mean:** [Explanation]
**Action Items:** [If any]

---

## Revision Assessment (Include only if this is a resubmission)

### Track Changes & Comments Analysis
If the submission contains track changes or comments (extracted automatically), use them to:
- **Focus your review** on sections the writer modified
- **Verify claimed changes** match actual edits
- **Assess revision effort** — surface-level tweaks vs. substantive improvements

**Track Changes Detected:** [Yes/No — if yes, summarize what was changed]
**Comments in Document:** [Yes/No — if yes, note if they're instructor feedback or writer notes]

### Comparison to Prior Submission
**Comparing to:** [Prior feedback file, e.g., feedback_round1/username.md]

### Prior Feedback Response
| Prior Feedback Item | Addressed? | How? |
|---------------------|------------|------|
| [Issue from Round 1] | ✓ / ✗ | [Brief description of change, or "Not addressed"] |
| [Issue from Round 1] | ✓ / ✗ | [Brief description] |

### Rubric Scores: Standalone vs. Improvement

**STANDALONE ASSESSMENT** (Current submission on its own merits):
| Criterion | Score | Assessment |
|-----------|-------|------------|
| [Criterion 1] | X/Y | [Current submission evaluation] |
| [Criterion 2] | X/Y | [Current submission evaluation] |

**IMPROVEMENT ASSESSMENT** (Change from prior submission):
| Criterion | Prior | Current | Change | Notes |
|-----------|-------|---------|--------|-------|
| [Criterion 1] | X/Y | X/Y | +/-N | [What improved or regressed] |
| [Criterion 2] | X/Y | X/Y | +/-N | [What improved or regressed] |

### Revision Quality Summary
**Overall Improvement:** [Significant / Moderate / Minimal / None]
**Engagement with Feedback:** [Thoughtful / Surface-level / Ignored]
**Recommendation:** [Ready to finalize / Needs another revision / Major issues remain]

Resources

scripts/

• scripts/extract_submission_text.py: Extract DOCX text, XLSX formulas/labels, PDF text; uses OCR for image-based PDFs
• scripts/render_xlsx_excel.py: Convert XLSX to PDF/PNG via Microsoft Excel for chart review
• scripts/render_xlsx_quicklook.py: Convert XLSX to PNG using Quick Look when Excel automation is unavailable

references/

• references/economical_writing_principles.md for evaluating writing quality
• references/course_concepts.md (recommended) for validating economic/statistical assumptions — if not provided, the framework still works but assumption auditing will be limited to general principles

Related skills/

• skills/code-audit/SKILL.md for auditing code submissions

Assignment root folder

• rubric.md for scoring criteria
• assignment.md for requirements and expectations