Lingliang Grading

The full grading workflow follows the unified skill adapted for this subject. The complete content is reproduced below for reference.

Lingliang Primary School Math Exam — Unified Grading Skill

Overview

This skill grades Lingliang primary school math exam papers by extracting rubrics from per-question PDF/JPG pairs, calibrating against reference data with known scores, then grading each student's scanned handwritten answer PDF on a per-question basis. It produces per-student JSON results and compiles them into final raw scores.

Subject: Lingliang (小學數學考試) — primary school math, 37 questions, 100 total marks, 25 students (5 reference + 20 grading targets).

Key differences from HKDSE subject grading:

• No DSE levels 1–5 — output is raw scores only
• No year-based directory structure (no reference year / grading year)
• Per-question rubric files (37 individual PDF+JPG pairs) instead of a single rubrics.pdf
• A question_manifest.csv drives rubric iteration
• An answer_page_map.json maps questions to pages in each student's 6-page PDF
• Student PDFs are scanned handwritten math work (not typed/printed)

CRITICAL RULES

1. Sub-agents are mandatory. Each student MUST be graded by a dedicated sub-agent. No batch grading of multiple students in a single sub-agent call.

2. Model restrictions:

   • Sub-agents: SAME model as the main agent. All sub-agents MUST use the same model that the main (orchestrating) agent is running on. Do NOT pass a model parameter override to sub-agents — let them inherit the main agent's model.
   • VLM tasks: Gemini ONLY. When VLM (vision-language model) API calls are needed — e.g., text extraction from scanned PDFs where PyMuPDF fails, or interpreting handwritten mathematical notation — use gemini-3.1-flash-lite-preview via the Gemini route exclusively. NO OTHER VLM MODELS OR ROUTES ARE ALLOWED.

3. Gemini only (for both added chains):

   • extraction skill and model-service skill must both use Gemini for API calls.
   • Read GEMINI_BASE_URL, VERTEX_API_KEY from env.txt.
   • Model must be gemini-3.1-flash-lite-preview.
   • Do not route these two chains to Aliyun/OpenRouter/native Moonshot/Kimi/Seed.
   • Chain to model-service skill: run a config-based route sanity check that confirms route=gemini is callable and non-gemini routes are ignored.

4. PDF rendering DPI: When converting PDF pages to images (e.g., for VLM extraction), default to 150 DPI to ensure text is legible. If processing speed is a concern, lowering DPI is acceptable as a trade-off, but do not go below 100 DPI.

5. Image preservation is mandatory when flagged by extraction. When a page's meaning depends on layout or visuals — e.g., geometric figures, handwritten mathematical notation with spatial arrangement (fractions, long division layouts), graphs, tables with spatial meaning, arrows/labels, or annotated drawings — the extraction API output must explicitly signal this (via needs_visual_reference + visual_reference_reason). The agent MUST inspect those returned hints and export the whole page image as a PNG artifact for every flagged page. This applies to both rubric/question pages AND student answer pages. Do not crop at this stage.

6. No LLM/VLM API calls for generating feedback or analysis text. See . VLM is permitted only for tasks requiring genuine visual inspection — text extraction from scanned PDFs where PyMuPDF fails, and interpreting handwritten mathematical notation (fractions, division symbols, etc.). VLM must NOT be used for any dimension assessable from extracted text.

• For PDF/image text extraction tasks in this grading workflow, invoke the extraction skill.
• For any direct LLM API calling/routing validation task, invoke the model-service skill (YAML-only, no .py edits).
• Both skills must use Gemini (gemini-3.1-flash-lite-preview) and read URL/API key values from env.txt; missing values mean that provider/API is unsupported in the current environment.

Skill Chain Integration

Use this grading skill as the orchestrator and explicitly chain the migrated skills:

1. Extraction path (extraction skill):
   • Phase 2 rubric/question extraction (per-question rubric PDF+JPG pairs)
   • Phase 3 reference-student answer extraction
   • Phase 4 masked-student answer extraction (one page at a time for student answers)
2. Model API path (model-service skill):
   • Provider route checks and API availability checks for Gemini (gemini-3.1-flash-lite-preview)
   • Controlled direct API calls for extraction-related troubleshooting
3. Boundary:
   • extraction and model-service are config-driven execution skills; do not modify their underlying Python scripts inside this grading workflow.

Workspace Layout

lingliang-grading-workspaces/grading_workspace/
├── .github/skills/lingliang-grading/SKILL.md
├── START.md
├── WARNING.md
├── env.txt                     # Local credentials (never commit)
├── env.txt.example             # Template
├── pyproject.toml
├── uv.lock
├── start.sh
├── data/                       # Local data (symlinked into workspace)
│   ├── reference/lingliang/
│   │   ├── student_answers/student_07.pdf, student_08.pdf, student_11.pdf, student_17.pdf, student_23.pdf
│   │   └── reference_mapping.json
│   ├── masked_data/lingliang/
│   │   ├── student_answers/student_01.pdf … student_26.pdf (excludes 07, 08, 11, 17, 23)
│   │   ├── rubrics/rubrics_q-{N}.pdf + rubrics_q-{N}.jpg (37 per-question pairs)
│   │   ├── question/q-{N}.txt + q-{N}.jpg (37 per-question materials)
│   │   └── question_manifest.csv
│   └── z_test_data/lingliang_prework/
│       └── answer_page_map.json
├── rubric/                     # Grading artifacts (no year nesting)
│   ├── grading_guide.md
│   ├── reference_calibration.md
│   ├── page_images/            # Full-page PNG artifacts for rubric/question pages (MANDATORY when visual content detected)
│   │   ├── rubrics/page_{P}.png   # Rubric pages with diagrams, reference answer figures
│   │   └── question/page_{P}.png  # Question pages with diagrams, graphs, geometric figures
│   ├── reference_scores.json
│   └── calibration/            # Intermediate calibration artifacts
│       └── (per-student reference grading outputs, draft rubrics, score calculations)
├── rubric/reference_data_analysis/  # Insights from reference data (Phase 3)
│   ├── score_correlation_analysis.md  # Observed patterns across score ranges
│   ├── rubric_gaps.md               # Gaps/ambiguities in rubric
│   └── rubric_refinements.md        # Supplementary rubric guidance
├── scripts/                    # Python helper scripts
│   ├── generate_class_report.py
│   ├── generate_student_reports.py
│   ├── validate_extraction.py
│   ├── validate_grading_output.py
│   └── validate_reports.py
├── extracted/                  # Extracted student text (no year nesting)
│   └── students/
│       ├── student_{NN}.txt
│       └── page_images/student_{NN}/page_{P}.png   # Visual-page artifacts (MANDATORY when [IMAGE_DATA] present)
└── output/                     # Grading output (no year nesting)
    ├── student_{NN}.json
    └── final_scores.json

Phase 1: Environment Setup

Step 1.1: Install Dependencies

source env.txt
uv sync

No GRADING_YEAR env var is needed — paths are fixed with no year-based structure.

Step 1.2: Verify Data Availability

Confirm the following exist:

• Student answer PDFs: data/masked_data/lingliang/student_answers/student_01.pdf … student_26.pdf (20 grading students, excluding 07, 08, 11, 17, 23)
• Per-question rubrics: data/masked_data/lingliang/rubrics/rubrics_q-1.pdf + rubrics_q-1.jpg … through all 37 questions
• Question materials: data/masked_data/lingliang/question/q-1.txt + q-1.jpg … through all 37 questions
• Question manifest: data/masked_data/lingliang/question_manifest.csv
• Answer page mapping: data/z_test_data/lingliang_prework/answer_page_map.json
• Reference data: data/reference/lingliang/reference_mapping.json
• Reference student PDFs: data/reference/lingliang/student_answers/student_07.pdf, student_08.pdf, student_11.pdf, student_17.pdf, student_23.pdf

Read BATCH_SIZE from env.txt (default: 5) for parallel sub-agent control.

Phase 2: Rubric Extraction

Step 2.1: Read Question Manifest

Read data/masked_data/lingliang/question_manifest.csv to get the full list of all 37 questions. The CSV contains columns:

• question — question identifier (e.g., "1", "3.a", "9.decimal", "15.b")
• question_page — page number in the student answer PDF
• rubric_page — page in the rubric PDF
• question_text — the question text
• question_text_path — path to extracted question text file (e.g., question/q-1.txt)
• question_image_path — path to question image (e.g., question/q-1.jpg)
• rubric_pdf_path — path to rubric PDF (e.g., rubrics/rubrics_q-1.pdf)
• rubric_image_path — path to rubric image (e.g., rubrics/rubrics_q-1.jpg)

⛔ Do NOT look for or read any gold_csv_path or *-gold.csv files during Phase 2. Those files contain ALL students' ground truth scores and must not be used for rubric extraction. Phase 2 uses ONLY rubric_pdf_path, rubric_image_path, question_text_path, and question_image_path.

Step 2.2: Extract Per-Question Rubric Content

For EACH of the 37 question rows in the manifest, extract the rubric by:

1. Read the rubric PDF from rubric_pdf_path (e.g., rubrics/rubrics_q-1.pdf) — this is the primary source for marking criteria and acceptable answers
2. Read the rubric image from rubric_image_path (e.g., rubrics/rubrics_q-1.jpg) — use this as a visual fallback or cross-reference if the PDF extraction is incomplete, or if the rubric contains handwritten annotations, diagrams, or mathematical notation that PyMuPDF cannot render
3. Read the question text from question_text_path (e.g., question/q-1.txt) — this provides the extracted question text for context
4. Read the question image from question_image_path (e.g., question/q-1.jpg) — use this to understand the visual layout of the question, especially for questions involving diagrams, graphs, tables, or geometric figures

For each question, extract and record:

• The question number and any sub-question identifiers
• The maximum marks available
• The specific marking criteria (what earns each mark)
• Acceptable answer forms (e.g., fractions, decimals, percentages)
• Whether method marks are awarded separately from answer marks
• Any special instructions (e.g., "show working", "units required", "accept equivalent")
• Diagrams or figures that are part of the question or rubric

Use PyMuPDF (fitz) for text extraction from rubric PDFs. Fall back to VLM only if text extraction yields empty or garbled content (common for scanned rubrics with handwritten annotations or mathematical notation).

Chain to extraction skill: perform this phase through the extraction skill's YAML-driven pipeline. Since there are 37 individual rubric files (not one large PDF), configure the extraction to process each rubric file independently. Force Gemini routing (model_routes -> gemini) with env-backed provider config only (GEMINI_BASE_URL, VERTEX_API_KEY).

Note on rubric extraction: Each rubric file is a single page covering one question. The JPG version is a rendered image of the same content — use it when PDF text extraction fails or when mathematical notation needs visual interpretation.

Processing strategy for 37 rubrics: Process rubric files in batches (e.g., 5–10 at a time) to balance parallelism with resource constraints. For each batch:

1. Extract text from all rubric PDFs in the batch
2. Cross-reference with the corresponding JPG images for any unclear content
3. Verify that mark allocations are captured correctly
4. Move to the next batch Track extraction progress and flag any questions where the rubric could not be fully extracted for manual review.

Step 2.3: Build Grading Guide

Create rubric/grading_guide.md with:

• All 37 questions listed with their question numbers (from manifest)
• Mark allocation per question and sub-question
• Every individual marking point (採分點) for each question — list each criterion that earns marks as its own entry. Do not collapse multiple marking points into a single bullet; every scorable criterion must appear explicitly
• Partial mark breakdown: For questions with staged or banded marking, document every stage explicitly — what the student must demonstrate to earn 0, 1, 2, … up to the maximum marks at each step
• Marking criteria: what earns each mark — include acceptable answers and common unacceptable answers where the rubric provides them
• For math questions: specify whether method marks and answer marks are separate, whether units are required, acceptable equivalent forms (e.g., fractions vs. decimals)
• Any special instructions (e.g., alternative acceptable answers, "accept any reasonable method", partial credit for correct working with wrong final answer)
• The question text for each question (from question_text_path) for context
• Total marks verification: confirm all per-question marks sum to 100

Grading guide structure: Organize the guide by question number in ascending order. For each question, use the following template:

### Question {N} ({max_marks} marks)

**Question text:** {from question_text_path}

**Marking criteria:**
- Criterion 1: {description} — {marks} mark(s)
- Criterion 2: {description} — {marks} mark(s)
- ...

**Acceptable answers:** {list of acceptable forms}
**Common errors:** {known incorrect approaches that should NOT earn marks}
**Special notes:** {any rubric-specific instructions}

For sub-questions (e.g., 3.a, 3.b, 9.decimal, 9.fraction), nest them within their parent question section with clear sub-headings.

Phase 3: Reference Calibration

Step 3.1: Read Reference Data

Read data/reference/lingliang/reference_mapping.json to get the mapping of student IDs to known scores.

Format:

{
  "mappings": [
    {"student_id": 7, "filename": "student_07.pdf", "total_score": 100, "max_score": 100, "score_percentage": 100.0},
    {"student_id": 8, "filename": "student_08.pdf", "total_score": 74, "max_score": 100, "score_percentage": 74.0},
    {"student_id": 11, "filename": "student_11.pdf", "total_score": 56, "max_score": 100, "score_percentage": 56.0},
    {"student_id": 17, "filename": "student_17.pdf", "total_score": 87, "max_score": 100, "score_percentage": 87.0},
    {"student_id": 23, "filename": "student_23.pdf", "total_score": 30, "max_score": 100, "score_percentage": 30.0}
  ]
}

Note: Reference files use student_{NN}.pdf naming (same format as grading students). The known ground-truth scores are: student_07 (100/100), student_08 (74/100), student_11 (56/100), student_17 (87/100), student_23 (30/100). These span a wide range from 30% to 100%, providing good calibration anchors.

Step 3.2: Extract Reference Student Answers

Extract text from ALL reference student answer PDFs in data/reference/lingliang/student_answers/.

Chain to extraction skill: use extraction pipeline in student-PDF mode (pdf_dir + optional students) to produce structured extraction outputs that feed reference calibration. Force Gemini routing for this chain.

Note: These are scanned handwritten math answers (6 pages each). VLM extraction is likely necessary for mathematical notation (fractions, division, etc.).

Step 3.3: Read Reference Rubrics

The rubrics for reference grading are the SAME as for grading — use the per-question rubric files already extracted in Phase 2 (from data/masked_data/lingliang/rubrics/). There is no separate reference-year rubric directory.

Step 3.4: Grade Reference Students (Iterative Rubric Refinement)

Grade ALL 5 reference students using the same rubric and grading guide (from Phase 2). For each reference student, apply the per-question mark allocation to produce a total raw score and percentage — exactly as you would for a masked student. Record these scores alongside their known total scores from reference_mapping.json.

This step is critical: the resulting scores provide empirical calibration — verifying that the rubric as interpreted produces scores consistent with known ground-truth scores. Launch sub-agents for reference students following the same rules as Phase 5 (one sub-agent per student, same model restriction).

Use answer_page_map.json to determine which page(s) of each student's 6-page PDF contain the answer for each question. This is essential for accurate per-question grading of scanned handwritten answers.

Save reference grading results to rubric/reference_scores.json:

{
  "scores": [
    {"student_id": 7, "total_score": 100, "ai_total_raw_score": 98, "ai_total_max_score": 100, "ai_percentage": 98.0, "score_percentage": 100.0},
    {"student_id": 8, "total_score": 74, "ai_total_raw_score": 71, "ai_total_max_score": 100, "ai_percentage": 71.0, "score_percentage": 74.0},
    {"student_id": 11, "total_score": 56, "ai_total_raw_score": 54, "ai_total_max_score": 100, "ai_percentage": 54.0, "score_percentage": 56.0},
    {"student_id": 17, "total_score": 87, "ai_total_raw_score": 85, "ai_total_max_score": 100, "ai_percentage": 85.0, "score_percentage": 87.0},
    {"student_id": 23, "total_score": 30, "ai_total_raw_score": 32, "ai_total_max_score": 100, "ai_percentage": 32.0, "score_percentage": 30.0}
  ]
}

Step 3.4a: Reference Data Analysis and Iterative Rubric Refinement

The rubric must not be finalised from the official PDF alone. The agent must learn from the reference student data to derive an effective grading rubric — one that actually produces scores consistent with known ground-truth scores. This is an iterative loop that continues until the rubric reliably reproduces expected scores.

This is especially important for primary school math where handwritten answers may use non-standard notation, alternative solution methods, or ambiguous layouts. The rubric must account for these realities — reference data analysis reveals what students actually write and how it maps to marks.

All analytical outputs from this step are saved to rubric/reference_data_analysis/.

Iterative loop:

1. Grade all reference students with the current rubric draft (Step 3.4 above).
2. Analyse grading results versus known scores:
   • Do AI-graded scores correlate with ground-truth scores? (Higher ground-truth → higher AI score?)
   • Which rubric criteria caused over-scoring or under-scoring?
   • Are the per-question mark allocations consistent with the rubric's intent?
   • What patterns distinguish high-scoring students from low-scoring ones?
   • For math-specific issues: Are method marks being awarded correctly when the final answer is wrong? Are equivalent representations (e.g., 1/2 vs 0.5) being accepted? Are units and labels being checked appropriately?
3. Write rubric/reference_data_analysis/score_correlation_analysis.md documenting the correlation between AI scores and known scores:
   • For each reference student: known score, AI score, difference
   • Per-question analysis where significant discrepancies exist
   • Patterns in over/under-scoring across the score range
   • Specific questions where the rubric interpretation needs adjustment
   • A summary table showing each reference student's known vs. AI score
4. Run the discrimination check (see criteria below). If the check fails, or if qualitative analysis reveals the rubric is inadequate: a. Document gaps in rubric/reference_data_analysis/rubric_gaps.md — gaps or ambiguities found in the rubric when applied to reference students (e.g., criteria that fail to distinguish partial credit correctly, missing guidance for common student errors, mark schemes that reward or penalise inconsistently, unclear treatment of alternative solution methods) b. Derive refinements in rubric/reference_data_analysis/rubric_refinements.md — supplementary rubric interpretations and practical guidance that resolve the gaps above (e.g., clarifying what constitutes a "correct method" for partial credit, specifying how to handle equivalent mathematical expressions, defining acceptable rounding behavior, handling crossed-out work vs. final answers) c. Revise rubric/grading_guide.md to incorporate these refinements — the grading guide is not simply extracted from the official rubric PDFs; it must incorporate insights from the reference data analysis d. Re-grade affected reference students with the improved rubric e. Repeat from step 2 until the discrimination check passes
5. The final rubric/grading_guide.md must incorporate insights from rubric/reference_data_analysis/ — it is not simply a transcription of the official rubric PDFs.

Discrimination check criteria — verify ALL of the following:

• AI scores correlate with known ground-truth scores (higher ground-truth → higher AI score, i.e., the rank order is preserved).
• The absolute difference between AI score and known score is within a reasonable tolerance (e.g., ≤ 10% of total marks for each student).
• The score spread across reference students is reasonable (i.e., students are not all clustered within a narrow band when ground-truth scores range from 30 to 100).

Only proceed to Phase 4 (actual student grading) once the discrimination check passes. Document the outcome in rubric/reference_calibration.md.

Temp file organization: All intermediate artifacts from calibration — including reference student grading outputs, draft rubrics, and intermediate score calculations — must be saved under rubric/calibration/ (a dedicated subdirectory). Final reference artifacts (reference_scores.json) are saved under rubric/. Final grading artifacts (grading_guide.md, reference_calibration.md) are saved under rubric/.

Step 3.5: Verify Score Correlation

Using the reference scores from Step 3.4, verify that the AI grading produces scores that correlate with known ground-truth scores. Specifically:

• Compute rank-order correlation (Spearman) between AI scores and known scores
• Verify student_07 (100/100) has the highest or near-highest AI score
• Verify student_23 (30/100) has the lowest or near-lowest AI score
• Verify the general ordering: student_07 > student_17 > student_08 > student_11 > student_23

These correlations validate that the rubric interpretation is reasonable. If the correlation is poor, return to Step 3.4a for further rubric refinement.

Step 3.6: Build Reference Calibration Document

Create rubric/reference_calibration.md with TWO sections:

Section 1 — Qualitative Calibration: Summarise observations from grading the reference students across the score spectrum:

• What a high-scoring answer (90–100%) looks like (key strengths)
• What a mid-high answer (70–89%) looks like
• What a mid answer (50–69%) looks like
• What a low answer (below 50%) looks like

Focus on the main question types for this exam (arithmetic, fractions, word problems, geometry). Include specific examples of how reference students at different score levels approached key questions.

Section 2 — Quantitative Score Correlation: Include the empirical AI vs. ground-truth comparison from Step 3.4:

• For each reference student: known score, AI score, difference, percentage error
• Overall correlation metrics (Spearman rank correlation, mean absolute error)
• Questions with the largest scoring discrepancies and notes on rubric adjustments made

This calibration data provides confidence that the rubric interpretation is sound before grading the 20 target students.

Phase 4: Student Answer Extraction

Step 4.1: Extract Text from Student PDFs

CRITICAL: Student answer extraction MUST be done 1 page at a time to ensure extraction quality. Do NOT batch multiple pages into a single extraction call.

For each of the 20 grading students in data/masked_data/lingliang/student_answers/:

1. Use PyMuPDF to extract text from the PDF, processing ONE PAGE AT A TIME
2. For each page, extract text independently, then concatenate all pages in order
3. If extraction yields empty/garbled text on any page, fall back to VLM image-to-text for that specific page only (1 page per VLM call for accuracy)
4. Save the concatenated extracted text to extracted/students/student_{NN}.txt

Note: Student PDFs are scanned handwritten math work (6 pages each). VLM extraction is very likely needed for most or all pages, since PyMuPDF typically cannot extract handwritten text. Use answer_page_map.json during grading (Phase 5) to locate which page contains the answer for each question.

Handwritten math extraction challenges:

• Students may write fractions as stacked numerator/denominator rather than using the / symbol — VLM must interpret these correctly
• Division may be shown as long division layouts, not inline expressions
• Crossed-out work and corrections are common — the VLM should identify the final answer vs. scratch work
• Red marks have been removed from the scans, but faint artifacts may remain
• Student handwriting quality varies significantly — some answers may be difficult to read even for humans
• Numbers and mathematical symbols may be ambiguous (e.g., 1 vs. 7, 6 vs. 0)

Chain to extraction skill: this phase must be implemented via the extraction skill with request_mode=page_by_page (or page_by_page_with_prev only when boundary context is necessary) to enforce one-page extraction quality requirements, and route via Gemini only.

Note: This 1-page-at-a-time rule applies to student answer PDFs. For rubric/criteria PDFs (Phase 2), multi-page extraction is acceptable since those are typeset documents with cleaner formatting.

Step 4.1a: Export Student Page Images (When Visual Content Detected)

After extraction, inspect the extraction results for visual-content hints. If any student's extraction output includes needs_visual_reference: true or [IMAGE_DATA] placeholders, export the corresponding full-page PNG images from the student's PDF to extracted/students/page_images/student_{NN}/page_{P}.png.

This is especially important for handwritten math answers containing:

• Geometric figures or construction drawings
• Long division or fraction layouts where spatial arrangement carries meaning
• Graphs, charts, or coordinate geometry plots
• Any visual content that cannot be fully represented in extracted text

Use PyMuPDF at 150 DPI (or the DPI set in Rule 4) to render each flagged page. These page images allow Phase 5 grading sub-agents to view the original handwritten content when the extracted text is insufficient.

Step 4.2: Verify Extraction Completeness

Confirm all 20 student text files exist and have non-trivial content. For each student:

1. Verify the extracted text file has content from all 6 pages
2. Check that mathematical expressions are rendered in a parseable format
3. Flag any students whose extraction may be incomplete or where handwriting recognition confidence is low
4. If a student has visual-page artifacts, confirm the referenced PNG files also exist

Step 4.3: Write and Run Extraction Validation Script

Write a script scripts/validate_extraction.py that:

• Iterates over all extracted/students/student_{NN}.txt files
• Checks each file: exists, non-empty, ≥ 50 characters
• Verifies each file has content from multiple pages (not just page 1)
• If extracted/students/page_images/student_{NN}/ exists, verify it contains at least one .png file and report the page-image count
• MANDATORY image check: If a student's .txt file contains [IMAGE_DATA], verify that extracted/students/page_images/student_{NN}/ exists AND contains at least one .png file. Fail validation if [IMAGE_DATA] is present but no page images exist — this means Step 4.1a was skipped
• If [VLM_DESCRIPTION: tags exist, report their count (supplementary, not blocking)
• Prints a summary table: student ID, file size, page count detected, image count, pass/fail
• Exits with a non-zero exit code if ANY student file fails

Run the script and confirm all students pass before proceeding to Phase 5. If any fail, re-run extraction for those students.

Phase 5: Per-Student Grading

Step 5.1: Launch Sub-Agents

For each of the 20 grading students, launch a dedicated sub-agent that:

1. Reads rubric/grading_guide.md

2. Reads rubric/reference_calibration.md (both qualitative and quantitative sections — including score correlation data)

   Calibration note: The reference_calibration.md data validates that the rubric interpretation is consistent. Use it as a sanity check — if a student's score seems unusually high or low, verify the per-question grading is correct against the rubric.

3. Reads the student's extracted text (extracted/students/student_{NN}.txt)

4. Reads answer_page_map.json to know which page(s) contain the answer for each question

5. Grades EVERY question (all 37) against the rubric — focus on accurate per-question mark allocation, applying the rubric criteria precisely

6. Computes total score and percentage from per-question marks

7. Outputs a JSON file to output/student_{NN}.json

8. Handling visual content with [IMAGE_DATA] (MANDATORY — direct image viewing): When the student's extracted text contains [IMAGE_DATA] for any question, the sub-agent MUST:

   • View the student's page image directly using the view tool on the PNG file at extracted/students/page_images/student_{NN}/page_{P}.png
   • View the question image from the per-question JPG file (data/masked_data/lingliang/question/q-{N}.jpg) to understand the question's visual context (diagrams, geometric figures, graphs)
   • View the rubric's reference answer image from the per-question rubric JPG (data/masked_data/lingliang/rubrics/rubrics_q-{N}.jpg) to understand what the correct visual answer looks like
   • Compare the student's visual work against the question image, rubric reference image, and grading guide criteria to award marks
   • Use any [VLM_DESCRIPTION] text as supplementary context, but do NOT rely on it as the sole basis for grading visual content
   • Reference specific visual elements in the evidence field (e.g., "student drew correct geometric construction with compass marks visible")
   • Do NOT give generic "benefit of doubt" scores — assess the actual visual content
   • If neither the page image nor VLM description is available, award 0 for visual criteria and note it in

IMPORTANT: Sub-agents must NOT use hard score restrictions to constrain their grading. The correct approach is: grade each question purely on rubric merit → sum the scores → report the total. Do NOT adjust per-question marks to force a particular total outcome.

Math-specific grading guidance for sub-agents:

• Method marks vs. answer marks: Where the rubric distinguishes these, award method marks for correct working even if the final answer is wrong. Conversely, do not award answer marks if the answer is correct but no working is shown (unless the rubric explicitly allows this).
• Equivalent forms: Accept mathematically equivalent representations unless the rubric specifies a required form. E.g., 0.5 = 1/2 = 50% should all be accepted unless the question says "express as a fraction."
• Units: Check whether the rubric requires units. If required and missing, deduct marks as specified. If the rubric is silent on units, follow the guidance from the grading guide (refined via reference calibration).
• Handwriting interpretation: If a digit or symbol is ambiguous in the extracted text, note the ambiguity in the evidence field. Grade based on the most reasonable interpretation, but flag it for review.
• Crossed-out work: Only grade the student's final answer, not crossed-out attempts. If the extraction includes both, identify which is the final answer.

Control parallelism with BATCH_SIZE — launch up to BATCH_SIZE sub-agents at a time.

Step 5.2: Per-Student Output Schema

Each output/student_{NN}.json MUST follow this schema:

{
  "student_id": "NN",
  "questions": [
    {
      "question_id": "1",
      "max_marks": 2,
      "awarded_marks": 2.0,
      "reasoning": "Step-by-step analysis citing rubric criteria: (1) [Rubric criterion] — student answer satisfies this because [specific part of answer]. Therefore 2/2 marks awarded.",
      "evidence": "Direct quote from student answer: '[verbatim student text]'."
    },
    {
      "question_id": "3.a",
      "max_marks": 3,
      "awarded_marks": 2.0,
      "reasoning": "Step-by-step analysis: (1) Correct method used — 2 marks; (2) Final answer has arithmetic error — 0 marks for answer mark. Therefore 2/3 marks awarded.",
      "evidence": "Student work: '[verbatim calculation steps]'. Final answer: '[student's incorrect answer]' vs. expected '[correct answer]'."
    }
  ],
  "total_raw_score": 78,
  "total_max_score": 100,
  "percentage": 78.0
}

Fields:

• student_id: Student number string (e.g., "01", "14", "26")
• questions: Array of per-question grading objects (37 entries)
  • question_id: String identifier matching the manifest (e.g., "1", "3.a", "9.decimal", "15.b")
  • max_marks: Maximum marks for this question
  • awarded_marks: Marks awarded (float, can be 0.5 increments if rubric allows)
  • reasoning: Step-by-step analysis of mark allocation, always grounded in the rubric first. For each marking point in the grading guide: (a) state the rubric criterion being assessed, (b) explain whether and how the student's answer satisfies it, (c) for partial marks, specify exactly what was present and what was absent. For math questions, distinguish between method marks and answer marks where applicable. General subject-knowledge conventions may supplement the rubric only where the rubric is genuinely silent — and this must be stated explicitly (e.g., "Rubric does not specify, but standard practice requires…").
  • evidence: Direct verbatim quote(s) from the student's answer that demonstrate satisfaction (or non-satisfaction) of each rubric criterion. Do not paraphrase — quote the student answer exactly. For math questions, include the student's working steps and final answer.
• total_raw_score: Sum of all awarded_marks (should be out of 100)
• total_max_score: Sum of all max_marks (should equal 100)
• percentage: (total_raw_score / total_max_score) × 100

Grading methodology — sub-agents MUST follow this sequence:

1. Read the grading guide first. For the question being graded, identify every individual marking point (採分點) and the marks associated with each stage.
2. Read the student's answer. Use answer_page_map.json to locate the correct page, then find the portion of the extracted text that responds to this question.
3. Assess each marking point in turn. For each criterion in the grading guide: does the student's answer meet it? Fully, partially, or not at all?
4. Assign marks per criterion. Award marks strictly based on rubric merit — do not adjust per-question marks to force a predetermined total outcome.
5. Justify every mark decision. The reasoning field must reference the specific rubric criterion and the specific student text (verbatim in evidence) that supports or fails each mark.
6. The rubric is the primary authority. Only apply general math conventions where the rubric is genuinely silent, and flag it when you do.

Step 5.3: Validate Sub-Agent Output

After each sub-agent completes:

1. Verify the JSON file exists and is valid
2. Check that total_raw_score equals sum of awarded_marks
3. Check that total_max_score equals 100
4. Check that no awarded_marks exceeds max_marks
5. Re-run failed sub-agents if necessary

Step 5.4: Write and Run Grading Output Validation Script

Write a script scripts/validate_grading_output.py that:

• Iterates over all output/student_{NN}.json files
• For each file: checks existence, valid JSON, required keys (student_id, questions, total_raw_score, total_max_score, percentage), value ranges (percentage ∈ [0,100], total_max_score == 100), and that total_raw_score equals the sum of all awarded_marks
• Prints a per-student pass/fail table with the specific failure reason for any failures
• Exits with a non-zero exit code if ANY student fails

Run the script. Re-run the sub-agent for any failing students. Re-run the script until all pass.

Phase 6: Score Compilation

Step 6.1: Aggregate All Student JSONs

Read all 20 output/student_{NN}.json files and compile into output/final_scores.json:

{
  "subject": "lingliang",
  "total_students": 20,
  "max_possible_score": 100,
  "students": [
    {
      "student_id": "01",
      "total_raw_score": 78,
      "total_max_score": 100,
      "percentage": 78.0
    }
  ],
  "statistics": {
    "mean_score": 65.3,
    "median_score": 67.0,
    "std_dev": 18.4,
    "min_score": 22,
    "max_score": 95,
    "mean_percentage": 65.3
  }
}

Step 6.2: Per-Question Score Analysis

Optionally, compute per-question statistics to identify questions that were particularly easy or difficult across the cohort:

• For each of the 37 questions: mean score, median score, percentage of students with full marks, percentage with zero marks
• Identify questions with the highest and lowest mean scores
• Flag questions where the scoring variance is unusually high (may indicate inconsistent rubric application)

This per-question analysis is informational — it does not affect individual student scores but provides useful diagnostic data.

Phase 7: Level Division

⚠️ DO NOT EXECUTE — Phase 7 is disabled for this subject. Lingliang is a primary school math exam with no level classification system. The grading output is raw scores only. Skip Phase 7 entirely.

Phase 8: Quality Assurance

⚠️ DO NOT EXECUTE — This phase is disabled for now. Skip Phase 8 entirely. Do not run any steps in this phase.

Step 8.1: Score Distribution Review

After score compilation, review the overall score distribution:

1. Check that the mean/median scores are reasonable for a primary school math exam
2. Identify any outliers (students with extremely high or low scores)
3. Verify the score distribution is plausible

Step 8.2: Spot-Check Sample Students

Randomly select 2–3 students and manually verify their grading:

1. Re-read the student's extracted text
2. Check that per-question marks align with the rubric criteria
3. Verify the total score matches the sum of per-question marks

Step 8.3: Cross-Student Consistency

Compare students with similar total scores:

1. Students with similar totals should have comparable per-question score patterns
2. Flag any cases where two students have very different scores on the same question without clear justification
3. Check that the score ordering produces a reasonable ranking of students

Step 8.4: Final Output Verification

Confirm all output files are complete and valid:

1. All student_{NN}.json files exist with valid JSON (20 students)
2. final_scores.json contains all 20 students with correct statistics
3. No missing or corrupted files

Subject-Specific Data Notes

Lingliang (Primary School Math)

• Masked data: data/masked_data/lingliang/
  • Rubrics: rubrics/rubrics_q-{N}.pdf + rubrics/rubrics_q-{N}.jpg (37 per-question pairs)
  • Question materials: question/q-{N}.txt + question/q-{N}.jpg
  • Question manifest: question_manifest.csv
  • Students: 20 (student_01.pdf – student_26.pdf, excluding 07, 08, 11, 17, 23)
• Reference data: data/reference/lingliang/
  • Students: 5 (student_07.pdf, student_08.pdf, student_11.pdf, student_17.pdf, student_23.pdf)
  • reference_mapping.json with known total scores
  • No level-based filenames — all use student_{NN}.pdf format

Error Handling

• Missing PDFs: Fail early with a clear error listing which files are missing
• Empty text extraction: Flag the student and retry extraction via Gemini (gemini-3.1-flash-lite-preview, 1 page per call)
• Invalid sub-agent output: Re-run the sub-agent with clearer instructions
• Score validation failures: Log warnings but do not silently adjust scores
• Missing reference data: Proceed with grading but note that calibration is incomplete
• Handwritten text recognition failures: For mathematical notation that VLM cannot interpret, flag the specific question and student for manual review
• Question manifest inconsistencies: If question_manifest.csv references files that don't exist on disk, fail with a clear listing of missing files before proceeding
• answer_page_map.json missing or invalid: If the page map cannot be loaded, fail early — it is required for accurate per-question extraction from student PDFs
• Total marks mismatch: If per-question marks in the grading guide don't sum to 100, halt and investigate before proceeding to grading

Checklist

• Dependencies installed (uv sync)
• model-service skill run at least once to confirm Gemini (gemini-3.1-flash-lite-preview) route availability from env.txt
• Student PDFs confirmed in masked data directory (20 grading students)
• question_manifest.csv verified (37 questions, all paths valid)
• answer_page_map.json verified and accessible
• Per-question rubric PDFs+JPGs read and grading guide built (rubric/grading_guide.md)
• Grading guide total marks verified (sum = 100)
• Reference students graded and score correlation verified (rubric/reference_scores.json)
• Rubric discrimination check passed (Step 3.4a) — AI scores correlate with known scores
• Reference calibration document built (qualitative + quantitative) (rubric/reference_calibration.md)
• extraction skill used in Phase 2/3/4 extraction tasks with Gemini (gemini-3.1-flash-lite-preview) routing
• All 20 student texts extracted (1 page at a time for student answers)
• Per-student grading complete (one sub-agent per student, 20 students)
• Grading output validated (scripts/validate_grading_output.py passes for all 20 students)
• Score compilation done (output/final_scores.json)
• Phase 7 (Level Division) skipped — not applicable for this subject
• Phase 8 (Quality Assurance) skipped — disabled for now