HKDSE Subject Grading — Unified Skill

Workspace Layout

{subject}-grading-workspaces/grading_workspace/
├── .github/skills/hkdse-{subject}-grading/SKILL.md
├── START.md
├── WARNING.md
├── env.txt                     # Local credentials (never commit)
├── env.txt.example             # Template
├── pyproject.toml
├── uv.lock
├── start.sh
├── data/                       # Symlink or mount to top-level data/
│   ├── reference/{subject}/{ref_year}/
│   │   ├── student_answers/level{L}_student{M}.pdf
│   │   ├── rubrics/ OR rubric_and_question/
│   │   ├── question/           (if separate)
│   │   └── reference_mapping.json
│   ├── masked_data/{subject}/{grade_year}/
│   │   ├── student_answers/student{N}.pdf
│   │   ├── rubrics/ OR rubric_and_question/
│   │   └── question/           (if separate)
│   └── groundtruth/{subject}/{grade_year}/groundtruth_mapping.json
├── rubric/{grade_year}/        # Generated rubric artifacts
│   ├── grading_guide.md
│   ├── reference_calibration.md
│   ├── reference_scores.json
│   ├── level_division.json
│   └── calibration/            # Intermediate calibration artifacts
│       └── (draft rubrics, reference grading outputs, score calculations)
├── rubric/reference_data_analysis/  # Insights from reference data (Phase 3)
│   ├── per_level_analysis.md        # Observed patterns per level 1–5
│   ├── rubric_gaps.md               # Gaps/ambiguities in official rubric
│   └── rubric_refinements.md        # Supplementary rubric guidance
├── scripts/                    # Python helper scripts
│   ├── generate_class_report.py
│   ├── generate_student_reports.py
│   ├── validate_extraction.py
│   ├── validate_grading_output.py
│   └── validate_reports.py
├── extracted/{grade_year}/     # Extracted student text
│   └── students/student{N}.txt
└── output/{grade_year}/        # Grading output
    ├── student{N}.json
    └── final_scores.json

Phase 1: Environment Setup

Step 1.1: Install Dependencies and Set Year

source env.txt
uv sync
export GRADING_YEAR={grade_year}   # Set from env.txt or manually

Step 1.2: Verify Data Availability

Confirm the following exist:

• Student answer PDFs in data/masked_data/{subject}/{grade_year}/student_answers/
• Rubric PDFs (check both rubrics/ and rubric_and_question/ directories)
• Question paper PDFs (if separate from rubrics)
• Reference data in data/reference/{subject}/{ref_year}/ with reference_mapping.json

Read BATCH_SIZE from env.txt (default: 5) for parallel sub-agent control.

Phase 2: Rubric Extraction

Step 2.1: Extract Rubric Content

Read ALL rubric PDFs from the masked data directory. Handle these patterns:

1. Single rubric file: rubrics/rubrics.pdf — extract directly
2. Split rubric parts: rubrics/rubrics_part1.pdf … rubrics_partN.pdf — read ALL parts in numerical order, concatenate content
3. Combined rubric + question: rubric_and_question/paper_part1.pdf … paper_partN.pdf — read ALL parts in order; both rubric criteria and question text are interleaved in these files
4. Level descriptors: If rubrics/level_descriptors.pdf exists, ALWAYS read it — it contains official level-band boundaries critical for level assignment

Use PyMuPDF (fitz) for text extraction. Fall back to VLM only if text extraction yields empty or garbled content.

Note on rubric extraction granularity: For rubric/criteria PDFs (which are typically typeset/printed), extracting multiple pages at a time is acceptable. However, for extremely long rubric files (e.g., 17+ split parts), consider extracting in manageable batches of a few pages each to avoid truncation or quality degradation.

Step 2.2: Extract Question Paper

If the question paper is separate from rubrics:

• Single file: question/paper.pdf — extract directly
• Split parts: question/paper_part1.pdf … paper_partN.pdf — read in order
• Files with non-standard names (e.g., Chinese filenames) — read all PDFs in question/

Step 2.3: Build Grading Guide

Create rubric/{grade_year}/grading_guide.md with:

• All questions/tasks listed with their question numbers
• Mark allocation per question and sub-question
• Marking criteria: what earns each mark or mark band
• Level-band boundaries (if present in rubric or level descriptors)
• Any special instructions (e.g., alternative acceptable answers)

Phase 3: Reference Calibration

Step 3.1: Read Reference Data

Read data/reference/{subject}/{ref_year}/reference_mapping.json to get the mapping of student IDs to known levels (1–5).

Format:

{
  "mappings": [
    {"student_id": 1, "filename": "level3_student1.pdf", "level": 3, ...},
    ...
  ]
}

Note: Reference files keep their original level-based names (level{L}_student{M}.pdf). The filename field in the mapping matches the actual file on disk. The level is also encoded in the filename, making it easy for agents to identify which level each student belongs to without parsing the JSON.

Step 3.2: Extract Reference Student Answers

Extract text from ALL reference student answer PDFs in data/reference/{subject}/{ref_year}/student_answers/.

Step 3.3: Read Reference Rubrics

Read rubric PDFs from the REFERENCE year directory (structure may differ from the grading year). Apply the same extraction rules as Phase 2.

Step 3.4: Grade Reference Students (Iterative Rubric Refinement)

Grade ALL reference students using the same rubric and grading guide (from Phase 2). For each reference student, apply the per-question mark allocation to produce a total raw score and percentage — exactly as you would for a masked student. Record these scores alongside their known levels from reference_mapping.json.

This step is critical: the resulting scores provide the empirical score-to-level mapping that replaces hard-coded boundaries. Launch sub-agents for reference students following the same rules as Phase 5 (one sub-agent per student, same model restriction).

Save reference grading results to rubric/{grade_year}/reference_scores.json:

{
  "reference_year": "{ref_year}",
  "scores": [
    {"student_id": 1, "level": 3, "total_raw_score": 42, "total_max_score": 80, "percentage": 52.5},
    {"student_id": 2, "level": 5, "total_raw_score": 72, "total_max_score": 80, "percentage": 90.0}
  ],
  "level_score_ranges": {
    "1": {"min_pct": 12.5, "max_pct": 22.0, "mean_pct": 17.3, "count": 2},
    "2": {"min_pct": 28.0, "max_pct": 38.5, "mean_pct": 33.3, "count": 2},
    "3": {"min_pct": 45.0, "max_pct": 55.0, "mean_pct": 50.0, "count": 2},
    "4": {"min_pct": 62.0, "max_pct": 74.0, "mean_pct": 68.0, "count": 2},
    "5": {"min_pct": 82.0, "max_pct": 95.0, "mean_pct": 88.5, "count": 2}
  }
}

Step 3.4a: Reference Data Analysis and Iterative Rubric Refinement

The rubric must not be finalised from the official PDF alone. The agent must learn from the reference student data to derive an effective grading rubric — one that actually discriminates between student levels as observed in practice. This is an iterative loop that continues until the rubric reliably separates levels.

This is especially important for Visual Arts and Chinese / English Writing, where the official rubric provides only broad bands and subjective judgment is required. For these subjects, the reference data analysis must be particularly thorough: document what distinguishes each level in observable, concrete terms.

All analytical outputs from this step are saved to rubric/reference_data_analysis/.

Iterative loop:

1. Grade all reference students with the current rubric draft (Step 3.4 above).
2. Analyse grading results versus known levels:
   • Are scores meaningfully different across levels?
   • Which rubric criteria caused confusion or over/under-scoring?
   • What did good (Level 4–5) answers do that poor (Level 1–2) answers did not?
   • Are there patterns in how students at each level approach questions differently?
3. Write rubric/reference_data_analysis/per_level_analysis.md documenting observed patterns for each level 1–5:
   • Typical strengths at this level
   • Common errors and misconceptions
   • What kinds of mistakes are acceptable vs. unacceptable
   • Concrete examples from reference student answers
   • What distinguishes this level from adjacent levels
4. Run the discrimination check (see criteria below). If the check fails, or if qualitative analysis reveals the rubric is inadequate: a. Document gaps in rubric/reference_data_analysis/rubric_gaps.md — gaps or ambiguities found in the official rubric when applied to reference students (e.g., criteria that fail to distinguish adjacent levels, missing guidance for common student behaviours, mark schemes that reward or penalise inconsistently) b. Derive refinements in rubric/reference_data_analysis/rubric_refinements.md — supplementary rubric interpretations and practical guidance that resolve the gaps above (e.g., clarifying what "adequate discussion" means at each level, adding concrete examples, specifying how to handle partial credit) c. Revise rubric/{grade_year}/grading_guide.md to incorporate these refinements — the grading guide is not simply extracted from the official rubric PDF; it must incorporate insights from the reference data analysis d. Re-grade affected reference students with the improved rubric e. Repeat from step 2 until the discrimination check passes
5. The final rubric/{grade_year}/grading_guide.md must incorporate insights from rubric/reference_data_analysis/ — it is not simply a transcription of the official rubric PDF.

Discrimination check criteria — verify ALL of the following:

• Each level has a meaningfully distinct score range (minimal overlap between adjacent levels).
• At least 4 of the 5 levels are represented by at least one reference student.
• The score spread across levels is reasonable (i.e., students are not all clustered within a narrow band).

Only proceed to Phase 4 (actual student grading) once the discrimination check passes. Document the outcome in rubric/{grade_year}/reference_calibration.md.

Temp file organization: All intermediate artifacts from calibration — including reference student grading outputs, draft rubrics, and intermediate score calculations — must be saved under rubric/{grade_year}/calibration/ (a dedicated subdirectory). Final artifacts (grading_guide.md, reference_calibration.md, reference_scores.json, level_division.json) remain directly under rubric/{grade_year}/ as before.

Step 3.5: Compute Score-to-Level Boundaries

Using the reference scores from Step 3.4, compute level boundaries to guide level assignment. One common approach is finding midpoints between adjacent levels' score ranges, but agents may use any reasonable method.

These boundaries are guidelines derived from reference data, not hard restrictions. They serve as a starting point for level assignment in Phase 7, where agents may refine them using additional information from the rubric and level descriptors.

Step 3.6: Build Reference Calibration Document

Create rubric/{grade_year}/reference_calibration.md with TWO sections:

Section 1 — Qualitative Calibration: For each level (1–5), summarise what rubric criteria reference students at that level demonstrated:

• What a Level 1 answer looks like (key deficiencies)
• What a Level 2 answer looks like
• What a Level 3 answer looks like
• What a Level 4 answer looks like
• What a Level 5 answer looks like

Focus on the main marking dimensions for the subject. Include specific examples of how reference students at each level addressed key questions.

Section 2 — Quantitative Score Ranges: Include the empirical score-to-level mapping from Step 3.4:

• Level 1: score range X%–Y% (mean Z%), N reference students
• Level 2: score range X%–Y% (mean Z%), N reference students
• Level 3: score range X%–Y% (mean Z%), N reference students
• Level 4: score range X%–Y% (mean Z%), N reference students
• Level 5: score range X%–Y% (mean Z%), N reference students
• Computed level boundaries (from Step 3.5)

This quantitative mapping provides useful guidance for level assignment in Phases 5 and 7. Sub-agents and level division may use these ranges alongside rubric information and level descriptors to determine levels from scores.

Phase 4: Student Answer Extraction

Step 4.1: Extract Text from Student PDFs

CRITICAL: Student answer extraction MUST be done 1 page at a time to ensure extraction quality. Do NOT batch multiple pages into a single extraction call.

For each student in data/masked_data/{subject}/{grade_year}/student_answers/:

1. Use PyMuPDF to extract text from the PDF, processing ONE PAGE AT A TIME
2. For each page, extract text independently, then concatenate all pages in order
3. If extraction yields empty/garbled text on any page, fall back to VLM image-to-text for that specific page only (1 page per VLM call for accuracy)
4. Save the concatenated extracted text to extracted/{grade_year}/students/student{N}.txt

Note: This 1-page-at-a-time rule applies to student answer PDFs. For rubric/criteria PDFs (Phase 2), multi-page extraction is acceptable since those are typeset documents with cleaner formatting.

Step 4.2: Verify Extraction Completeness

Confirm all student text files exist and have non-trivial content. Flag any students whose extraction may be incomplete.

Step 4.3: Write and Run Extraction Validation Script

Write a script scripts/validate_extraction.py that:

• Iterates over all extracted/{grade_year}/students/student{N}.txt files
• Checks each file: exists, non-empty, ≥ 50 characters
• Prints a summary table: student ID, file size, pass/fail
• Exits with a non-zero exit code if ANY student file fails

Run the script and confirm all students pass before proceeding to Phase 5. If any fail, re-run extraction for those students.

Phase 5: Per-Student Grading

Step 5.1: Launch Sub-Agents

For each student, launch a dedicated sub-agent that:

1. Reads rubric/{grade_year}/grading_guide.md

2. Reads rubric/{grade_year}/reference_calibration.md (both qualitative and quantitative sections — including score-to-level ranges)

   Anti-bias note: The reference_calibration.md score ranges are empirical data from one prior year. Do NOT treat them as fixed cutoffs. The preliminary_level should reflect the quality of the student's work against rubric criteria. If the student's score falls slightly outside a reference range but clearly matches a level's qualitative description, assign that level and note the reasoning.

3. Reads the student's extracted text (extracted/{grade_year}/students/student{N}.txt)

4. Grades EVERY question/sub-question against the rubric — focus on accurate per-question mark allocation first, without constraining scores to fit a predetermined level

5. Computes total score and percentage from per-question marks

6. Derives preliminary_level using the reference score ranges and any other available level information (rubric, level descriptors, reference calibration)

7. Outputs a JSON file to output/{grade_year}/student{N}.json

IMPORTANT: Sub-agents must NOT use hard score restrictions to constrain their grading. The correct approach is: grade each question purely on rubric merit → sum the scores → map the total to a level using reference score ranges. Do NOT adjust per-question marks to force a particular level outcome.

Control parallelism with BATCH_SIZE — launch up to BATCH_SIZE sub-agents at a time.

Step 5.2: Per-Student Output Schema

Each output/{grade_year}/student{N}.json MUST follow this schema:

{
  "student_id": N,
  "questions": [
    {
      "question_id": "Q1a",
      "max_marks": 4,
      "awarded_marks": 3.0,
      "reasoning": "Brief explanation of mark allocation",
      "evidence": "Relevant quote or reference from student answer"
    }
  ],
  "total_raw_score": 45,
  "total_max_score": 60,
  "percentage": 75.0,
  "preliminary_level": 4,
  "level_reasoning": "Based on reference calibration, this student's performance aligns with Level 4 characteristics: ..."
}

Fields:

• student_id: Integer student number
• questions: Array of per-question grading objects
  • question_id: String identifier (e.g., "Q1a", "Q2bii")
  • max_marks: Maximum marks for this question
  • awarded_marks: Marks awarded (float, can be 0.5 increments if rubric allows)
  • reasoning: Why these marks were awarded
  • evidence: Supporting evidence from the student's answer
• total_raw_score: Sum of all awarded_marks
• total_max_score: Sum of all max_marks
• percentage: (total_raw_score / total_max_score) × 100
• preliminary_level: Level (1–5) derived using the reference score ranges and available level information. This is a best-fit estimate
• level_reasoning: Explanation for the level assignment, referencing the score range it falls into (e.g., "Score 52.5% falls within Level 3 reference range 45.0%–55.0%")

Step 5.3: Validate Sub-Agent Output

After each sub-agent completes:

1. Verify the JSON file exists and is valid
2. Check that total_raw_score equals sum of awarded_marks
3. Check that no awarded_marks exceeds max_marks
4. Check that preliminary_level is between 1 and 5
5. Re-run failed sub-agents if necessary

Step 5.4: Write and Run Grading Output Validation Script

Write a script scripts/validate_grading_output.py that:

• Iterates over all output/{grade_year}/student{N}.json files
• For each file: checks existence, valid JSON, required keys (student_id, questions, total_raw_score, total_max_score, percentage, preliminary_level), value ranges (preliminary_level ∈ [1,5], percentage ∈ [0,100]), and that total_raw_score equals the sum of all awarded_marks
• Prints a per-student pass/fail table with the specific failure reason for any failures
• Exits with a non-zero exit code if ANY student fails

Run the script. Re-run the sub-agent for any failing students. Re-run the script until all pass.

Phase 6: Score Compilation

Step 6.1: Aggregate All Student JSONs

Read all output/{grade_year}/student{N}.json files and compile into output/{grade_year}/final_scores.json:

{
  "subject": "{subject}",
  "year": "{grade_year}",
  "total_students": N,
  "max_possible_score": 60,
  "students": [
    {
      "student_id": 1,
      "total_raw_score": 45,
      "total_max_score": 60,
      "percentage": 75.0,
      "preliminary_level": 4
    }
  ],
  "statistics": {
    "mean_score": 42.5,
    "median_score": 43.0,
    "std_dev": 8.2,
    "min_score": 20,
    "max_score": 58,
    "mean_percentage": 70.8
  }
}

Phase 7: Level Division

Step 7.1: Determine Level Boundaries

Agents may use the extracted reference data (from Phase 3) to inform level division. The following sources are available — agents should consider all relevant information and exercise judgment:

• Reference score ranges (rubric/{grade_year}/reference_scores.json) — the empirical score ranges from grading reference students in Phase 3. These provide an empirical basis for mapping scores to levels.
• Official level descriptors (from level_descriptors.pdf if available) — these describe the qualitative characteristics expected at each level
• Criteria-based boundaries derived from the grading guide's mark scheme
• Reference calibration document (rubric/{grade_year}/reference_calibration.md) — contains both qualitative descriptions and quantitative score ranges per level

There are no hard restrictions on how level boundaries must be computed. Agents should use the available reference data, rubric information, and level descriptors to define reasonable level boundaries that reflect the subject's scoring patterns.

⚠️ Reference Year Bias Prevention:

The reference year's score ranges are a starting point for calibration only, not fixed thresholds to be carried forward. Before finalising level boundaries, the agent must:

1. After compiling all current-year student scores (Phase 6), compute the empirical score distribution of the current cohort (mean, median, spread, approximate percentiles).
2. Compare this distribution to the reference year's level score ranges. Identify any significant cohort-level shift (e.g., current-year scores are systematically 10–20% lower or higher than reference year scores at the same qualitative level of work).
3. If a significant shift exists, proportionally adjust the level boundaries to account for it. The goal is that level division reflects the actual quality distribution of the current cohort as judged against rubric criteria — not a fixed mapping derived exclusively from one prior year's data.
4. Mandatory anti-bias rule: If the reference year shows Level 5 students scoring 70–100%, but this year's highest-performing students score around 60%, do NOT conclude that no Level 5 students exist. Instead, determine whether 60% represents genuinely Level 5 work under the rubric, and if so, adjust the Level 5 boundary down accordingly.
5. The final level_division.json must include an "adjustment_notes" field. It must either explain any deviation from raw reference boundaries and the reasoning behind the adjustment, or explicitly state "No adjustment; current cohort distribution aligns with reference year ranges."

Step 7.2: Assign Final Levels

For each student, map their total score to a level (1–5) using the reference-derived boundaries from Step 7.1. Save to rubric/{grade_year}/level_division.json:

{
  "method": "reference_score_based",
  "reference_year": "{ref_year}",
  "boundaries": {
    "level_1": {"min_percentage": 0, "max_percentage": 27.5},
    "level_2": {"min_percentage": 27.5, "max_percentage": 41.75},
    "level_3": {"min_percentage": 41.75, "max_percentage": 60.0},
    "level_4": {"min_percentage": 60.0, "max_percentage": 78.0},
    "level_5": {"min_percentage": 78.0, "max_percentage": 100}
  },
  "reference_score_ranges": {
    "level_1": {"min_pct": 12.5, "max_pct": 22.0, "mean_pct": 17.3},
    "level_2": {"min_pct": 28.0, "max_pct": 38.5, "mean_pct": 33.3},
    "level_3": {"min_pct": 45.0, "max_pct": 55.0, "mean_pct": 50.0},
    "level_4": {"min_pct": 62.0, "max_pct": 74.0, "mean_pct": 68.0},
    "level_5": {"min_pct": 82.0, "max_pct": 95.0, "mean_pct": 88.5}
  },
  "adjustment_notes": "No adjustment; current cohort distribution aligns with reference year ranges.",
  "students": [
    {
      "student_id": 1,
      "total_raw_score": 45,
      "percentage": 75.0,
      "preliminary_level": 4,
      "final_level": 4,
      "level_reasoning": "Score 75.0% falls within Level 4 reference range (62.0%–74.0%), closest to Level 4 mean (68.0%)"
    }
  ]
}

Note: The boundaries values above are examples. Agents should derive appropriate boundaries from the available reference data and rubric information.

Step 7.3: Cross-Validate with Reference

Compare the level distribution of graded students against the reference year's distribution. Flag any significant discrepancies (e.g., if the reference year had students at all 5 levels but the graded year has none at Level 1).

Subject-Specific Data Notes

ICT (Information and Communication Technology)

• Masked data (2025): data/masked_data/ict/2025/
  • Rubrics: rubrics/rubrics.pdf (single file)
  • Question: question/paper_part1.pdf, question/paper_part2.pdf
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2024): data/reference/ict/2024/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part17.pdf (17 split parts)
  • Question: question/paper.pdf
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Music

• Masked data (2024): data/masked_data/music/2024/
  • Rubric AND question combined: rubric_and_question/paper_part1.pdf … paper_part21.pdf
  • No separate rubrics/ or question/ directories
  • Students: 27 (student1.pdf – student27.pdf)
• Reference data (2025): data/reference/music/2025/
  • Rubric AND question combined: rubric_and_question/paper_part1.pdf … paper_part24.pdf
  • Students: 31 labeled, level-based names (level1_student1.pdf … level5_student8.pdf)

Ethics and Religious Studies

• Masked data (2025): data/masked_data/religion/2025/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part7.pdf (7 split parts)
  • Question: question/paper.pdf
  • Students: 15 (student1.pdf – student15.pdf)
• Reference data (2024): data/reference/religion/2024/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part7.pdf (7 split parts)
  • Question: question/paper.pdf
  • Students: 14 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Tourism and Hospitality Studies

• Masked data (2025): data/masked_data/tourism/2025/
  • Rubrics: rubrics/rubrics.pdf (main mark scheme) AND rubrics/level_descriptors.pdf (official level band descriptors)
  • Always read level_descriptors.pdf — it contains the official level boundaries
  • Question: question/paper.pdf
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2024): data/reference/tourism/2024/
  • Rubrics: rubrics/rubrics.pdf AND rubrics/level_descriptors.pdf
  • Question: question/paper.pdf
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Visual Arts

• Masked data (2020): data/masked_data/visual-arts/2020/
  • Rubrics: rubrics/rubrics.pdf (single file)
  • Question: question/ directory — files may have Chinese filenames (e.g., 2020_視藝_P1QAB.pdf, 2020_視藝_P2QAB.pdf); read ALL PDFs in this directory
  • Students: 18 (student1.pdf – student18.pdf)
• Reference data (2025): data/reference/visual-arts/2025/
  • NOTE: Reference year (2025) is NEWER than grading year (2020)
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part5.pdf (5 split parts)
  • Question: question/paper_part1.pdf … paper_part5.pdf (5 split parts)
  • Students: 24 labeled, level-based names (level1_student1.pdf … level5_student6.pdf)
  • Use reference data purely for rubric calibration; do not assume rubric changes between years
• ⚠️ Art creation grading: Art-making/creation components MUST be graded via VLM directly from student PDF page images — NOT from extracted text. Visual elements (drawings, compositions, color, spatial arrangement) cannot be captured by text extraction. Only written/critical appreciation portions should use extracted text.

Biology

• Masked data (2025): data/masked_data/biology/2025/
  • Rubrics: rubrics/rubrics.pdf (single file)
  • Question: question/paper.pdf
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2024): data/reference/biology/2024/
  • Rubrics: rubrics/rubrics.pdf
  • Question: question/paper.pdf
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)
  • reference_mapping.json available

Economics

• Masked data (2024): data/masked_data/economics/2024/
  • Rubrics: rubrics/marking_scheme.pdf and rubrics/general_rubrics.pdf (two files)
  • Question papers: question/paper1.pdf and question/paper2.pdf (two papers)
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2025): data/reference/economics/2025/
  • NOTE: Reference year (2025) is NEWER than grading year (2024)
  • Rubrics: rubrics/general_rubrics.pdf
  • Question papers: question/paper1.pdf and question/paper2.pdf (two papers)
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)
  • reference_mapping.json available

Error Handling

• Missing PDFs: Fail early with a clear error listing which files are missing
• Empty text extraction: Flag the student and attempt VLM fallback
• Invalid sub-agent output: Re-run the sub-agent with clearer instructions
• Score validation failures: Log warnings but do not silently adjust scores
• Missing reference data: Proceed with grading but note that calibration is incomplete

Phase 8: Quality Assurance

Step 8.1: Score Distribution Review

After level division, review the overall score distribution:

1. Check that the mean/median scores are reasonable for the subject
2. Identify any outliers (students with extremely high or low scores)
3. Verify the level distribution is plausible (not all students in one level)

Step 8.2: Spot-Check Sample Students

Randomly select 2–3 students and manually verify their grading:

1. Re-read the student's extracted text
2. Check that per-question marks align with the rubric criteria
3. Verify the total score matches the sum of per-question marks
4. Confirm the assigned level is consistent with reference calibration

Step 8.3: Cross-Student Consistency

Compare students at the same level:

1. Students assigned to the same level should have comparable scores
2. Flag any cases where two students have very different scores but the same level
3. Check that level boundaries produce a reasonable ordering of students

Step 8.4: Final Output Verification

Confirm all output files are complete and valid:

1. All student{N}.json files exist with valid JSON
2. final_scores.json contains all students with correct statistics
3. rubric/{grade_year}/level_division.json has valid boundaries and all student assignments
4. No missing or corrupted files

Checklist

• Year set via GRADING_YEAR env var
• Dependencies installed (uv sync)
• Student PDFs confirmed in masked data directory
• Rubric PDFs read and grading guide built
• Reference students graded and score ranges computed (reference_scores.json)
• Rubric discrimination check passed (Step 3.4a)
• Reference calibration document built (qualitative + quantitative)
• All student texts extracted (1 page at a time for student answers)
• Per-student grading complete (one sub-agent per student)
• Score compilation done (final_scores.json)
• Level division done (rubric/{grade_year}/level_division.json)
• Quality assurance completed (score distribution, spot-checks, consistency)