Hkdse Ict Grading

The full grading workflow follows the unified skill defined in .github/skills/hkdse-subject-grading/SKILL.md. The complete content of that skill is reproduced below for reference.

HKDSE Subject Grading — Unified Skill

Overview

This skill grades HKDSE elective subject exam papers by extracting rubrics from PDFs, calibrating against prior-year reference data with known levels, then grading each student's typed/printed answer PDF on a per-question basis. It produces per-student JSON results and compiles them into final scores with level assignments (1–5).

Subjects supported: ICT, Music, Ethics and Religious Studies, Tourism and Hospitality Studies, Visual Arts — and any similar subjects with typed written responses.

Key differences from Chinese Writing grading:

• No dedicated Chinese handwriting scoring dimensions (字體標點, 錯別字 are not standalone grading categories — VLM is still permitted for visual inspection where needed; see WARNING.md)
• No Chinese task 1 / task 2 split
• No per-dimension Chinese writing rubric (content, expression, structure, tone)
• Instead: unified per-question mark allocation from official rubrics

CRITICAL RULES

1. Sub-agents are mandatory. Each student MUST be graded by a dedicated sub-agent. No batch grading of multiple students in a single sub-agent call.

2. Model restrictions:

   • Sub-agents: SAME model as the main agent. All sub-agents MUST use the same model that the main (orchestrating) agent is running on. Do NOT pass a model parameter override to sub-agents — let them inherit the main agent's model.
   • VLM tasks: Gemini ONLY. When VLM (vision-language model) API calls are needed — e.g., text extraction from scanned PDFs where PyMuPDF fails, or art creation grading requiring visual inspection — use gemini-3.1-flash-lite-preview via the Gemini route exclusively. NO OTHER VLM MODELS OR ROUTES ARE ALLOWED.

3. Gemini only (for both added chains):

   • extraction skill and model-service skill must both use Gemini for API calls.
   • Read GEMINI_BASE_URL, VERTEX_API_KEY from env.txt.
   • Model must be gemini-3.1-flash-lite-preview.
   • Do not route these two chains to Aliyun/OpenRouter/native Moonshot/Kimi/Seed.
   • Chain to model-service skill: run a config-based route sanity check that confirms route=gemini is callable and non-gemini routes are ignored.

4. PDF rendering DPI: When converting PDF pages to images (e.g., for VLM extraction), default to 150 DPI to ensure text is legible. If processing speed is a concern, lowering DPI is acceptable as a trade-off, but do not go below 100 DPI.

5. Decide image preservation from extraction results, not pre-run config. When a page's meaning may depend on layout or visuals — e.g., diagrams, geometry-style constructions, charts, screenshots, tables with spatial meaning, arrows/labels, or annotated drawings — the extraction API output must explicitly signal this (for example via needs_visual_reference + visual_reference_reason). The agent should inspect those returned hints and only then decide whether to export the whole page image as an extraction artifact. Do not crop at this stage.

6. No LLM/VLM API calls for generating feedback or analysis text. See WARNING.md. VLM is permitted only for tasks requiring genuine visual inspection — text extraction from scanned PDFs where PyMuPDF fails, and grading inherently visual student work (e.g., art creation, handwriting quality, diagrams). VLM must NOT be used for any dimension assessable from extracted text.

7. Read split-part PDFs in numerical order. Files named _part1.pdf, _part2.pdf, …, _partN.pdf MUST be read in sequence. Concatenate the extracted content in order. Ignore _original.pdf files (these are the unsplit source).

8. Reference data informs level assignment. Grade reference students to compute empirical score ranges per level. Agents may use these ranges, along with rubric criteria and official level descriptors, to define level boundaries. There are no hard restrictions on the level division method.

9. Always set the year before running any operation. Either pass --year YEAR to scripts or export GRADING_YEAR=YYYY at the start.

10. All report and commentary text must be written in Traditional Chinese (繁體中文). This applies to all narrative, feedback, analysis, and section content in generated DOCX reports. English is only permitted for: variable names, file paths, technical identifiers, chart axis labels, and JSON field names. Exception: if the subject being assessed is an English-language subject, English is used throughout.

11. Skill-chain requirement (new):

• For PDF/image text extraction tasks in this grading workflow, invoke the extraction skill.
• For any direct LLM API calling/routing validation task, invoke the model-service skill (YAML-only, no .py edits).
• Both skills must use Gemini (gemini-3.1-flash-lite-preview) and read URL/API key values from env.txt; missing values mean that provider/API is unsupported in the current environment.

Skill Chain Integration

Use this grading skill as the orchestrator and explicitly chain the migrated skills:

1. Extraction path (extraction skill):
   • Phase 2 rubric/question extraction
   • Phase 3 reference-student answer extraction
   • Phase 4 masked-student answer extraction (one page at a time for student answers)
2. Model API path (model-service skill):
   • Provider route checks and API availability checks for Gemini (gemini-3.1-flash-lite-preview)
   • Controlled direct API calls for extraction-related troubleshooting
3. Boundary:
   • extraction and model-service are config-driven execution skills; do not modify their underlying Python scripts inside this grading workflow.

Workspace Layout

Canonical ICT pilot location: tasks/ict-grading/workspace/

Legacy compatibility path: ict-grading-workspaces/grading_workspace/ (symlink to the canonical workspace)

tasks/ict-grading/workspace/
├── .github/skills/hkdse-{subject}-grading/SKILL.md
├── START.md
├── WARNING.md
├── env.txt                     # Local credentials (never commit)
├── env.txt.example             # Template
├── pyproject.toml              # Symlink to ../../../envs/ict-grading/pyproject.toml
├── uv.lock                     # Symlink to ../../../envs/ict-grading/uv.lock
├── start.sh
├── data/                       # Local data (pre-copied into workspace)
│   ├── reference/ict/2024/
│   │   ├── student_answers/level{L}_student{M}.pdf (original level-based names)
│   │   ├── rubrics/rubrics_part1.pdf … rubrics_part17.pdf
│   │   ├── question/paper.pdf
│   │   └── reference_mapping.json
│   └── masked_data/ict/2025/
│       ├── student_answers/student1.pdf … student10.pdf
│       ├── rubrics/rubrics.pdf
│       └── question/paper_part1.pdf, paper_part2.pdf
├── rubric/{grade_year}/        # Grading-year artifacts
│   ├── grading_guide.md
│   ├── reference_calibration.md  # Summary derived from reference year; used by current-year grading
│   ├── page_images/            # Optional full-page PNG artifacts for rubric/question pages
│   └── level_division.json
├── rubric/{ref_year}/          # Reference-year calibration artifacts (e.g., rubric/2024/)
│   ├── reference_scores.json   # Per-student scores for the reference cohort
│   └── calibration/            # Intermediate calibration artifacts
│       └── (per-student reference grading outputs, draft rubrics, score calculations)
├── rubric/reference_data_analysis/  # Insights from reference data (Phase 3)
│   ├── per_level_analysis.md        # Observed patterns per level 1–5
│   ├── rubric_gaps.md               # Gaps/ambiguities in official rubric
│   └── rubric_refinements.md        # Supplementary rubric guidance
├── scripts/                    # Optional workspace link to ../../../skills/hkdse-ict-report/scripts
│   ├── generate_class_report.py
│   ├── generate_student_reports.py
│   ├── validate_extraction.py
│   ├── validate_grading_output.py
│   └── validate_reports.py
├── extracted/{grade_year}/     # Extracted student artifacts
│   └── students/
│       ├── student{N}.txt
│       └── page_images/student{N}/page_{P}.png   # Optional visual-page artifacts
└── output/{grade_year}/        # Grading output
    ├── student{N}.json
    └── final_scores.json

Phase 1: Environment Setup

Step 1.1: Install Dependencies and Set Year

source env.txt
uv sync
export GRADING_YEAR={grade_year}   # Set from env.txt or manually

Step 1.2: Verify Data Availability

Confirm the following exist:

• Student answer PDFs: data/masked_data/ict/2025/student_answers/student1.pdf … student10.pdf
• Rubric: data/masked_data/ict/2025/rubrics/rubrics.pdf
• Question papers: data/masked_data/ict/2025/question/paper_part1.pdf, paper_part2.pdf
• Reference data: data/reference/ict/2024/reference_mapping.json
• Reference rubrics: data/reference/ict/2024/rubrics/rubrics_part1.pdf … rubrics_part17.pdf

Read BATCH_SIZE from env.txt (default: 5) for parallel sub-agent control.

Phase 2: Rubric Extraction

Step 2.1: Extract Rubric Content

Read ALL rubric PDFs from the masked data directory. Handle these patterns:

1. Single rubric file: rubrics/rubrics.pdf — extract directly
2. Split rubric parts: rubrics/rubrics_part1.pdf … rubrics_partN.pdf — read ALL parts in numerical order, concatenate content
3. Combined rubric + question: rubric_and_question/paper_part1.pdf … paper_partN.pdf — read ALL parts in order; both rubric criteria and question text are interleaved in these files
4. Level descriptors: If rubrics/level_descriptors.pdf exists, ALWAYS read it — it contains official level-band boundaries critical for level assignment

Use PyMuPDF (fitz) for text extraction. Fall back to VLM only if text extraction yields empty or garbled content.

Chain to extraction skill: perform this phase through the extraction skill's YAML-driven pipeline (target=rubrics; choose request_mode by document length and continuity needs). Force Gemini routing (model_routes -> gemini) with env-backed provider config only (GEMINI_BASE_URL, VERTEX_API_KEY).

If a rubric page contains grading-relevant visual structure that plain text may flatten or lose — for example diagrams, screenshots, tables whose layout matters, or annotated figures — make sure the extraction output marks this explicitly (for example with needs_visual_reference=true and a concise visual_reference_reason). After reading the API result, the agent may then export the corresponding whole-page PNGs under rubric/{grade_year}/page_images/rubrics/. Reuse existing PNGs on reruns whenever possible.

Note on rubric extraction granularity: For rubric/criteria PDFs (which are typically typeset/printed), extracting multiple pages at a time is acceptable. However, for extremely long rubric files (e.g., 17+ split parts), consider extracting in manageable batches of a few pages each to avoid truncation or quality degradation.

Step 2.2: Extract Question Paper

If the question paper is separate from rubrics:

• Single file: question/paper.pdf — extract directly
• Split parts: question/paper_part1.pdf … paper_partN.pdf — read in order
• Files with non-standard names (e.g., Chinese filenames) — read all PDFs in question/

Chain to extraction skill: perform this phase through the extraction skill (target=question) and persist structured extraction artifacts before drafting grading_guide.md. Force Gemini routing for this chain.

Apply the same rule to question pages: if the extraction result indicates diagrams, UI mockups, charts, geometry-like figures, or other layout-sensitive content, export the relevant full-page PNGs under rubric/{grade_year}/page_images/question/ after the API response is reviewed.

Step 2.3: Build Grading Guide

Create rubric/{grade_year}/grading_guide.md with:

• All questions/tasks listed with their question numbers
• Mark allocation per question and sub-question (e.g., Q1a: 3 marks, Q1b: 4 marks)
• Every individual marking point (採分點) for each question — list each criterion that earns marks as its own entry. Do not collapse multiple marking points into a single bullet; every scorable criterion must appear explicitly
• Partial mark breakdown: For questions with staged or banded marking, document every stage explicitly — what the student must demonstrate to earn 0, 1, 2, … up to the maximum marks at each step. For example: "1 mark: identifies the correct concept; 2 marks: provides a valid example; 3 marks: explains the mechanism with correct terminology"
• Marking criteria: what earns each mark or mark band — include exemplar acceptable answers and common unacceptable answers where the rubric provides them
• Level-band boundaries (if present in rubric or level descriptors)
• Any special instructions (e.g., alternative acceptable answers, "accept any reasonable answer", units required, method marks vs. answer marks)

Phase 3: Reference Calibration

Step 3.1: Read Reference Data

Read data/reference/{subject}/{ref_year}/reference_mapping.json to get the mapping of student IDs to known levels (1–5).

Format:

{
  "mappings": [
    {"student_id": 1, "filename": "level3_student1.pdf", "level": 3, ...},
    ...
  ]
}

Note: Reference files keep their original level-based names (level{L}_student{M}.pdf). The filename field in the mapping matches the actual file on disk. The level is also encoded in the filename, making it easy for agents to identify which level each student belongs to without parsing the JSON.

Step 3.2: Extract Reference Student Answers

Extract text from ALL reference student answer PDFs in data/reference/{subject}/{ref_year}/student_answers/.

Chain to extraction skill: use extraction pipeline in student-PDF mode (pdf_dir + optional students) to produce structured extraction outputs that feed reference calibration. Force Gemini routing for this chain.

Step 3.3: Read Reference Rubrics

Read rubric PDFs from the REFERENCE year directory (structure may differ from the grading year). Apply the same extraction rules as Phase 2.

Step 3.4: Grade Reference Students (Iterative Rubric Refinement)

Grade ALL reference students using the same rubric and grading guide (from Phase 2). For each reference student, apply the per-question mark allocation to produce a total raw score and percentage — exactly as you would for a masked student. Record these scores alongside their known levels from reference_mapping.json.

This step is critical: the resulting scores provide the empirical score-to-level mapping that replaces hard-coded boundaries. Launch sub-agents for reference students following the same rules as Phase 5 (one sub-agent per student, same model restriction).

Save reference grading results to rubric/{ref_year}/reference_scores.json:

{
  "reference_year": "{ref_year}",
  "scores": [
    {"student_id": 1, "level": 3, "total_raw_score": 42, "total_max_score": 80, "percentage": 52.5},
    {"student_id": 2, "level": 5, "total_raw_score": 72, "total_max_score": 80, "percentage": 90.0}
  ],
  "level_score_ranges": {
    "1": {"min_pct": 12.5, "max_pct": 22.0, "mean_pct": 17.3, "count": 2},
    "2": {"min_pct": 28.0, "max_pct": 38.5, "mean_pct": 33.3, "count": 2},
    "3": {"min_pct": 45.0, "max_pct": 55.0, "mean_pct": 50.0, "count": 2},
    "4": {"min_pct": 62.0, "max_pct": 74.0, "mean_pct": 68.0, "count": 2},
    "5": {"min_pct": 82.0, "max_pct": 95.0, "mean_pct": 88.5, "count": 2}
  }
}

Step 3.4a: Reference Data Analysis and Iterative Rubric Refinement

The rubric must not be finalised from the official PDF alone. The agent must learn from the reference student data to derive an effective grading rubric — one that actually discriminates between student levels as observed in practice. This is an iterative loop that continues until the rubric reliably separates levels.

This is especially important for Visual Arts and Chinese / English Writing, where the official rubric provides only broad bands and subjective judgment is required. For these subjects, the reference data analysis must be particularly thorough: document what distinguishes each level in observable, concrete terms.

All analytical outputs from this step are saved to rubric/reference_data_analysis/.

Iterative loop:

1. Grade all reference students with the current rubric draft (Step 3.4 above).
2. Analyse grading results versus known levels:
   • Are scores meaningfully different across levels?
   • Which rubric criteria caused confusion or over/under-scoring?
   • What did good (Level 4–5) answers do that poor (Level 1–2) answers did not?
   • Are there patterns in how students at each level approach questions differently?
3. Write rubric/reference_data_analysis/per_level_analysis.md documenting observed patterns for each level 1–5:
   • Typical strengths at this level
   • Common errors and misconceptions
   • What kinds of mistakes are acceptable vs. unacceptable
   • Concrete examples from reference student answers
   • What distinguishes this level from adjacent levels
4. Run the discrimination check (see criteria below). If the check fails, or if qualitative analysis reveals the rubric is inadequate: a. Document gaps in rubric/reference_data_analysis/rubric_gaps.md — gaps or ambiguities found in the official rubric when applied to reference students (e.g., criteria that fail to distinguish adjacent levels, missing guidance for common student behaviours, mark schemes that reward or penalise inconsistently) b. Derive refinements in rubric/reference_data_analysis/rubric_refinements.md — supplementary rubric interpretations and practical guidance that resolve the gaps above (e.g., clarifying what “adequate discussion” means at each level, adding concrete examples, specifying how to handle partial credit) c. Revise rubric/{grade_year}/grading_guide.md to incorporate these refinements — the grading guide is not simply extracted from the official rubric PDF; it must incorporate insights from the reference data analysis d. Re-grade affected reference students with the improved rubric e. Repeat from step 2 until the discrimination check passes
5. The final rubric/{grade_year}/grading_guide.md must incorporate insights from rubric/reference_data_analysis/ — it is not simply a transcription of the official rubric PDF.

Discrimination check criteria — verify ALL of the following:

• Each level has a meaningfully distinct score range (minimal overlap between adjacent levels).
• At least 4 of the 5 levels are represented by at least one reference student.
• The score spread across levels is reasonable (i.e., students are not all clustered within a narrow band).

Only proceed to Phase 4 (actual student grading) once the discrimination check passes. Document the outcome in rubric/{grade_year}/reference_calibration.md.

Temp file organization: All intermediate artifacts from calibration — including reference student grading outputs, draft rubrics, and intermediate score calculations — must be saved under rubric/{ref_year}/calibration/ (a dedicated subdirectory, e.g. rubric/2024/calibration/). This separates reference-year working files from the current grading year. Final reference artifacts (reference_scores.json) are saved under rubric/{ref_year}/. Final grading-year artifacts (grading_guide.md, reference_calibration.md, level_division.json) are saved under rubric/{grade_year}/.

Step 3.5: Compute Score-to-Level Boundaries

Using the reference scores from Step 3.4, compute level boundaries to guide level assignment. One common approach is finding midpoints between adjacent levels' score ranges, but agents may use any reasonable method.

These boundaries are guidelines derived from reference data, not hard restrictions. They serve as a starting point for level assignment in Phase 7, where agents may refine them using additional information from the rubric and level descriptors.

Step 3.6: Build Reference Calibration Document

Create rubric/{grade_year}/reference_calibration.md with TWO sections:

Section 1 — Qualitative Calibration: For each level (1–5), summarise what rubric criteria reference students at that level demonstrated:

• What a Level 1 answer looks like (key deficiencies)
• What a Level 2 answer looks like
• What a Level 3 answer looks like
• What a Level 4 answer looks like
• What a Level 5 answer looks like

Focus on the main marking dimensions for the subject. Include specific examples of how reference students at each level addressed key questions.

Section 2 — Quantitative Score Ranges: Include the empirical score-to-level mapping from Step 3.4:

• Level 1: score range X%–Y% (mean Z%), N reference students
• Level 2: score range X%–Y% (mean Z%), N reference students
• Level 3: score range X%–Y% (mean Z%), N reference students
• Level 4: score range X%–Y% (mean Z%), N reference students
• Level 5: score range X%–Y% (mean Z%), N reference students
• Computed level boundaries (from Step 3.5)

This quantitative mapping provides useful guidance for level assignment in Phases 5 and 7. Sub-agents and level division may use these ranges alongside rubric information and level descriptors to determine levels from scores.

Phase 4: Student Answer Extraction

Step 4.1: Extract Text from Student PDFs

CRITICAL: Student answer extraction MUST be done 1 page at a time to ensure extraction quality. Do NOT batch multiple pages into a single extraction call.

For each student in data/masked_data/{subject}/{grade_year}/student_answers/:

1. Use PyMuPDF to extract text from the PDF, processing ONE PAGE AT A TIME
2. For each page, extract text independently, then concatenate all pages in order
3. If extraction yields empty/garbled text on any page, fall back to VLM image-to-text for that specific page only (1 page per VLM call for accuracy)
4. Save the concatenated extracted text to extracted/{grade_year}/students/student{N}.txt
5. If a page contains meaningful visual structure that should survive extraction (e.g., diagrams, drawn annotations, tables with layout meaning, screenshots, labeled arrows, formula layout, or mixed text+graphics), also preserve the whole page PNG under extracted/{grade_year}/students/page_images/student{N}/ — but only after the extraction result signals this need.

Chain to extraction skill: this phase must be implemented via the extraction skill with request_mode=page_by_page (or page_by_page_with_prev only when boundary context is necessary) to enforce one-page extraction quality requirements, and route via Gemini only.

Adjust the extraction prompt/output contract so the API returns visual-preservation hints (for example needs_visual_reference and visual_reference_reason). After extraction, inspect those returned rows; if any item on a page is flagged, export that whole page to PNG with PyMuPDF at 150 DPI. Do not crop.

Note: This 1-page-at-a-time rule applies to student answer PDFs. For rubric/criteria PDFs (Phase 2), multi-page extraction is acceptable since those are typeset documents with cleaner formatting.

Step 4.2: Verify Extraction Completeness

Confirm all student text files exist and have non-trivial content. Flag any students whose extraction may be incomplete. If a student has visual-page artifacts, confirm the referenced PNG files also exist.

Step 4.3: Write and Run Extraction Validation Script

Write a script scripts/validate_extraction.py that:

• Iterates over all extracted/{grade_year}/students/student{N}.txt files
• Checks each file: exists, non-empty, ≥ 50 characters
• If extracted/{grade_year}/students/page_images/student{N}/ exists, verify it contains at least one .png file and report the page-image count
• Prints a summary table: student ID, file size, pass/fail
• Exits with a non-zero exit code if ANY student file fails

Run the script and confirm all students pass before proceeding to Phase 5. If any fail, re-run extraction for those students.

Phase 5: Per-Student Grading

Step 5.1: Launch Sub-Agents

For each student, launch a dedicated sub-agent that:

1. Reads rubric/{grade_year}/grading_guide.md

2. Reads rubric/{grade_year}/reference_calibration.md (both qualitative and quantitative sections — including score-to-level ranges)

   Anti-bias note: The reference_calibration.md score ranges are empirical data from one prior year. Do NOT treat them as fixed cutoffs. The preliminary_level should reflect the quality of the student's work against rubric criteria. If the student's score falls slightly outside a reference range but clearly matches a level's qualitative description, assign that level and note the reasoning.

3. Reads the student's extracted text (extracted/{grade_year}/students/student{N}.txt)

4. Grades EVERY question/sub-question against the rubric — focus on accurate per-question mark allocation first, without constraining scores to fit a predetermined level

5. Computes total score and percentage from per-question marks

6. Derives preliminary_level using the reference score ranges and any other available level information (rubric, level descriptors, reference calibration)

7. Outputs a JSON file to output/{grade_year}/student{N}.json

IMPORTANT: Sub-agents must NOT use hard score restrictions to constrain their grading. The correct approach is: grade each question purely on rubric merit → sum the scores → map the total to a level using reference score ranges. Do NOT adjust per-question marks to force a particular level outcome.

Control parallelism with BATCH_SIZE — launch up to BATCH_SIZE sub-agents at a time.

Step 5.2: Per-Student Output Schema

Each output/{grade_year}/student{N}.json MUST follow this schema:

{
  "student_id": N,
  "questions": [
    {
      "question_id": "Q1a",
      "max_marks": 4,
      "awarded_marks": 3.0,
      "reasoning": "Step-by-step analysis citing rubric criteria: (1) [Rubric criterion 1] — student answer satisfies this because [specific part of answer]; (2) [Rubric criterion 2] — student answer partially satisfies this: [what was present] vs. [what was missing]; (3) [Rubric criterion 3] — student answer does not satisfy this because [reason]. Therefore 3/4 marks awarded.",
      "evidence": "Direct quote from student answer for each criterion: '[verbatim student text satisfying criterion 1]'; '[verbatim student text partially satisfying criterion 2]'. Missing element: '[what the rubric requires but the student did not provide]'."
    }
  ],
  "total_raw_score": 45,
  "total_max_score": 60,
  "percentage": 75.0,
  "preliminary_level": 4,
  "level_reasoning": "Based on reference calibration, this student's performance aligns with Level 4 characteristics: ..."
}

Fields:

• student_id: Integer student number
• questions: Array of per-question grading objects
  • question_id: String identifier (e.g., "Q1a", "Q2bii")
  • max_marks: Maximum marks for this question
  • awarded_marks: Marks awarded (float, can be 0.5 increments if rubric allows)
  • reasoning: Step-by-step analysis of mark allocation, always grounded in the rubric first. For each marking point in the grading guide: (a) state the rubric criterion being assessed, (b) explain whether and how the student's answer satisfies it, (c) for partial marks, specify exactly what was present and what was absent. General subject-knowledge conventions may supplement the rubric only where the rubric is genuinely silent — and this must be stated explicitly (e.g., "Rubric does not specify, but convention requires…").
  • evidence: Direct verbatim quote(s) from the student's answer that demonstrate satisfaction (or non-satisfaction) of each rubric criterion. Do not paraphrase — quote the student answer exactly. For partial-mark cases, quote both the satisfying portion and identify the missing element.
• total_raw_score: Sum of all awarded_marks
• total_max_score: Sum of all max_marks
• percentage: (total_raw_score / total_max_score) × 100
• preliminary_level: Level (1–5) derived using the reference score ranges and available level information. This is a best-fit estimate
• level_reasoning: Explanation for the level assignment, referencing the score range it falls into (e.g., "Score 52.5% falls within Level 3 reference range 45.0%–55.0%")

Grading methodology — sub-agents MUST follow this sequence:

1. Read the grading guide first. For the question being graded, identify every individual marking point (採分點) and the marks associated with each stage.
2. Read the student's answer. Locate the portion of the extracted text that responds to this question.
3. Assess each marking point in turn. For each criterion in the grading guide: does the student's answer meet it? Fully, partially, or not at all?
4. Assign marks per criterion. Award marks strictly based on rubric merit — do not adjust per-question marks to force a predetermined level outcome.
5. Justify every mark decision. The reasoning field must reference the specific rubric criterion and the specific student text (verbatim in evidence) that supports or fails each mark.
6. The rubric is the primary authority. Only apply general subject-knowledge conventions where the rubric is genuinely silent, and flag it when you do.

Step 5.3: Validate Sub-Agent Output

After each sub-agent completes:

1. Verify the JSON file exists and is valid
2. Check that total_raw_score equals sum of awarded_marks
3. Check that no awarded_marks exceeds max_marks
4. Check that preliminary_level is between 1 and 5
5. Re-run failed sub-agents if necessary

Step 5.4: Write and Run Grading Output Validation Script

Write a script scripts/validate_grading_output.py that:

• Iterates over all output/{grade_year}/student{N}.json files
• For each file: checks existence, valid JSON, required keys (student_id, questions, total_raw_score, total_max_score, percentage, preliminary_level), value ranges (preliminary_level ∈ [1,5], percentage ∈ [0,100]), and that total_raw_score equals the sum of all awarded_marks
• Prints a per-student pass/fail table with the specific failure reason for any failures
• Exits with a non-zero exit code if ANY student fails

Run the script. Re-run the sub-agent for any failing students. Re-run the script until all pass.

Phase 6: Score Compilation

Step 6.1: Aggregate All Student JSONs

Read all output/{grade_year}/student{N}.json files and compile into output/{grade_year}/final_scores.json:

{
  "subject": "{subject}",
  "year": "{grade_year}",
  "total_students": N,
  "max_possible_score": 60,
  "students": [
    {
      "student_id": 1,
      "total_raw_score": 45,
      "total_max_score": 60,
      "percentage": 75.0,
      "preliminary_level": 4
    }
  ],
  "statistics": {
    "mean_score": 42.5,
    "median_score": 43.0,
    "std_dev": 8.2,
    "min_score": 20,
    "max_score": 58,
    "mean_percentage": 70.8
  }
}

Phase 7: Level Division

Step 7.1: Determine Level Boundaries

Agents may use the extracted reference data (from Phase 3) to inform level division. The following sources are available — agents should consider all relevant information and exercise judgment:

• Reference score ranges (rubric/{grade_year}/reference_scores.json) — the empirical score ranges from grading reference students in Phase 3. These provide an empirical basis for mapping scores to levels.
• Official level descriptors (from level_descriptors.pdf if available) — these describe the qualitative characteristics expected at each level
• Criteria-based boundaries derived from the grading guide's mark scheme
• Reference calibration document (rubric/{grade_year}/reference_calibration.md) — contains both qualitative descriptions and quantitative score ranges per level

There are no hard restrictions on how level boundaries must be computed. Agents should use the available reference data, rubric information, and level descriptors to define reasonable level boundaries that reflect the subject's scoring patterns.

⚠️ Reference Year Bias Prevention:

The reference year's score ranges are a starting point for calibration only, not fixed thresholds to be carried forward. Before finalising level boundaries, the agent must:

1. After compiling all current-year student scores (Phase 6), compute the empirical score distribution of the current cohort (mean, median, spread, approximate percentiles).
2. Compare this distribution to the reference year's level score ranges. Identify any significant cohort-level shift (e.g., current-year scores are systematically 10–20% lower or higher than reference year scores at the same qualitative level of work).
3. If a significant shift exists, proportionally adjust the level boundaries to account for it. The goal is that level division reflects the actual quality distribution of the current cohort as judged against rubric criteria — not a fixed mapping derived exclusively from one prior year's data.
4. Mandatory anti-bias rule: If the reference year shows Level 5 students scoring 70–100%, but this year's highest-performing students score around 60%, do NOT conclude that no Level 5 students exist. Instead, determine whether 60% represents genuinely Level 5 work under the rubric, and if so, adjust the Level 5 boundary down accordingly.
5. The final level_division.json must include an "adjustment_notes" field. It must either explain any deviation from raw reference boundaries and the reasoning behind the adjustment, or explicitly state "No adjustment; current cohort distribution aligns with reference year ranges."

Step 7.2: Assign Final Levels

For each student, map their total score to a level (1–5) using the reference-derived boundaries from Step 7.1. Save to rubric/{grade_year}/level_division.json:

{
  "method": "reference_score_based",
  "reference_year": "{ref_year}",
  "boundaries": {
    "level_1": {"min_percentage": 0, "max_percentage": 27.5},
    "level_2": {"min_percentage": 27.5, "max_percentage": 41.75},
    "level_3": {"min_percentage": 41.75, "max_percentage": 60.0},
    "level_4": {"min_percentage": 60.0, "max_percentage": 78.0},
    "level_5": {"min_percentage": 78.0, "max_percentage": 100}
  },
  "reference_score_ranges": {
    "level_1": {"min_pct": 12.5, "max_pct": 22.0, "mean_pct": 17.3},
    "level_2": {"min_pct": 28.0, "max_pct": 38.5, "mean_pct": 33.3},
    "level_3": {"min_pct": 45.0, "max_pct": 55.0, "mean_pct": 50.0},
    "level_4": {"min_pct": 62.0, "max_pct": 74.0, "mean_pct": 68.0},
    "level_5": {"min_pct": 82.0, "max_pct": 95.0, "mean_pct": 88.5}
  },
  "adjustment_notes": "No adjustment; current cohort distribution aligns with reference year ranges.",
  "students": [
    {
      "student_id": 1,
      "total_raw_score": 45,
      "percentage": 75.0,
      "preliminary_level": 4,
      "final_level": 4,
      "level_reasoning": "Score 75.0% falls within Level 4 reference range (62.0%–74.0%), closest to Level 4 mean (68.0%)"
    }
  ]
}

Note: The boundaries values above are examples. Agents should derive appropriate boundaries from the available reference data and rubric information.

Step 7.3: Cross-Validate with Reference

Compare the level distribution of graded students against the reference year's distribution. Flag any significant discrepancies (e.g., if the reference year had students at all 5 levels but the graded year has none at Level 1).

Subject-Specific Data Notes

ICT (Information and Communication Technology)

• Masked data (2025): data/masked_data/ict/2025/
  • Rubrics: rubrics/rubrics.pdf (single file)
  • Question: question/paper_part1.pdf, question/paper_part2.pdf
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2024): data/reference/ict/2024/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part17.pdf (17 split parts)
  • Question: question/paper.pdf
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Music

• Masked data (2025): data/masked_data/music/2025/
  • Rubric AND question combined: rubric_and_question/paper_part1.pdf … paper_part24.pdf
  • No separate rubrics/ or question/ directories
  • Students: 5 (student1.pdf – student5.pdf)
• Reference data (2024): data/reference/music/2024/
  • Rubric AND question combined: rubric_and_question/paper_part1.pdf … paper_part21.pdf
  • Students: 21 labeled, level-based names (level1_student1.pdf … level5_student8.pdf)

Ethics and Religious Studies

• Masked data (2025): data/masked_data/religion/2025/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part7.pdf (7 split parts)
  • Question: question/paper.pdf
  • Students: 15 (student1.pdf – student15.pdf)
• Reference data (2024): data/reference/religion/2024/
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part7.pdf (7 split parts)
  • Question: question/paper.pdf
  • Students: 14 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Tourism and Hospitality Studies

• Masked data (2025): data/masked_data/tourism/2025/
  • Rubrics: rubrics/rubrics.pdf (main mark scheme) AND rubrics/level_descriptors.pdf (official level band descriptors)
  • Always read level_descriptors.pdf — it contains the official level boundaries
  • Question: question/paper.pdf
  • Students: 10 (student1.pdf – student10.pdf)
• Reference data (2024): data/reference/tourism/2024/
  • Rubrics: rubrics/rubrics.pdf AND rubrics/level_descriptors.pdf
  • Question: question/paper.pdf
  • Students: 10 labeled, level-based names (level1_student1.pdf … level5_student2.pdf)

Visual Arts

• Masked data (2020): data/masked_data/visual-arts/2020/
  • Rubrics: rubrics/rubrics.pdf (single file)
  • Question: question/ directory — files may have Chinese filenames (e.g., 2020_視藝_P1QAB.pdf, 2020_視藝_P2QAB.pdf); read ALL PDFs in this directory
  • Students: 18 (student1.pdf – student18.pdf)
• Reference data (2025): data/reference/visual-arts/2025/
  • NOTE: Reference year (2025) is NEWER than grading year (2020)
  • Rubrics: rubrics/rubrics_part1.pdf … rubrics_part5.pdf (5 split parts)
  • Question: question/paper_part1.pdf … paper_part5.pdf (5 split parts)
  • Students: 24 labeled, level-based names (level1_student1.pdf … level5_student6.pdf)
  • Use reference data purely for rubric calibration; do not assume rubric changes between years
• ⚠️ Art creation grading: Art-making/creation components MUST be graded via VLM directly from student PDF page images — NOT from extracted text. Visual elements (drawings, compositions, color, spatial arrangement) cannot be captured by text extraction. Only written/critical appreciation portions should use extracted text.

Error Handling

• Missing PDFs: Fail early with a clear error listing which files are missing
• Empty text extraction: Flag the student and retry extraction via Gemini (gemini-3.1-flash-lite-preview, 1 page per call)
• Invalid sub-agent output: Re-run the sub-agent with clearer instructions
• Score validation failures: Log warnings but do not silently adjust scores
• Missing reference data: Proceed with grading but note that calibration is incomplete

Phase 8: Quality Assurance

⚠️ DO NOT EXECUTE — This phase is disabled for now. Skip Phase 8 entirely. Do not run any steps in this phase.

Step 8.1: Score Distribution Review

After level division, review the overall score distribution:

1. Check that the mean/median scores are reasonable for the subject
2. Identify any outliers (students with extremely high or low scores)
3. Verify the level distribution is plausible (not all students in one level)

Step 8.2: Spot-Check Sample Students

Randomly select 2–3 students and manually verify their grading:

1. Re-read the student's extracted text
2. Check that per-question marks align with the rubric criteria
3. Verify the total score matches the sum of per-question marks
4. Confirm the assigned level is consistent with reference calibration

Step 8.3: Cross-Student Consistency

Compare students at the same level:

1. Students assigned to the same level should have comparable scores
2. Flag any cases where two students have very different scores but the same level
3. Check that level boundaries produce a reasonable ordering of students

Step 8.4: Final Output Verification

Confirm all output files are complete and valid:

1. All student{N}.json files exist with valid JSON
2. final_scores.json contains all students with correct statistics
3. rubric/{grade_year}/level_division.json has valid boundaries and all student assignments
4. No missing or corrupted files

Checklist

• Year set via GRADING_YEAR env var
• Dependencies installed (uv sync)
• model-service skill run at least once to confirm Gemini (gemini-3.1-flash-lite-preview) route availability from env.txt
• Student PDFs confirmed in masked data directory
• Rubric PDFs read and grading guide built
• Reference students graded and score ranges computed (reference_scores.json)
• Rubric discrimination check passed (Step 3.4a)
• Reference calibration document built (qualitative + quantitative)
• extraction skill used in Phase 2/3/4 extraction tasks with Gemini (gemini-3.1-flash-lite-preview) routing
• All student texts extracted (1 page at a time for student answers)
• Per-student grading complete (one sub-agent per student)
• Score compilation done (final_scores.json)
• Level division done (rubric/{grade_year}/level_division.json)
• Quality assurance completed (score distribution, spot-checks, consistency)