Lingliang Evaluation

The full evaluation workflow follows the unified skill adapted for this subject. The complete content is reproduced below for reference.

Lingliang Evaluation — Unified Skill (Adapted)

Overview

This skill evaluates the accuracy of AI-generated grades for the Lingliang primary school math exam by comparing them against ground truth raw scores (total out of 100). It computes statistical metrics including correlation coefficients and error measures, generates visualization figures, and produces comprehensive DOCX evaluation reports.

Prerequisites: This skill requires completed grading output from the lingliang grading skill. Grading output is accessed from ../grading_workspace/output/lingliang/. Ground truth is local at data/groundtruth/lingliang/groundtruth_mapping.json.

• ../grading_workspace/output/lingliang/student_XX.json — Per-student grading results
• ../grading_workspace/output/lingliang/final_scores.json — Aggregated scores with statistics
• Ground truth: data/groundtruth/lingliang/groundtruth_mapping.json

CRITICAL RULES

1. Ground truth is raw scores (total_score out of 100), NOT ordinal levels. Use both rank-based (Spearman) and absolute-error (MAE, RMSE) metrics.

2. Primary metrics include both rank-order (Spearman ρ) and error metrics (Pearson r, MAE, RMSE, ±5-point accuracy). These together measure whether the AI grading produces scores that correlate with and closely approximate ground truth scores.

3. No level division data — evaluate directly against raw score ground truth. There are no DSE levels (1–5) in this subject.

4. Model restrictions:

   • Sub-agents: SAME model as the main agent. All sub-agents MUST use the same model that the main (orchestrating) agent is running on. Do NOT pass a model parameter override to sub-agents — let them inherit the main agent's model.
   • VLM tasks: Kimi K2.5 ONLY. When VLM (vision-language model) API calls are needed — e.g., text extraction from scanned PDFs where PyMuPDF fails, or art creation grading requiring visual inspection — use Kimi K2.5 (kimi-k2.5) exclusively. NO OTHER VLM MODELS ARE ALLOWED.

5. No LLM/VLM API calls for generating analysis or report text. See WARNING.md. All commentary and analysis must be generated by sub-agents.

6. Paths are fixed — no year variable needed. All paths use lingliang directly without year nesting (e.g., output/lingliang/, data/groundtruth/lingliang/).

7. All report and commentary text must be written in Traditional Chinese (繁體中文). This applies to all narrative, feedback, analysis, and section content in generated DOCX reports. English is only permitted for: variable names, file paths, technical identifiers, chart axis labels, and JSON field names.

Workspace Layout

lingliang-grading-workspaces/evaluation_workspace/
├── .github/skills/lingliang-evaluation/SKILL.md
├── START.md
├── WARNING.md
├── env.txt
├── env.txt.example
├── pyproject.toml
├── uv.lock
├── start.sh
├── data/                           # Local data (pre-copied into workspace)
│   └── groundtruth/lingliang/groundtruth_mapping.json
├── evaluation/lingliang/           # Evaluation output (this skill generates these)
│   ├── metrics.json                # Machine-readable metrics
│   ├── rubric_quality_benchmark.json  # Agent-facing rubric quality benchmark
│   ├── grading_quality_audit.json  # Per-student grading quality audit
│   ├── grading_consistency_audit.json # Cross-student same-question consistency audit
│   ├── grading_quality_report.md   # Diagnostic markdown summary
│   ├── eval_report.docx            # Class-level evaluation report
│   ├── figures/                    # Evaluation charts (PNG)
│   │   ├── score_scatter.png
│   │   ├── error_distribution.png
│   │   └── per_student_table.png
│   └── students/                   # Individual student evaluation reports
│       └── student_XX.docx
└── scripts/                        # Python helper scripts
    ├── config.py                   # Path configuration
    ├── validate_extraction.py
    ├── validate_grading_output.py
    └── validate_reports.py

JSON Schema Reference

Grading Output (from grading skill)

Per-student JSON (output/lingliang/student_XX.json):

{
  "student_id": 1,
  "questions": [...],
  "total_raw_score": 77,
  "total_max_score": 100,
  "percentage": 77.0
}

Final scores (output/lingliang/final_scores.json):

{
  "subject": "lingliang",
  "total_students": 20,
  "students": [
    {"student_id": 1, "total_raw_score": 77, "percentage": 77.0}
  ]
}

Ground Truth

{
  "mappings": [
    {
      "student_id": 1,
      "student_label": "student_01",
      "student_name": "陳喜楓",
      "filename": "student_01.pdf",
      "total_score": 77,
      "max_score": 100,
      "score_percentage": 77.0
    }
  ]
}

Phase 1: Environment Setup

Step 1.1: Verify Prerequisites

source env.txt
uv sync

Verify all required inputs exist:

1. Grading output: ../grading_workspace/output/lingliang/final_scores.json
2. Per-student JSONs: ../grading_workspace/output/lingliang/student_01.json etc. (20 grading students)
3. Ground truth: data/groundtruth/lingliang/groundtruth_mapping.json

Step 1.2: Load and Parse Data

1. Load ground truth mapping — extract student_id → total_score, score_percentage pairs
2. Load final scores — extract student_id → total_raw_score, percentage
3. Match students by ID between grading output and ground truth

Phase 2: Compute Evaluation Metrics

Step 2.1: Core Metrics

Compute the following metrics:

Step 2.2: Save Metrics

Save to evaluation/lingliang/metrics.json:

{
  "subject": "lingliang",
  "total_students": 20,
  "metrics": {
    "spearman_rho": 0.85,
    "spearman_p_value": 0.001,
    "pearson_r": 0.88,
    "pearson_p_value": 0.0001,
    "mae": 4.2,
    "rmse": 5.8,
    "within_5_points_pct": 65.0,
    "within_10_points_pct": 90.0,
    "max_error": 15
  },
  "per_student": [
    {
      "student_id": 1,
      "gt_score": 77,
      "gt_percentage": 77.0,
      "predicted_score": 74,
      "predicted_percentage": 74.0,
      "score_diff": -3
    }
  ]
}

Phase 5: Final Evaluation — Grading Quality Judge (LLM-as-Judge)

ℹ️ This phase runs after Phase 2. It acts as an independent judge to assess whether the grading output (output/lingliang/student_XX.json) contains correct marks, well-grounded reasoning, and accurate evidence — verified against the actual rubric and student answers.

⛔ HARD RULE: NO Heuristic / Keyword-Matching Scripts

You MUST NOT write Python scripts that use keyword matching, regex patterns, string length checks, or any other heuristic rules to evaluate grading quality. Such approaches cannot determine whether reasoning is actually correct or whether evidence actually matches the student's answer — they can only check if certain words appear in text, which is fundamentally insufficient and produces misleading quality scores.

All quality evaluation in this phase MUST be performed by sub-agents that read the rubric, read the student's actual answer, and judge whether the grading output is correct.

Step 5.1: Gather Required Inputs

For each student, collect these required inputs:

1. Grading guide (rubric): ../grading_workspace/rubric/lingliang/grading_guide.md — Contains model answers, marking criteria, and mark allocation per question
2. Student's extracted answer: ../grading_workspace/extracted/lingliang/students/student_XX.txt — The student's actual written responses (extracted text from PDF)
3. Grading output: ../grading_workspace/output/lingliang/student_XX.json — The AI-generated grading with awarded_marks, reasoning, and evidence per question

Also collect these optional diagnostic inputs if they exist:

4. Student page-image artifacts: ../grading_workspace/extracted/students/page_images/student_XX/ — Useful when layout/diagram information may have been lost in text extraction
5. Rubric/question page-image artifacts: ../grading_workspace/rubric/page_images/ — Useful when rubric/question interpretation may depend on figures or layout

Step 5.1a: Rubric Quality Benchmark (Agent-Facing Rubric Quality)

Before auditing student grading outputs, launch one or more sub-agents to evaluate the quality of the agent-facing rubric representation itself — primarily ../grading_workspace/rubric/lingliang/grading_guide.md, plus related rubric/question page-image artifacts when they are needed to interpret visual marking criteria.

This benchmark is about whether the rubric is safe and reliable for downstream agents to use, not whether the official rubric document exists.

The sub-agent MUST evaluate the rubric on these 3 dimensions, scoring each on a 0–2