Paper Data Processing

Key portability rules (enforced by the agent before every script run):

1. Scripts MUST read inputs from data/raw/ or data/processed/ and write outputs to data/processed/ or reports/.
2. Scripts MUST NOT contain absolute paths. Use Path(__file__).resolve().parent.parent to derive the workspace root, or accept paths via CLI arguments.
3. If the agent detects hardcoded absolute paths in a script, it MUST patch them to workspace-relative paths before execution.
4. The user provides exactly one thing: WoS .txt export files. Everything else is generated.

Pre-flight Interactive Inquiry (MANDATORY before first script run): The agent MUST ask the user:

1. “您的项目根目录在哪里？” — 用户确认后设置 PROJECT_ROOT 环境变量
2. “数据文件在哪里？格式是什么？ (.txt / .csv / .xlsx)”
   • .txt → 复制到 data/raw/，从 Stage A 开始
   • .csv / .xlsx → agent 读取前 5 行检测字段，根据字段完整度决定入口阶段：
     • 有 UT + 基础列 (TI/AB/PY) → 复制为 wos_merged.csv，跳过 Stage A，从 Stage B 开始
     • 有 UT + 国家/机构列 → 复制为 wos_cleaned.csv，跳过 A+B，从 Stage C 开始
     • 有 UT + 国家 + 作者列 → 复制到对应位置，跳过 A+B+C，直接 Stage D 质量检查
     • 无 UT 列 → 提示用户可能非 WoS 数据，尝试字段映射或要求用户确认 All scripts derive paths from ROOT = Path(os.environ.get("PROJECT_ROOT", Path(__file__).resolve().parent.parent)).

Scope

• In scope:
  • WoS txt parsing and canonical paper table generation.
  • Country and institution extraction/normalization.
  • Region extraction/normalization (EU27 merged to EU for regional analysis).
  • Author extraction and disambiguation.
  • Schema and quality checks before analysis handoff.
  • Patching scripts to use portable paths when hardcoded paths are detected.
• Out of scope:
  • BERTopic or any other thematic modeling.
  • Mobility trend interpretation and report writing.
  • Visualization-driven analysis conclusions.

Stage Workflow

1. Stage A — Parse raw WoS txt files from data/raw/ into a paper-level canonical CSV at data/processed/wos_merged.csv. SKIP if user provided CSV/Excel that already has UT + base columns.
2. Stage B — Enrich with country/institution/region fields → data/processed/wos_cleaned.csv. SKIP if input already has country/institution fields.
3. Stage C — Build author-level outputs → data/processed/paper_authors.csv, data/processed/authors_summary.csv. SKIP if input already has author fields.
4. Stage D — Run quality gates and emit handoff summary. Always runs.

Core Scripts

Before running any script, the agent MUST:

1. Read the script and check for hardcoded absolute paths.
2. If found, replace them with workspace-relative paths.
3. Confirm the script's input file exists.

Contract Outputs

All outputs in data/processed/:

Required Quality Gates

• Gate 1: data/processed/wos_merged.csv exists and is readable.
• Gate 2: UT column exists, non-empty, unique at paper level.
• Gate 3: Required columns in data/processed/wos_cleaned.csv including country/region collaboration fields.
• Gate 4: data/processed/paper_authors.csv exists and contains UT linkage.

Handoff Rule

Analysis can start only when all quality gates pass. The paper-processing-analysis-handoff skill is the mandatory next step.

Usage Pattern

• Full rerun: stages A → D.
• Partial rerun: changed stage + dependent downstream stages.
• Schema change in upstream stage forces downstream rerun.

New Machine Setup

1. Clone or copy the workspace folder.
2. python -m venv .venv && .venv/Scripts/activate (Windows) or source .venv/bin/activate (Linux/Mac).
3. pip install -r requirements.txt.
4. Place data files into data/raw/ (supports .txt, .csv, .xlsx).
5. Run stages — agent auto-detects format and skips completed stages. No path editing required.

References

• references/runbook.md
• references/contracts.md
• references/quality-gates.md
• references/troubleshooting.md
• checklists/release-checklist.md

Done Criteria

• All processing stages complete without unhandled errors.
• All required artifacts exist in data/processed/.
• Quality gates pass and handoff is explicitly marked ready.