Data Sourcer

Conflict & Security

Governance & Institutions

Economic

Resources & Commodities

Climate & Environment

Displacement & Migration

Sanctions & Geopolitics

═══════════════════════════════════════════════════

PIPELINE INTEGRATION: Automatic Data Source Routing

═══════════════════════════════════════════════════

When running inside the pipeline (R1b), the data-sourcer receives the case file as input, which contains:

• A.1: Required papers with paper hierarchy (Core / Supporting / Mentioned)
• A.3: Figure strategy with figure roles (Motivator, Evidence, Mechanism, etc.)
• A.5: Slide outline with frame types

The data-sourcer does NOT freestyle — it follows a 3-step routing logic.

Step 1: Figure Role → Data Tier (automatic)

Rule: If a figure is classified as Motivator or Context, the data-sourcer MUST attempt Tier 1 before accepting Tier 2. Document the attempt in the pipeline logbook.

Step 2: Topic Keywords → Data Domain → Source (automatic)

The data-sourcer extracts keywords from the case file (papers, lecture title, figure descriptions) and routes to sources:

Step 3: Access Pattern Selection

R vs Python: Use R when the figure will be rendered with ggplot2. Use Python when the figure uses matplotlib. The data-sourcer outputs download scripts in whichever language the data-visualizer will use.

Step 4: Output Format

The data-sourcer produces L{XX}_data_sources.md:

# Data Sources — L{XX}

## Figure-by-Figure Routing

| Fig# | Role | Topic keywords | Tier | Source | Access pattern | Script |
|------|------|---------------|------|--------|---------------|--------|
| fig1 | Motivator | commodity prices, coffee, oil | Tier 1 | WB Pink Sheet | Direct Excel | `fig1_download.py` |
| fig2 | Evidence | MSS coefficients, IV results | Tier 2 | MSS (2004) Table 3 | Manual entry | hardcoded in `fig2_plot.py` |
| fig5 | Evidence | DV coefficients | Tier 2 | DV (2013) Table 4 | Manual entry | hardcoded in `fig5_plot.py` |
| fig7 | Context | food prices, Arab Spring | Tier 1 | FAO bulk ZIP | Bulk download | `fig7_download.py` |

## Tier 1 Attempts Log
| Figure | Source tried | Result | Fallback |
|--------|-------------|--------|----------|
| fig1 | WB Pink Sheet Excel | SUCCESS | — |
| fig7 | FAO API | TIMEOUT | FAO bulk ZIP: SUCCESS |

How This Connects to the Syllabus

The orchestrator reads the syllabus (or course structure) during Phase A.1 and extracts:

1. Lecture topic → data domain
2. Required papers → specific datasets those papers used
3. Paper hierarchy (Core/Supporting/Mentioned) → how much data effort per paper

The data-sourcer does NOT need to read the syllabus directly. The case file is its interface — all routing information is already extracted by the orchestrator. This means the same data-sourcer works for any course: change the syllabus, the orchestrator produces a different case file, and the data-sourcer routes accordingly.

═══════════════════════════════════════════════════

MODE 1: SCAN — Auto-Match Data to Slide Content

═══════════════════════════════════════════════════

Read a .qmd lecture file and identify where empirical data could strengthen the content.

Steps

S1: Parse Lecture Content

1. Read the .qmd file completely
2. Extract: slide titles, source citations, mentioned countries/regions, mentioned variables, time periods
3. Identify: theoretical claims that could use empirical support, existing charts/figures, exercises

S2: Match Against Known Sources

For each identified data need, match against the Known Data Sources table:

Matching rules:

• Mentions of "GDP", "income", "growth" → World Bank WDI, Penn World Tables
• Mentions of "conflict", "civil war", "battle deaths" → UCDP, ACLED
• Mentions of "trade", "exports", "imports" → UN Comtrade, CEPII BACI
• Mentions of "democracy", "institutions", "governance" → V-Dem, Polity5, WGI
• Mentions of "oil", "minerals", "resources", "commodities" → WB Commodity Prices, USGS, BP Review
• Mentions of "climate", "rainfall", "temperature", "drought" → CRU TS, ERA5, SPEI
• Mentions of "refugees", "displacement", "migration" → UNHCR, IDMC
• Mentions of "sanctions" → GSDB, OFAC
• Mentions of specific countries → filter datasets to relevant country coverage
• Mentions of historical periods → check temporal coverage of matched sources

S3: Classify Data Opportunities

For each match, classify as:

1. Chart/Figure opportunity: Data that could become a visual on a slide

   • Time series: "GDP growth in conflict-affected countries 2000–2024"
   • Cross-section: "Democracy score vs. conflict incidence"
   • Map: "Geo-coded conflict events in Sub-Saharan Africa"

2. Exercise data: Data students could work with

   • "Download WDI data for 5 countries, compute growth rates, discuss"
   • "Replicate Table 1 of Miguel et al. (2004) using UCDP data"

3. Updated statistics: Fresher numbers for claims in slides

   • "Slide says '40% of civil wars restart within 10 years' — check UCDP for current figure"
   • "Update battle death counts to most recent UCDP release"

4. Trend illustration: Long-run patterns supporting lecture narrative

   • "Declining interstate war since 1945 (COW data)"
   • "Rising commodity dependence in Sub-Saharan Africa (WDI)"

S4: Generate Data Report

# Data Source Scan: [Lecture Filename]
**Date:** [YYYY-MM-DD]

## Slide-by-Slide Data Opportunities

### Slide: "[Title]" (slide N)
**Current content:** [Brief description]
**Data opportunity:** [What data could add]
**Recommended source:** [Source name + specific indicator]
**Type:** Chart / Exercise / Updated Stat / Trend
**Effort:** Quick (API call) / Medium (download + clean) / Large (merge multiple sources)

[Repeat for each opportunity]

## Summary
- Total data opportunities: N
- Quick wins (API-downloadable): N
- Recommended priority: [Top 3 to implement first]

## Download Commands
[Pre-written commands for top recommendations]

═══════════════════════════════════════════════════

MODE 2: SEARCH — Find Specific Data

═══════════════════════════════════════════════════

Search for a specific dataset, indicator, or data source.

Steps

1. Parse the user's query: what variable? what geography? what time period?
2. Check Known Data Sources table first
3. If not found: use WebSearch to find the dataset
4. Return: source name, URL, download method, format, coverage, citation

Output Format

# Data Search: [Query]

## Best Match
- **Source:** [Name]
- **URL:** [Link]
- **Coverage:** [Countries × Years]
- **Variables:** [Key indicators available]
- **Format:** [CSV/XLSX/API]
- **Citation:** [How to cite this data]
- **Download:** [Command or URL]

## Alternatives
[Other sources that partially match]

═══════════════════════════════════════════════════

MODE 3: DOWNLOAD — Retrieve Data

═══════════════════════════════════════════════════

Download specific datasets and organize them in the course folder.

Steps

D1: Identify Source and Method

Based on the source, use the appropriate download method:

World Bank WDI (via wbgapi or direct URL):

# pip install wbgapi
import wbgapi as wb
# GDP per capita for conflict-affected countries
data = wb.data.DataFrame('NY.GDP.PCAP.CD', economy=['SOM', 'SSD', 'COD', 'AFG', 'YEM'], time=range(2000, 2025))
data.to_csv('data/wdi_gdppc_conflict.csv')

World Bank (direct CSV):

# Download specific indicator via API
curl -o data/indicator.csv "https://api.worldbank.org/v2/country/all/indicator/NY.GDP.PCAP.CD?format=csv&date=2000:2024"

UCDP (REST API):

# Battle-related deaths
curl -o data/ucdp_brd.json "https://ucdpapi.pcr.uu.se/api/battledeaths/24.1?pagesize=1000"

UCDP (direct download):

curl -o data/ucdp_ged.csv "https://ucdp.uu.se/downloads/ged/ged241-csv.zip"
unzip data/ged241-csv.zip -d data/ucdp/

V-Dem:

# Download full dataset (large, ~500MB)
curl -o data/vdem.csv "https://v-dem.net/data/dataset-archive/"
# Or use specific indicators via API

World Bank Commodity Prices:

curl -o data/commodity_prices.xlsx "https://thedocs.worldbank.org/en/doc/5d903e848db1d1b83e0ec8f744e55570-0350012021/CMO-Historical-Data-Monthly.xlsx"

D2: Clean and Organize

1. Download raw data to Data/raw/
2. If needed: clean, subset to relevant countries/years, rename columns
3. Save processed version to Data/processed/
4. Generate a data dictionary: Data/README.md

D3: Folder Structure

Data/
├── raw/                          # Original downloads
│   ├── wdi_gdppc_2024.csv
│   ├── ucdp_ged_24.1.csv
│   └── vdem_v14.csv
├── processed/                    # Cleaned, subsetted
│   ├── L01_conflict_trends.csv
│   ├── L02_resources_conflict.csv
│   └── L03_shocks_panel.csv
├── scripts/                      # Download/cleaning scripts
│   ├── download_wdi.py
│   ├── download_ucdp.py
│   └── clean_merge.py
└── README.md                     # Data dictionary

D4: Generate Data Dictionary

# Course Data Dictionary

## L01: Foundations — Conflict Trends
**File:** `processed/L01_conflict_trends.csv`
**Sources:** UCDP/PRIO ACD v24.1, World Bank WDI
**Variables:**
- `country`: Country name
- `year`: Year (1946–2024)
- `conflict_active`: Binary indicator for active conflict
- `battle_deaths`: Best estimate of battle-related deaths
- `gdp_pc`: GDP per capita (constant 2015 USD)
**Coverage:** 195 countries, 1946–2024
**Citation:** Gleditsch et al. (2002); Sundberg & Melander (2013)

═══════════════════════════════════════════════════

TOPIC-SPECIFIC RECOMMENDATIONS

═══════════════════════════════════════════════════

Pre-computed recommendations for common lecture topics in the conflict/geoeconomics course:

L01: Foundations of Conflict Economics

• UCDP/PRIO ACD: Conflict onset/incidence trends since 1946
• World Bank WDI: GDP and development indicators for conflict countries
• COW: Long-run interstate war data (declining trend)
• Recommended chart: "Number of active conflicts by type, 1946–2024" (UCDP)

L02: Greed, Grievance, and State Capacity

• WB Commodity Prices: Oil/mineral price series (resource curse channel)
• V-Dem: State capacity indicators, ethnic power relations
• Fragile States Index: Composite fragility scores
• UCDP: Conflict in resource-rich vs. resource-poor countries
• Recommended chart: "Commodity prices vs. conflict onset in Africa" (WB + UCDP)

L03: Economic Shocks and Conflict

• CRU TS / SPEI: Rainfall and temperature anomalies for Africa
• WB Commodity Prices: Agricultural commodity prices (coffee, cocoa)
• UCDP GED: Geo-coded conflict events matched to weather stations
• WDI: Agricultural value added as share of GDP
• Recommended chart: "Rainfall deviation and conflict events in Sub-Saharan Africa" (CRU + UCDP GED)

L04–L06 (future): Resources, Displacement, Historical Conflict

• EITI: Extractive revenue by country
• UNHCR: Refugee flows and displacement
• Maddison Project: Historical GDP for long-run state formation
• PRIO GRID: Cell-level data for spatial analyses

L07–L10 (future): Geoeconomics

• GSDB: Sanctions episodes and effectiveness
• UN Comtrade / CEPII BACI: Bilateral trade disruptions
• BP/Energy Institute: Energy dependence and trade
• USGS: Critical mineral supply chains

Important Principles

• Always cite data sources — add \source{Dataset (Year)} to slides using downloaded data
• Prefer open/free sources — avoid paywalled datasets unless user has access
• Check data freshness — note the release version/year of downloaded data
• Reproducibility — save download scripts so data can be refreshed next semester
• Do NOT fabricate data — if a download fails or data doesn't exist, say so clearly

Execution Command

# Scan a lecture for data opportunities
/data-sourcer scan lecture_01.qmd

# Search for specific data
/data-sourcer search "rainfall anomalies Sub-Saharan Africa"

# Download a specific dataset
/data-sourcer download "World Bank WDI GDP per capita conflict countries 2000-2024"