Ar Retrieval

From the scoping report, extract:

• The recommended research question
• All sub-questions
• Scope boundaries (temporal range, geographic scope, included/excluded topics)
• The research track (literature_review / policy_analysis / comparative_case_study)

Step 2: Execute Retrieval Searches

Run 15-25 WebSearch calls organised into 4 source categories:

Academic sources (5-8 searches):

• "{research_question}" peer-reviewed journal
• "{topic}" systematic review OR literature review
• One search per sub-question: "{sub_question}" academic research
• "{topic}" theoretical framework {discipline}
• "{topic}" methodology {discipline} research

Policy sources (3-5 searches, more for policy_analysis track):

• "{topic}" government report official
• "{topic}" policy document white paper
• "{topic}" official statement institutional
• "{topic}" regulatory framework

Media sources (3-4 searches):

• "{topic}" news analysis commentary
• "{topic}" public debate discourse
• "{topic}" media coverage {year_range}

Reports and working papers (3-4 searches):

• "{topic}" think tank report
• "{topic}" research brief working paper
• "{topic}" {region} report (if geographic scope is set)

Adjust emphasis by track:

• literature_review: weight academic sources (8+ searches)
• policy_analysis: weight policy + media sources (5+ policy searches)
• comparative_case_study: add per-case searches (e.g., "{topic}" {case_name})

Step 3: Create Source Records

For each unique result found, create a SourceRecord JSON file:

1. Generate a source ID: .venv/bin/python3 utils/schemas.py id src

2. Populate fields:

   • source_id: the generated ID
   • title: exact title from search result
   • authors: author names — if the initial search result does not include author names, run an additional targeted WebSearch: "{title}" authors published {year} to find them. Only use [] as a last resort after this additional search also fails. NEVER use placeholder values like "Various" or "Unknown".
   • year: publication year if available
   • source_type: classify as journal / policy / media / book / archive / dataset
   • venue: journal name, publisher, or outlet
   • url_or_path: URL from search result
   • jurisdiction: geographic jurisdiction if relevant
   • peer_reviewed: true if from a peer-reviewed journal
   • primary_or_secondary: classify
   • reliability_score: 0.0 (will be scored by auditor)
   • verification_status: "unverified" (will be verified by auditor)

3. Write to {workspace}/sources/{category}/src-{id}.json where category is academic, policy, or media

Step 4: Deduplicate

Check for duplicate sources by comparing:

• URL match (exact)
• Title similarity (near-exact match after lowercasing)

Remove duplicates, keeping the first occurrence.

Step 5: Write Corpus Manifest

Write {workspace}/sources/corpus_manifest.json:

{
  "total_sources": 0,
  "academic_count": 0,
  "policy_count": 0,
  "media_count": 0,
  "report_count": 0,
  "search_queries_used": ["query1", "query2"],
  "source_ids": ["src-a1b2c3d4", "src-e5f6g7h8"]
}

Step 6: Update Status

Update {workspace}/status.json:

• Set state to CORPUS_BUILDING
• Add "retrieval" to stages_completed if not present

Quality Rules

• Every source must have a valid URL — do not fabricate URLs
• Do not invent sources that were not found via WebSearch
• Aim for at least 15 sources across all categories
• Ensure geographic and temporal scope match the scoping report
• Flag if fewer than 5 academic sources are found — this is a feasibility concern
• Use British English in all output text