Ar Source Auditor

Process

Step 1: Load All Sources

Read every source record file listed in the corpus manifest.

Step 2: Evaluate Each Source

For each source, assess and update the following fields:

Source type verification (source_type):

• Verify the classification is correct (journal / policy / media / book / archive / dataset)
• Reclassify if the original assignment was wrong

Reliability scoring (reliability_score, 0.0 to 1.0): Score based on these criteria:

• Is it peer-reviewed? (+0.25)
• Is it from a recognised academic publisher or reputable outlet? (+0.20)
• Are the authors identifiable and credentialed? (+0.15)
• Is the publication date within the temporal scope? (+0.15)
• Is it relevant to the research question? (+0.15)
• Is it a primary source or original research? (+0.10)

Verification status (verification_status):

• verified: the source clearly exists, metadata is accurate
• partially_verified: the source likely exists but some metadata could not be confirmed
• unverified: could not confirm the source exists or metadata is questionable

Peer review status (peer_reviewed):

• Set to true only for sources published in peer-reviewed journals or conference proceedings

Inclusion recommendation: Add a new field audit_recommendation with values:

• include: reliability >= 0.4 and relevant to scope
• exclude: reliability < 0.3 or clearly outside scope
• flag: reliability 0.3-0.4 or relevance is borderline — needs human decision

Audit notes: Add a field audit_notes with a brief explanation of the scoring rationale.

Step 2.5: Resolve Missing Authors via WebSearch

For any source where authors is empty ([]), or contains only placeholder values like "Various", "Unknown", "et al.", or "N/A":

1. Run a WebSearch to find the real authors: "{title}" authors {year}
2. If the source has a URL, also try a domain-specific search: site:{domain_from_url} "{title}"
3. Look for author names in the search results — check for:
   • Author bylines on the article/paper page
   • Citation metadata (e.g., "Cited as: Smith, J., Jones, K. (2024)")
   • Google Scholar or publisher listings
4. Update the authors field with the actual author names found
5. If multiple authors are found, list them all (up to 6; use "et al." after the 6th)
6. If authors still cannot be identified after searching, set authors to ["[authors unidentified]"] and add a note to audit_notes: "Author identification failed despite targeted search"

This step is critical — proper author attribution is essential for academic citations. Do NOT leave "Various" or "Unknown" in the final source records.

Step 3: Verify Metadata via WebSearch

For sources with verification_status of unverified, run a targeted WebSearch:

• "{title}" "{first_author}" {year} to confirm the source exists
• Update verification_status based on results

Step 4: Write Updated Source Files

Overwrite each source record file with the updated fields (reliability_score, verification_status, peer_reviewed, source_type, audit_recommendation, audit_notes).

Step 5: Write Audit Report

Write {workspace}/sources/audit/source_audit_report.json:

{
  "total_sources": 0,
  "included": 0,
  "excluded": 0,
  "flagged": 0,
  "average_reliability": 0.0,
  "by_category": {
    "academic": { "count": 0, "avg_reliability": 0.0 },
    "policy": { "count": 0, "avg_reliability": 0.0 },
    "media": { "count": 0, "avg_reliability": 0.0 }
  },
  "excluded_sources": [
    { "source_id": "string", "title": "string", "reason": "string" }
  ],
  "flagged_sources": [
    { "source_id": "string", "title": "string", "concern": "string" }
  ]
}

Step 6: Write Audit Summary

Write {workspace}/sources/audit/source_audit_summary.md:

A human-readable summary (600 words max) that includes:

• Total sources collected and breakdown by category
• How many are recommended for inclusion, exclusion, and flagged
• Average reliability scores by category
• Any concerns about corpus coverage or gaps
• List of excluded sources with brief reasons
• List of flagged sources needing human decision

This summary is presented to the user at Gate 2 for approval.

Quality Rules

• Be honest about reliability — do not inflate scores
• Flag sources where metadata cannot be verified rather than silently including them
• Ensure the corpus has adequate coverage of the research question's key themes
• If fewer than 8 sources are recommended for inclusion, flag this as a coverage concern
• Use British English in all output text