Lit Review Hinf5016

Inputs

Ask the user (or read from their request):

The paper count is fixed at 8 to match the HINF 5016 assignment. The ranking, writer, and verification stages are all hard-coded around N=8; do not accept a user override.

If any of these are missing and the topic is unclear, ask one consolidated question, then proceed.

Pipeline

Run the stages sequentially. Each stage reads its prompt from prompts/, fills the placeholders from the previous stages' outputs, and writes its result to a numbered file under the run directory. Do not skip stages. Do not parallelize.

Stage details:

Each prompt uses mustache-style placeholders like {{TOPIC}} or {{CANDIDATES_JSON}}. Replace them with the relevant content (the user's topic, or a JSON dump of the previous stage's output) before reading the prompt as your instructions.

Stage 1 — Query builder. Read prompts/01_query_builder.md. Substitute {{TOPIC}}. Produce a search plan as JSON: keywords, MeSH terms, boolean query, inclusion/exclusion bullets, year range, language. Save to 01_search_plan.json.

Stage 2 — Search. Read prompts/02_search.md. Substitute {{SEARCH_PLAN_JSON}} with stage 1's content. Run 3–6 distinct web_search queries spanning PubMed and Google Scholar; vary keyword combinations for recall. Target 20–40 unique candidates. Record every candidate's title, authors, year, venue, DOI, URL, abstract, and source_db. Also record the exact query strings you sent in queries_used. Save to 02_candidates.json.

Stage 3 — Screen. Read prompts/03_screen.md. Substitute {{TOPIC}}, {{INCLUSION_BULLETS}}, {{EXCLUSION_BULLETS}}, {{CANDIDATES_JSON}}. For every candidate output decision (include/exclude/maybe), reason, relevance_score. Save to 03_screening.json.

Stage 4 — Snowball. Pick the highest-relevance included paper as seed. Read prompts/04_snowball.md. Substitute {{SEED_JSON}} and {{TOPIC}}. Run web_search to find forward citations (papers citing the seed) and backward references (papers the seed cites). One round only; 10–20 new candidates total. Save to 04_snowball.json.

Stage 5 — Extract. Build the extraction pool deterministically (no subjective re-ranking):

1. Start with every stage-3 candidate whose decision == "include", sorted by relevance_score descending.
2. Deduplicate against stage-4 snowball results by DOI, then by normalized title.
3. If the result is fewer than 12 papers, append stage-4 forward-snowball candidates in the order they appear, then backward-snowball candidates, until you reach 12 — or until both lists are exhausted.
4. Cap the pool at 16 papers total.

Then for each paper in the pool, read prompts/05_extract.md, substitute {{PAPER_JSON}}, and use web_search to retrieve full text from PMC / publisher / arXiv. Output the four fields with verbatim quote_span excerpts AND persist the actual retrieved text (trimmed) into retrieved_text so the verifier can check spans from disk. If full text is paywalled, set source_type = "abstract_only" and extract from the abstract. Append one JSON object per paper as a line to 05_extractions.jsonl.

Stage 6 — Rank. Read prompts/06_rank.md. Substitute {{TOPIC}} and {{EXTRACTIONS_JSON}}. Pick exactly 8 papers honoring the diversity constraints (no shared first author; ≤2 papers per dataset; ≤3 papers per method family; ≥3 distinct theme tags). Output selected (with rank 1..8 and theme_tag), rejected (with one-sentence reasons), and diversity_notes. Save to 06_ranking.json.

Stage 7 — Synthesize. Read prompts/07_synthesize.md. Substitute {{EXTRACTIONS_AND_RANK_JSON}}. Produce themes (2–4 groups covering all 8 papers), similarity matrix (6–12 informative pairs), gaps (3–6), and cross-paper observations (3–6 sentences). Save to 07_synthesis.json.

Stage 8 — Write. Read prompts/08_write_review.md. Substitute {{TOPIC}}, {{TITLE}}, {{INCLUSION_BULLETS}}, {{EXCLUSION_BULLETS}}, {{PRISMA_JSON}}, {{EXTRACTIONS_JSON}}, {{SYNTHESIS_JSON}}. Produce the full Markdown manuscript. Use the section structure verbatim. Vancouver references numbered by first appearance. Save to 08_literature_review.md.

Audit bundle

Alongside the 8 numbered files, also write to the run directory:

• 00_input.json — the user's topic, title, output dir, and the start timestamp.
• prisma_flow.json — counts: identified (stage 2 + 4), screened (stage 3 total), eligible (stage 5 attempted), included (stage 6 = 8). Plus quote_span_verified / quote_span_total after the verification step.
• search_log.json — every web_search query string you ran across stages 2, 4, 5, with the stage label and result count.

05_extractions.jsonl is itself part of the audit bundle and must contain a retrieved_text field on every line — the verifier reads from this file alone, so quote-span checks remain reproducible after the agent session ends.

You do not need a token_usage.json since the agent harness owns token counting; if the harness exposes a way to read usage, write it, otherwise omit.

Verification (run before declaring done)

1. Counts. 06_ranking.json.selected has length 8. 08_literature_review.md has eight ### 3.1 … ### 3.8 subsections.
2. Structure. 08_literature_review.md headings appear in this order: # {title} → ## Abstract → ## 1. Introduction → ## 2. Methods → ### 2.1 Eligibility criteria → ### 2.2 Information sources → ### 2.3 Study selection → ## 3. Results → ### 3.1 … ### 3.8 → ## 4. Discussion → ## 5. Conclusions → ## References.
3. Citations. Vancouver [n] numbers in ## References are in order of first appearance in the body.
4. Quote spans (disk-only). For every paper in 05_extractions.jsonl, read its retrieved_text field from the file and check that each of the four quote_span values is a substring of that retrieved_text after this normalization: lowercase both sides, collapse runs of whitespace to a single space, strip leading/trailing whitespace. Do not rely on agent memory — the verifier must be reproducible from disk alone. Record pass/fail counts across the 8 selected papers (5_extractions.jsonl × 4 fields = 32 spans) in prisma_flow.json as quote_span_verified / quote_span_total. If any span fails, list the failing paper_id and field name in your final summary so the user knows which to manually check.
5. Diversity. 06_ranking.json.diversity_notes should explain how the constraints were applied. Spot-check that no two of the 8 papers share a first author.

Final summary to the user

After verification, report back in 4–6 lines:

• Path to 08_literature_review.md.
• The 8 selected papers (Vancouver index → first-author-et-al year → short title).
• PRISMA counts (identified / screened / eligible / included).
• Quote-span verification pass rate.
• Any paywalled / abstract-only papers, by index.
• Reminder: the draft is a starting point. The user must read it end to end, fix any flagged quote spans, and apply the AMIA template themselves. The technical report is the user's own writing — point them at the audit bundle.

Hard rules

• Never fabricate a paper. Every paper in 05_extractions.jsonl must trace back to a real web_search hit. If the search returned nothing usable, report that and stop — do not invent.
• Never invent a quote. quote_span is a verbatim string from text the agent actually saw. The verification step exists precisely to catch this.
• Exactly 8 papers in the final review. Not 7, not 9.
• Do not generate the technical report. It is a separate part of the assignment that the student must write personally.
• Do not produce a PDF or apply AMIA formatting. Markdown only.
• Run stages sequentially. Do not start stage N+1 until stage N's output file is on disk.