Literature Review

Usage

Provide the agent with a research topic or question. Optionally specify the desired scope (time range, source types), the number of papers to include, or a particular synthesis format.

Conduct a literature review on LLM evaluation benchmarks published between 2022-2025.

Focus: What benchmarks exist, what do they measure, and what gaps remain
in evaluating reasoning, safety, and real-world task completion?

Examples

Example 1: Literature Review on LLM Evaluation Benchmarks

User Request:

Review the literature on LLM evaluation benchmarks from 2022-2025, focusing on reasoning, safety, and task completion.

Search Strategy:

Inclusion/Exclusion Criteria:

PRISMA-Style Flow:

Records identified through search: 847
After duplicate removal:           612
After title/abstract screening:    148
After full-text assessment:         42
Final papers included:              42

Synthesis Table (excerpt):

Synthesized Finding (excerpt):

The literature reveals a clear trajectory from static knowledge tests (MMLU) toward dynamic, agentic evaluations (AgentBench, SWE-bench) that measure an LLM's ability to act in realistic environments. However, a significant gap persists: no single benchmark suite comprehensively evaluates reasoning, safety, and task completion together. Most benchmarks optimize for one dimension, creating a fragmented evaluation landscape where models can appear strong on reasoning benchmarks while performing poorly on safety metrics.

Example 2: Systematic Review Protocol with PRISMA-Style Flow

User Request:

Create a systematic review protocol for studying the effectiveness of retrieval-augmented generation (RAG) in reducing LLM hallucinations.

Research Question (PICO format):

• Population: Large language models (GPT-4 class and above)
• Intervention: Retrieval-augmented generation (RAG)
• Comparison: Non-RAG baseline (standard prompting)
• Outcome: Hallucination rate reduction (measured by factual accuracy metrics)

Search Strategy:

("retrieval-augmented generation" OR "RAG") AND
("hallucination" OR "factual accuracy" OR "faithfulness") AND
("large language model" OR "LLM" OR "GPT" OR "Claude")

Databases: Semantic Scholar, arXiv, ACM Digital Library, Google Scholar Date range: January 2023 to December 2025

Screening Protocol:

Phase 1 — Title/Abstract Screening:
  Include if: Empirically measures hallucination with and without RAG
  Exclude if: Theoretical only, no quantitative results, not LLM-focused

Phase 2 — Full-Text Review:
  Include if: Reports specific hallucination metrics (FActScore, ROUGE-L
              against ground truth, human evaluation scores)
  Exclude if: RAG used for non-factual tasks (creative writing, code gen)

Data Extraction Template:

Expected PRISMA Diagram:

Identification:  ~1,200 records from 4 databases
Screening:       ~400 after title/abstract review
Eligibility:     ~80 after full-text assessment
Included:        ~35 meeting all criteria

Best Practices

• Document every decision for reproducibility. Record search queries, database dates, and the exact inclusion/exclusion criteria so another researcher could replicate the review.
• Use structured data extraction. A consistent extraction template ensures you capture the same fields from every paper, making cross-study comparison possible.
• Distinguish between quantity and quality of evidence. A theme supported by 20 blog posts is weaker than one supported by 3 rigorous RCTs. Weight your synthesis accordingly.
• Report what you did not find. Gaps in the literature are as important as the findings. Explicitly note underexplored areas, missing populations, or untested conditions.
• Keep the synthesis thematic, not paper-by-paper. A literature review that reads as a list of paper summaries is less useful than one organized around themes that weave findings together.
• Update the search before finalizing. If the review takes weeks to write, re-run the search before submitting to catch newly published work.

Edge Cases

• Insufficient literature: If fewer than 5 relevant papers exist, acknowledge this and reframe the output as a "scoping review" or "research gap analysis" rather than a comprehensive literature review.
• Preprint-heavy fields: In fast-moving areas like AI/ML, much of the relevant work may be on arXiv without peer review. Include preprints but flag their review status and note this as a limitation of the evidence base.
• Conflicting study results: When studies reach opposite conclusions, compare their methodologies, sample sizes, and conditions. Present the conflict transparently rather than arbitrarily siding with one study.
• Interdisciplinary topics: Reviews spanning multiple fields (e.g., AI + healthcare) may require searching domain-specific databases (PubMed) in addition to CS databases. Adjust the search strategy accordingly and note where terminology differs between fields.
• Non-English sources: If the topic has significant literature in other languages, note this limitation if you restrict to English and flag key non-English works that appear in citation lists.