Literature Survey

STEP 0 (MANDATORY): Verify Web Tools

Do this FIRST before any other step.

This pipeline uses WebSearch and WebFetch for paper discovery and content extraction. These are built-in Claude Code tools that work without browser extensions.

Verify tools are available:

1. Call WebSearch with a test query (e.g., "metamaterial absorber THz")
2. If WebSearch works → proceed to Step 1 (this is the expected path)
3. If WebSearch fails → inform user that web tools are unavailable and the pipeline cannot run

No browser extension or campus VPN is needed. WebSearch provides paper discovery, and WebFetch provides content extraction from publisher pages (open-access papers).

Step 1: Clarify the Design Target

Extract or ask the user for:

Frequency conversion: Pipeline expects GHz.

• THz to GHz: multiply by 1000
• Wavelength (um) to GHz: f = 299792.458 / wavelength_um

Summarize the target back to the user for confirmation before proceeding.

Step 2: Plan Queries + Scrape Google Scholar

2a: Generate search queries

The query planner uses frequency band aliasing to generate 15-20 query variants from a single target. For example, a target of 6-14 GHz spawns queries with aliases: "microwave", "GHz", "X-band", "Ku-band", "K-band".

Band alias mapping:

• < 30 GHz → microwave, GHz, + specific IEEE bands (L/S/C/X/Ku)
• 30-300 GHz → millimeter wave, mmWave, mm-wave, + Ka/E/W-band
• 300-10,000 GHz → terahertz, THz, sub-THz (if < 500 GHz), far-infrared (if > 3000 GHz)
• > 10,000 GHz → infrared, IR, mid-IR, far-IR, near-IR

cd D:/Claude && python -c "
from target_to_hypothesis.skills.target_interpreter import interpret_target
from target_to_hypothesis.skills.query_planner import plan_queries
from target_to_hypothesis.utils.llm import make_llm_fn
import json, os
os.environ['OPENAI_API_KEY'] = 'USER_KEY'
llm_fn = make_llm_fn(max_tokens=2000)
target = interpret_target(description='USER_DESC', freq_min_ghz=FMIN, freq_max_ghz=FMAX, constraints={}, llm_fn=llm_fn)
query_plan = plan_queries(target, llm_fn=llm_fn, max_primary=5, max_secondary=5)
print(json.dumps({
    'primary': query_plan.primary_queries,
    'secondary': query_plan.secondary_queries,
    'embedding_query': query_plan.embedding_query,
    'related_mechanisms': query_plan.related_mechanisms,
    'related_geometry_terms': query_plan.related_geometry_terms,
}, indent=2))
"

The LiteratureQueryPlan output includes:

• primary_queries — up to 5 main search queries (band-aliased variants)
• secondary_queries — up to 5 geometry-focused supplementary queries
• exclusion_terms — terms to filter out (e.g., "review article")
• related_mechanisms — physical mechanisms to look for (impedance matching, etc.)
• related_geometry_terms — geometry keywords for secondary search
• embedding_query — rich natural-language description for embedding search (Step 3.5)

LLM configuration: make_llm_fn supports any OpenAI-compatible API:

llm_fn = make_llm_fn(
    api_key='...',           # OpenAI API key (or env OPENAI_API_KEY)
    model='gpt-5.4',        # default model
    base_url=None,           # custom endpoint (DeepSeek, local vLLM, Azure, etc.)
    temperature=0.2,
    max_tokens=4000,
    max_retries=3,
    json_mode=False,         # set True for structured JSON output
)

2b: Search for papers using WebSearch

For each primary and secondary query, use WebSearch to find relevant papers:

For each query in query_plan.primary_queries + query_plan.secondary_queries:
  results = WebSearch(query="{query} metamaterial site:scholar.google.com OR site:sciencedirect.com OR site:ieee.org OR site:nature.com")

From each WebSearch result, extract:

• Paper title (from result title)
• URL (DOI or publisher URL)
• Year (from snippet or title)
• Authors (from snippet)

Also run targeted searches for the specific design target:

WebSearch(query="metamaterial absorber {frequency_range} {response_type} design fabrication")
WebSearch(query="{topology_keywords} metamaterial {frequency_band} absorption")

Build candidate list from all search results, dedup by title/DOI.

2b-alt: Use WebFetch for open-access full text

For open-access papers (PMC, arXiv, MDPI, Nature open access), use WebFetch to extract content:

content = WebFetch(url="https://pmc.ncbi.nlm.nih.gov/articles/PMC...", prompt="Extract: title, authors, abstract, key design parameters, absorption performance, frequency range, materials used, and figure captions.")

WebFetch works well with:

• PMC (pmc.ncbi.nlm.nih.gov) — full text
• arXiv (arxiv.org) — full text
• MDPI (mdpi.com) — open access
• Nature/Springer open access articles

WebFetch does NOT work with:

• Google Scholar search pages (JS-rendered)
• Paywalled publisher pages (IEEE, Elsevier closed access)

2c: Alternative retrieval backends (if browser unavailable)

The pipeline supports 4 retrieval backends. The agent uses browser by default, but the pipeline internally uses these when retrieval_mode is set:

from target_to_hypothesis.skills.retriever import retrieve_papers, SemanticScholarBackend, CrossrefBackend, ArxivBackend

# The pipeline calls this internally with pre-scraped candidates:
papers = retrieve_papers(
    plan=query_plan,
    backends=[SemanticScholarBackend(), CrossrefBackend(), ArxivBackend()],
    per_query_limit=20,     # papers per query per backend
    total_limit=40,         # final cap after dedup
)

Retrieval scoring formula (applied to all candidates):

retrieval_score = 0.40 × keyword_overlap + 0.30 × recency + 0.30 × citation_score

• keyword_overlap (0-1): fraction of query keywords found in abstract/title
• recency: 1.0 (≤2 yrs), 0.8 (≤5 yrs), 0.5 (≤10 yrs), 0.3 (older)
• citation_score: 1.0 (≥100), 0.7 (≥20), 0.4 (≥5), 0.2 (<5)

FALLBACK (if browser unavailable): Use retrieval_mode="semantic_scholar".

Step 3: Search Padilla Lab Related Work (ALWAYS)

Every literature review MUST include top-5 Padilla Lab papers.

1. Use WebSearch to find Padilla's most relevant papers:

WebSearch(query="Willie Padilla metamaterial absorber {user_frequency_band} {user_response_type}")
WebSearch(query="Willie Padilla Duke metamaterial terahertz absorber most cited")

Also fetch his publication list:

WebFetch(url="https://scholars.duke.edu/person/willie.padilla/publications", prompt="List all publications by Willie Padilla related to metamaterial absorbers. Include title, year, journal, and any citation info.")

2. From the results, identify the top 5 most relevant to the user's target. Consider frequency overlap, device type, and design approach. Always include his foundational papers:

   • "Perfect metamaterial absorber" (Landy et al., PRL 2008, ~8700 citations)
   • "A metamaterial absorber for the terahertz regime" (Tao et al., Opt. Express 2008, ~1750 citations)
   • "Metamaterial electromagnetic wave absorbers" (Watts et al., Adv. Mater. 2012, ~2450 citations)

3. Build:

padilla_papers = [
    {"title": "...", "year": 2008, "venue": "Physical Review Letters",
     "citations": 8696, "relevance_note": "..."},
    # ... 4 more
]

4. Extract content from open-access Padilla papers using WebFetch:

For Padilla papers available on open-access platforms (arXiv, PMC, Optica open access):

WebFetch(url="paper_url", prompt="Extract: abstract, key design parameters, unit cell geometry, materials, absorption spectrum details, and figure captions with descriptions.")

Step 4: Extract Full Text from Top Papers

For open-access papers, use WebFetch to get rich content. For paywalled papers, use abstract-only analysis.

For top 10 papers with DOI or URL:

A. Attempt full text extraction via WebFetch

content = WebFetch(
  url="https://doi.org/{DOI}" or paper.url,
  prompt="Extract the following from this research paper:
    1. Title and authors
    2. Full abstract
    3. Design parameters (unit cell dimensions, period, thickness, materials)
    4. Absorption/transmission performance (peak values, bandwidth, frequency range)
    5. Physical mechanisms (impedance matching, Fabry-Perot, magnetic resonance, etc.)
    6. Figure captions and descriptions (especially absorption spectra and unit cell geometry)
    7. Key conclusions and design insights
    Format as structured text with clear sections."
)

Works well with open-access publishers:

• PMC (pmc.ncbi.nlm.nih.gov)
• arXiv (arxiv.org)
• MDPI (mdpi.com)
• Optica/OSA open access
• Nature/Springer open access

Falls back to abstract-only for:

• Paywalled IEEE, Elsevier, Wiley articles
• Pages that return login/paywall content

B. Store extracted content

full_text_dict[paper.paper_id] = {
    "title": extracted_title,
    "abstract": extracted_abstract,
    "parameters": extracted_params,
    "performance": extracted_perf,
    "mechanisms": extracted_mechanisms,
    "figure_captions": extracted_fig_captions,
    "source": "full_text" if len(content) > 500 else "abstract_only",
}

C. Download figures (if URLs found)

If WebFetch extracts figure URLs from the page, download them:

curl -L -o "D:/Claude/artifacts/figures/{paper_id}_fig{N}.png" "FIGURE_URL"

D. Build figure-to-paper mapping

Maintain a figure_map dictionary that tracks all downloaded figures per paper:

figure_map = {}  # keyed by paper_id

# After each successful curl download:
if paper_id not in figure_map:
    figure_map[paper_id] = []
figure_map[paper_id].append({
    "path": f"D:/Claude/artifacts/figures/{paper_id}_fig{N}.png",
    "caption": caption_text,
    "fig_num": N
})

This mapping is consumed in Step 6 to embed figures per-paper in the report.

Reference: browser_paper_reader module (for Python pipeline)

The target_to_hypothesis pipeline internally uses browser_paper_reader for content extraction:

• detect_publisher(url) — Maps domain → publisher (12 supported)
• is_paywalled(page_text) — Detects login walls (16 indicators)
• parse_paper_html(page_text, url) → ExtractedPaperContent dataclass
• prepare_full_text_for_reader(content, max_chars=8000) — Condenses for LLM

Step 5: Analyze Papers and Generate Hypotheses

DO NOT run the Python pipeline (run_pipeline / target_to_hypothesis). It makes 15+ OpenAI API calls and takes 10-20 minutes. Instead, analyze the papers you already gathered from WebSearch/WebFetch directly.

5a: Rank papers by relevance

Score each paper (0-1) based on:

• Frequency overlap with target (most important — paper's operating frequency vs user's target)
• Design approach match (topology, materials, layer count)
• Recency (newer papers preferred: 1.0 if ≤2yr, 0.8 ≤5yr, 0.5 ≤10yr, 0.3 older)
• Citation count / venue quality (1.0 if ≥100, 0.7 ≥20, 0.4 ≥5, 0.2 <5)

Sort papers by combined score. Keep top 15-20.

5b: Synthesize across papers

From the ranked papers, identify:

• Common geometries: SRR, cross, patch, ring, fishnet, ELC, cut-wire, etc.
• Shared physical mechanisms: impedance matching, multi-resonance, Fabry-Perot cavity, LC circuit, magnetic dipole
• Design insights: parameter trends (how period scales with frequency, typical layer thicknesses)
• Material patterns: common substrates (polyimide, silicon, GaAs, Rogers) and conductors (Au, Al, Cu)

5c: Generate design hypotheses

Propose 3-5 design candidates. For each hypothesis:

• Topology (family name): e.g., cross_resonator, split_ring, patch_with_slots
• Score (0-1) based on weighted criteria:
  • Evidence support from papers (25%)
  • Target frequency/performance match (25%)
  • Simplicity of fabrication (15%)
  • Simulation cost (10%)
  • Fabrication realism (10%)
  • CST template readiness (15%) — query the template library (see below)
• Justification: which papers support this design
• Estimated parameters: period, thickness, materials, expected performance
• Physical mechanism: how it achieves absorption

CST Template Readiness scoring — Read D:/Claude/MetaClaw/.templates/library-index.json and check if any existing template matches:

For each hypothesis topology family + target frequency range:
  1. Read .templates/library-index.json
  2. Find templates with matching family
  3. Compute frequency overlap fraction

  Scoring:
  - Exact match (same family, >80% freq overlap):  readiness = 1.0
  - Partial match (same family, >50% freq overlap): readiness = 0.7
  - Family match only (different freq):             readiness = 0.4
  - No match in library:                            readiness = 0.1

  If a template exists, note in the hypothesis:
  "Existing template: {id}, {best_absorption}% absorption in {iterations} iterations.
   Reusable parameters: {params}. Expected fast convergence."

This score increases as more experiments are completed and saved to the library.

5d: Proceed directly to Step 6

Do NOT run any Python scripts. Go straight to writing research_notes.md.

Reference: target_to_hypothesis Python Pipeline (DO NOT RUN unless user explicitly requests)

The pipeline below is documented for reference only. It makes 15+ OpenAI API calls and takes 10-20 minutes. Only run if the user explicitly says "run the Python pipeline".

5-ref-a: PipelineConfig — Full Reference

All available configuration fields:

@dataclass
class PipelineConfig:
    # --- Retrieval ---
    retrieval_mode: str = "browser"           # "browser" | "semantic_scholar" | "embedding"
    semantic_scholar_api_key: str = None       # API key for Semantic Scholar (optional, higher rate limits)
    per_query_limit: int = 20                  # papers per query per backend
    total_paper_limit: int = 40                # final cap after dedup + scoring

    # --- Embedding & LLM ---
    openai_api_key: str = None                 # for embedding search + LLM calls
    embedding_model: str = "text-embedding-3-large"  # OpenAI embedding model
    llm_model: str = "gpt-5.4"                # LLM for analysis/synthesis
    use_llm: bool = True                       # set False to skip LLM-based steps
    llm_fn: callable = None                    # custom LLM callable (overrides model/key)

    # --- Pre-scraped data (from Steps 2-4) ---
    pre_scraped_papers: list[dict] = None      # raw dict format
    pre_scraped_candidates: list[PaperCandidate] = None  # structured format
    full_text_contents: dict[str, ExtractedPaperContent] = None  # from Step 4

    # --- Frequency filtering ---
    use_frequency_filter: bool = True          # enable 4-stage frequency filter
    frequency_filter_top_n: int = 30           # keep top N after frequency filtering

    # --- Full text ---
    full_text_top_n: int = 10                  # papers to attempt full-text extraction

    # --- Hypothesis generation ---
    hypothesis_mode: str = "paper_based"       # "paper_based" | "ontology"
    max_hypotheses: int = 5                    # max hypotheses to generate

    # --- Interactive mode ---
    interactive: bool = False                  # enable Step 0 interactive target clarification
    dialog_fn: callable = None                 # callable for user dialogs in interactive mode

    # --- Phase 3 ---
    run_phase3: bool = True                    # enable deep per-paper analysis
    phase3_top_n: int = 10                     # papers to analyze in Phase 3

    # --- Padilla Lab ---
    padilla_papers: list[dict] = None          # pre-populated by agent (Step 3)

    # --- Output ---
    save_artifacts: bool = True                # save all intermediate artifacts
    artifacts_dir: str = None                  # custom output directory (default: D:/Claude/artifacts)

5-ref-b: Run the pipeline

cd D:/Claude && python -c "
import json, os, sys
os.environ['OPENAI_API_KEY'] = 'USER_KEY'

from target_to_hypothesis.pipelines.run_target_to_hypothesis import PipelineConfig, run_pipeline

config = PipelineConfig(
    retrieval_mode='browser',
    hypothesis_mode='paper_based',
    pre_scraped_candidates=final_candidates,
    full_text_contents=full_text_dict,
    padilla_papers=padilla_papers,
    openai_api_key=os.environ['OPENAI_API_KEY'],
    embedding_model='text-embedding-3-large',
    llm_model='gpt-5.4',
    max_hypotheses=5,
    per_query_limit=20,
    total_paper_limit=30,
    use_frequency_filter=True,
    frequency_filter_top_n=30,
    full_text_top_n=10,
    save_artifacts=True,
    run_phase3=True,
    phase3_top_n=10,
)

result = run_pipeline(
    description='USER_DESC',
    freq_min_ghz=FMIN, freq_max_ghz=FMAX,
    constraints={}, config=config,
)
print(f'Run: {result.run_id}')
print(f'Papers: {len(result.paper_candidates)}')
print(f'Evidence: {result.evidence_grades.overall_evidence_quality}')
print(f'Hypotheses: {len(result.ranked_hypotheses.hypotheses)}')
for i, h in enumerate(result.ranked_hypotheses.hypotheses):
    print(f'  {i+1}. {h.family_display_name or h.family} (score={h.score:.3f})')
print(f'Report: {result.report_path}')
"

5-ref-c: Pipeline internal steps

The pipeline runs Steps 3.5-12 internally:

Step 3.5: Embedding search Uses EmbeddingSearcher with OpenAI text-embedding-3-large:

class EmbeddingSearcher:
    def __init__(self, papers, run_id, model_name, api_key, base_url=None, cache_dir=None)
    def search(self, query=None, target_description=None, top_k=10) -> list[dict]
    def search_with_examples(self, example_papers, top_k=10) -> list[dict]

• Embeds all candidate papers (title + abstract) and the target description into vectors
• Ranks by cosine similarity (custom NumPy implementation, no external vector DB)
• Embeddings cached at ~/.cache/metamaterial-embeddings/ (SHA-256 fingerprinted per run)
• Batch embedding: 100 texts per API call with retry logic
• Non-critical: graceful fallback with warning if embedding fails

Step 4: Frequency filtering (4-stage pipeline)

from target_to_hypothesis.skills.frequency_filter import filter_papers_by_frequency

filtered = filter_papers_by_frequency(
    papers=candidates,
    target_min_ghz=FMIN,
    target_max_ghz=FMAX,
    is_broadband=False,    # True for broadband designs (relaxes tolerance)
)

Frequency filter scoring:

• Has frequency data AND overlaps: 0.5 + 0.5 × overlap_fraction (+ 0.1 bonus if band also matches, capped at 1.0)
• Band-label match only (no freq data): 0.4
• Has frequencies but no overlap: 0.2
• No frequency info, no band match: 0.1
• Has frequencies, no band match, no overlap: 0.05

IEEE band designation table used internally:

L-band: 1-2 GHz | S-band: 2-4 GHz | C-band: 4-8 GHz | X-band: 8-12 GHz
Ku-band: 12-18 GHz | K-band: 18-27 GHz | Ka-band: 26.5-40 GHz
V-band: 40-75 GHz | W-band: 75-110 GHz