arXiv Paper Screener

1. Load the field profile from references/field_profiles.json.
2. Search arXiv using the configured keywords for that field and sort by submitted date descending.
3. Filter papers to the requested time window on the client side.
4. Enrich each paper with:
   • institutions and citation count from OpenAlex
   • if OpenAlex does not provide institution metadata, read the arXiv PDF first page and infer the first institution from the author-affiliation block
   • GitHub repository link and star count when code is available
5. Filter out papers whose first institution is not in the allowlist file:
   • global qs100_universities_snapshot
   • global major_ai_and_internet_companies
   • the field-specific institutions for the current field
6. Rank the filtered papers:
   • if code is available, prefer GitHub stars
   • if code is not available, use citation count
7. Write both:
   • a machine-readable results.json
   • a readable results.md
   • write to temporary files first, then atomically replace the final outputs so readers never see half-updated results

Important Notes

• arXiv itself does not provide a stable public per-paper citation metric in the paper metadata API. For the no-code fallback ranking, use OpenAlex cited_by_count.
• arXiv metadata does not reliably expose author affiliations. Institution filtering first uses OpenAlex authorship/institution metadata, then falls back to reading the PDF first page when institution metadata is missing.
• Only the first institution is used for filtering and matching. Later institutions on the paper are ignored.
• The PDF fallback is conservative: it scans the first page front matter and tries to identify the earliest allowlisted institution mention before Abstract or Introduction.
• Output files should be replaced atomically. Do not stream partial results into the final results.json or results.md paths while a long screening run is still in progress.
• The institution allowlists are conservative by design. Papers from institutions outside the allowlist are filtered out even if they may still be good.
• Treat references/institution_allowlists.json as a curated snapshot. Update it periodically.

Outputs

Prefer writing outputs to:

./daily_paper/arxiv_search/<field-slug>_<window>/

The markdown output should include:

• search field and time window
• total arXiv candidates
• total filtered papers
• ranking rule summary
• one row per paper with:
  • title
  • date
  • institutions
  • code link if present
  • GitHub stars if present
  • citation count
  • ranking metric actually used
  • arXiv link

Resources

• references/field_profiles.json: keyword profiles by field
• references/institution_allowlists.json: QS100 university snapshot, major companies, field-specific institution allowlists
• scripts/search_and_rank_arxiv.py: end-to-end search, enrich, filter, and rank script

Quick Start

python3 ./skills/arxiv-paper-screener/scripts/search_and_rank_arxiv.py \
  --field "spatial intelligence" \
  --window "2026-03" \
  --limit 20 \
  --outdir ./daily_paper/arxiv_search

Validation

Before finishing:

• Confirm the field resolved to a known profile or a reasonable keyword fallback.
• Confirm the time window was parsed correctly.
• Confirm institution filtering actually used the allowlist file.
• Confirm papers with GitHub links were ranked by stars.
• Confirm papers without code were ranked by citation count.
• If very few papers survive filtering, say so clearly and mention the allowlist may be too restrictive.