Daily paper recommendation workflow — search arXiv and Semantic Scholar, score and recommend papers
Help users discover the latest research papers by searching arXiv and Semantic Scholar, scoring them by relevance, recency, popularity, and quality, and generating a recommended papers list.
Execute the search script (scripts are located in server/scripts/research-news/):
cd server/scripts/research-news
python search_arxiv.py \
--config "$CONFIG_PATH" \
--output arxiv_filtered.json \
--max-results 200 \
--top-n 10 \
--categories "cs.AI,cs.LG,cs.CL,cs.CV,cs.MM,cs.MA,cs.RO"
Read arxiv_filtered.json containing scored and ranked papers.
Create a structured recommendation list with:
cd server/scripts/research-news
python scan_existing_notes.py --vault "$VAULT_PATH" --output existing_notes_index.json
python link_keywords.py --index existing_notes_index.json --input input.md --output output.md
All scripts are in server/scripts/research-news/:
search_arxiv.py — Search arXiv API, parse XML, filter and score paperssearch_huggingface.py — Search HuggingFace Daily Paperssearch_x.py — Search X (Twitter) for research newssearch_xiaohongshu.py — Search Xiaohongshu for research postsscan_existing_notes.py — Scan existing notes directory, build keyword indexlink_keywords.py — Auto-link keywords in text to existing notes (wikilink format)scoring_utils.py — Shared scoring utilitiescommon_words.py — Common words list for keyword filtering| Dimension | Weight | Description |
|---|---|---|
| Relevance | 40% | Keyword match in title/abstract, category match |
| Recency | 20% | Publication date (30d: +3, 90d: +2, 180d: +1) |
| Popularity | 30% | Citation count / influence |
| Quality | 10% | Innovation indicators from abstract |
Based on evil-read-arxiv — an automated paper reading workflow. MIT License.