Docs Scraping

• Per-page files: <stable-page-stem>.ext.md
• Index file: index.ext.md

Behavior requirements

1. Resolve source metadata from the user request:
   • source key (safe folder/script slug)
   • docs root URL
   • discovery method (sitemap.xml preferred)
2. Reuse project script conventions:
   • Bun shebang (#!/usr/bin/env bun)
   • *.sh.ts naming
   • helpers under scripts/helpers/
3. Use resilient scraping strategy:
   • Primary: r.jina.ai markdown proxy
   • Fallback: direct HTML fetch + conversion to markdown
4. Normalize filenames from docs paths:
   • deterministic flattening (for example docs__guides__intro.ext.md)
5. Regenerate output cleanly:
   • remove old *.ext.md in target source directory
   • write fresh per-page files and index.ext.md

Generated page format

• Top ---- section with summary metadata:
  • captured timestamp
  • source root
  • source page URL/path
  • keywords
  • concise summary
• Middle body with markdown content snapshot
• Bottom ---- notes/comments/lessons section

Index format

• Top ---- section with source/capture metadata and summary totals
• Full page inventory with links to local page files and capture status
• Bottom ---- notes/comments/lessons section

Implementation workflow

1. Inspect existing scraper scripts for reuse patterns (scripts/scrapes/scrape_*.sh.ts).
2. Create or update scripts/scrapes/scrape_<source>_docs.sh.ts.
3. Run the script once to generate docs output.
4. Report totals (pages, ok, failed) and notable blocked pages.
5. If a new script entrypoint was introduced, update README.md and AGENTS.md.

Safety constraints

• Never embed secrets, auth headers, or private tokens in script or output files.
• Skip private/authenticated docs pages unless explicit credentials handling is requested and safe.
• Keep scripts idempotent and deterministic where practical.

Return checklist

• Script path(s) added/updated
• Output directory generated/refreshed
• Discovery strategy used
• Capture totals and notable failures
• Exact rerun command (bun scripts/scrapes/scrape_<source>_docs.sh.ts)