swain-search

If existing troves contain relevant sources:

1. Report what was found — show the trove ID, matching source titles, and relevant excerpts
2. Suggest extend over create — if an existing trove covers the same topic, extend it rather than creating a parallel trove
3. Cross-link — if the topic is adjacent but distinct, create a new trove but note the related trove in synthesis.md

This step runs in all modes (Create, Extend, Discover) and before any web searches. Existing trove content is always checked first.

Create mode

Build a new trove from scratch.

Step 1 — Gather inputs

Ask the user (or infer from context) for:

1. Trove ID — a slug for the topic (e.g., websocket-vs-sse). Suggest one if the context is clear.
2. Tags — keywords for discovery (e.g., real-time, websocket, sse)
3. Sources — any combination of:
   • Web search queries ("search for WebSocket vs SSE comparisons")
   • URLs (web pages, forum threads, docs)
   • Video/audio URLs
   • Local file paths
4. Freshness TTL overrides — optional, defaults are fine for most troves

If invoked from swain-design (e.g., spike entering Active), the artifact context provides the topic, tags, and sometimes initial sources.

Step 2 — Collect and normalize

For each source, use the appropriate capability. Read skills/swain-search/references/normalization-formats.md for the exact markdown structure per source type.

Web search queries:

1. Use a web search capability to find relevant results
2. Select the top 3-5 most relevant results
3. For each: fetch the page, normalize to markdown per the web page format
4. If no web search capability is available, tell the user and skip

Web page URLs:

1. Fetch the page using a browser or page-fetching capability
2. Strip boilerplate (nav, ads, sidebars, cookie banners)
3. Normalize to markdown per the web page format
4. If fetch fails, record the URL in manifest with a failed: true flag and move on

Paywall proxy fallback:

After fetching a web page, check if a paywall proxy is available for the URL's domain:

1. Run skills/swain-search/scripts/resolve-proxy.sh <url>
   • Exit 1: no proxy configured — use the direct fetch content as-is
   • Exit 0: outputs PROXY:<name>:<proxy-url> and SIGNAL:<text> lines
2. If exit 0, check the fetched content for each SIGNAL text (case-sensitive literal match)
3. If any signal matches (or the article body is under ~200 words):
   • Log: "Paywall detected for <url> — trying proxy fallback"
   • Try each PROXY URL in order, fetching via the same page-fetching capability used for web pages
   • First proxy that returns substantive content (more than the truncated original) wins
   • Set proxy-used: <name> and notes: "Full article retrieved via <name> proxy" in the manifest entry
4. If no signals match: use the direct fetch content as-is (no proxy needed)
5. If all proxies fail: keep the original truncated content, set notes: "Paywalled; proxies exhausted — content from direct fetch only"

The registry lives at skills/swain-search/references/paywall-proxies.yaml. Add new domains or proxies there — no skill file changes needed.

Video/audio URLs:

1. Use a media transcription capability to get the transcript
2. Normalize to markdown per the media format (timestamps, speaker labels, key points)
3. If no transcription capability is available, tell the user and skip — or accept a pre-made transcript

Local files:

1. Use a document conversion capability (PDF, DOCX, etc.) or read directly if already markdown
2. Normalize per the document format
3. For markdown files: add frontmatter only, preserve content

Forum threads / discussions:

1. Fetch and normalize per the forum format (chronological, author-attributed)
2. Flatten nested threads to chronological order with reply-to context

Repositories:

1. Clone or read the repository contents
2. Mirror the original directory tree under sources/<source-id>/
3. Default: mirror the full tree. For large repositories (thousands of files), ingest selectively and set selective: true in the manifest entry
4. Populate the highlights array with paths to the most important files (relative to the source-id directory)

Documentation sites:

1. Crawl or fetch the documentation site
2. Mirror the section hierarchy under sources/<source-id>/
3. Default: mirror the full site. For large sites, ingest selectively and set selective: true
4. Populate the highlights array with paths to the most important pages
5. Preserve internal link structure where possible

Each normalized source gets a slug-based source ID and lives in a directory-per-source layout:

• Flat sources (web, forum, media, document, local): sources/<source-id>/<source-id>.md
• Hierarchical sources (repository, documentation-site): sources/<source-id>/ with the original tree mirrored inside

Source ID generation:

• Derive the source ID as a slug from the source title or URL (e.g., mdn-websocket-api, strangeloop-2025-realtime)
• When a slug collides with an existing source ID: append __word1-word2 using two random words from skills/swain-search/references/wordlist.txt
• If the wordlist is missing, append __ followed by 4 hex characters (e.g., __a3f8) as a fallback

Step 3 — Generate manifest

Create manifest.yaml following the schema in skills/swain-search/references/manifest-schema.md. Include:

• Trove metadata (id, created date, tags)
• Default freshness TTL per source type
• One entry per source with provenance (URL/path, fetch date, content hash, type)

Compute content hashes as bare hex SHA-256 digests (no prefix) of the normalized markdown content:

shasum -a 256 sources/mdn-websocket-api/mdn-websocket-api.md | cut -d' ' -f1

Step 4 — Generate synthesis

Create synthesis.md — a structured distillation of key findings across all sources.

Structure the synthesis by theme, not by source. Group related findings together, cite sources by ID, and surface:

• Key findings — what the sources collectively say about the topic
• Points of agreement — where sources converge
• Points of disagreement — where sources conflict or present alternatives
• Gaps — what the sources don't cover that might matter

Keep it concise. The synthesis is a starting point, not a comprehensive report — the user or artifact author will refine it.

Step 5 — Commit and stamp

Use the dual-commit pattern (same as swain-design lifecycle stamps) to give the trove a reachable commit hash.

Before Commit A — append a history entry to manifest.yaml with a -- placeholder for the commit hash: