Transforms raw source files registered in wiki_sources.json into Karpathy-style LLM wiki nodes inside the wiki-root. Generates concept pages, cluster pages, index, and table of contents. Use when building or rebuilding the wiki from raw content.
Requires pyyaml and Python 3.8+. Also requires rlm-factory plugin installed.
pip install -r requirements.txt
Status: Active Author: Richard Fremmerlid Domain: Obsidian Wiki Engine
Transforms raw source documents (registered via wiki_sources.json) into
Karpathy-style wiki nodes inside the designated wiki_root. Each concept
gets its own {concept}.md node with wikilinks, cross-references, and an
RLM summary layer.
{wiki_root}/
wiki/
_index.md ← master concept index
_toc.md ← table of contents
_{cluster}.md ← per-topic cluster page
{concept}.md ← individual wiki node
rlm/
{concept}/
summary.md ← 1-5 sentence distilled summary
bullets.md ← key idea bullets
deep.md ← full multi-pass distillation
meta/
wiki_sources.json ← raw source registry (created by wiki-init)
config.yaml ← wiki settings
agent-memory.json ← stale file tracking
wiki_builder.py runs a 3-step pipeline internally:
ingest.py → concept_extractor.py → wiki node formatting
.md files
with wikilinks, cluster pages, _index.md, and _toc.mdWhen two source files (e.g. arch-docs/auth.md and research/oauth.md) produce
the same concept slug, they are merged into one wiki node that:
source_files frontmattermulti_source: true for identificationThis implements the core Karpathy "compile" metaphor: N raw files → M concept nodes (M ≤ N).
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root --source arch-docs
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root \
--rlm-cache-dir /path/to/project/.agent/learning/rlm_wiki_cache
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root --dry-run
Each concept node uses this structure:
---