Knowledge Base Manager

This keeps _surface.md short no matter how large the wiki grows.

Step 1: Detect Mode

First, confirm the knowledge base path. If the user hasn't specified one, ask: "Where should the knowledge base live?"

Once you have the path, detect the current mode:

Pre-check: If the user explicitly asks for a health check or lint (e.g. "check my wiki", "is my knowledge base healthy"), go directly to Mode 4: Lint regardless of inbox state or other signals.

Otherwise, follow this priority:

1. Path doesn't exist or directory is empty → First-time creation (fresh)
2. Path exists with files but no _surface.md → First-time creation (migrate existing docs)
3. Structure is complete + inbox/ has files → Incremental mode (digest from inbox)
4. Structure is complete + user provided content or URL in conversation → Incremental mode (from conversation/URL)

Mode 1: First-Time Creation

Fresh Knowledge Base

Execute in order:

1. Create directory structure (inbox/, wiki/)
2. Generate README.md (use template below, replace knowledge base name)
3. Generate _rules.md (use template below)
4. Create _surface.md with header line only
5. Create log.md with first entry
6. Tell the user: "Knowledge base created. Drop materials into inbox/ or tell me what to organize."

Migrate Existing Documents

The directory already has files but isn't in knowledge base format:

1. Create any missing structure files (_surface.md, _rules.md, README.md, log.md, inbox/, wiki/)
2. Read every existing file one by one — do not skip any
3. Classify each file:
   • Already organized knowledge (clear topic, structured) → move to wiki/, add index line to _surface.md
   • Raw material (unstructured notes, downloaded articles) → move to inbox/, then digest using incremental mode
   • Rules or documentation → extract useful content into _rules.md or README.md
4. After all files are processed, update log.md

Mode 2: Incremental Ingestion

Detect Source

Check in this priority order:

1. User gave a URL in conversation → Source C (fetch from URL)
2. User gave text content in conversation → Source B (conversation content)
3. Neither of the above → check if inbox/ has files → Source A (inbox digestion)
4. Inbox is also empty → tell user: "Nothing to process. Drop materials in inbox/ or give me content directly."

Source A: Digest from Inbox

1. List all files in inbox/
2. Process each file:
   • Read the full content
   • Extract core knowledge, remove redundancy
   • Check the current index to find related existing pages:
     • Flat phase: read _surface.md
     • Categorized phase: read _surface.md to identify the relevant category, then read that category's _index.md
   • Related page exists → update the existing wiki page with new information
   • No related page → create a new page:
     • Flat phase: in wiki/
     • Categorized phase: in the appropriate subfolder. If the content doesn't fit any existing category, create a new subfolder with its own _index.md and add a new category entry to _surface.md
   • Update the index:
     • Flat phase: update _surface.md
     • Categorized phase: update the subfolder's _index.md (add entry, update the category summary and last-updated line), and update the category line in _surface.md if the description needs adjusting
   • Bubble-up check: after updating a wiki page, re-read the parent index (_index.md or _surface.md) and verify its category summary still accurately reflects the content. If not, update it. Knowledge flows bottom-up: page changes propagate upward through the index hierarchy
   • Move the processed file to inbox/_processed/ (never delete originals — they serve as the source of truth if re-ingestion is needed)
   • Append a record to log.md

Source B: From Conversation

User provided text content directly in the conversation:

1. Understand the content, extract core knowledge
2. Check the index to find related existing pages (same flat/categorized logic as Source A)
3. Create new page or update existing page
4. Update the appropriate index (_surface.md or subfolder _index.md) and log.md. Run bubble-up check (same as Source A)

Source C: From URL

User provided a link:

1. Use WebFetch to retrieve the page content
2. Based on content length:
   • Short (< 3000 words) → save original to inbox/ as backup, extract knowledge into wiki/
   • Long (≥ 3000 words) → summarize, write only core content to wiki/, keep original URL in page frontmatter
   • Fetch failed → tell user "Can't access this link", ask them to paste the content manually
3. Move the saved original from inbox/ to inbox/_processed/ (same as Source A — keep the traceability chain)
4. Update the appropriate index (_surface.md or subfolder _index.md) and log.md. Run bubble-up check (same as Source A)

Mode 3: Auto-Reorganization

After every incremental operation, check the file count in wiki/:

Reorganization steps (must get user consent first):

1. Read all wiki page titles and _surface.md content
2. Cluster by topic, propose 3–7 subfolder names
3. Show the plan to the user, e.g.:

   Proposed organization:
   - crypto/ (12 files): on-chain analysis, token research
   - tools/ (8 files): tool guides, API docs
   - research/ (6 files): paper summaries, study notes
   

4. After user confirms:
   • Create subfolders
   • Create _index.md in each new subfolder (migrate the relevant entries from _surface.md)
   • Move files into their subfolders
   • Rewrite _surface.md: replace individual page entries with category-level entries pointing to each subfolder's _index.md
   • Update cross-references ([[]] links) in wiki pages to reflect new paths
   • Record the reorganization in log.md

Category splitting: If any single subfolder exceeds 30 pages, suggest splitting it into sub-categories (e.g. crypto/ → crypto/defi/, crypto/mev/). The same reorganization rules apply: show the plan, get user consent, then execute. Each new sub-category gets its own _index.md, and the parent _index.md is rewritten to list sub-categories instead of individual pages.

Important: Any operation that moves, renames, or restructures files requires showing the plan and getting explicit user approval before execution.

Mode 4: Lint (Health Check)

When the user asks to check the knowledge base health:

Pre-check: First verify the knowledge base structure exists at the given path. If the path doesn't exist, has no _surface.md, or is missing wiki/, tell the user: "No valid knowledge base found at this path. Want me to create one?" Do not proceed with lint checks on a non-existent or incomplete structure.

If the structure is valid, run all checks below and present a report.

Check 1: Orphan Pages

Wiki pages that exist as files but have no entry in any index (_surface.md or _index.md).

• Scan all .md files under wiki/ (excluding _index.md files)
• For each file, verify it appears in _surface.md (flat phase) or the parent folder's _index.md (categorized phase)
• Report any unindexed pages

Check 2: Broken References

Index entries or [[]] cross-references that point to pages that don't exist.

• Read _surface.md and all _index.md files, check every link target exists as a file
• Read all wiki pages, check every [[page-name]] reference resolves to an existing file
• Report any broken links

Check 3: Duplicate / Overlapping Pages

Multiple pages covering the same topic, potentially with conflicting information.

• Read _surface.md (and _index.md files if categorized) to get all page titles and descriptions
• Identify pages with very similar titles or descriptions
• For suspected duplicates, read the first 2–3 paragraphs of each page to confirm whether they genuinely overlap (title similarity alone is not reliable — e.g. "DeFi Basics" and "DeFi Introduction" might be the same, or might cover different aspects)
• Report confirmed overlapping pairs, noting any contradictions between them

Check 4: Stale Pages

Pages with outdated content based on their updated frontmatter date.

• Read frontmatter of all wiki pages
• Flag pages where updated is more than 90 days ago
• Report with the date, so the user can decide what to refresh

Check 5: Index Consistency

Mismatches between the index and the actual file structure.

• In categorized phase: verify every subfolder in wiki/ has an _index.md
• Verify page counts in _surface.md category entries match actual file counts
• Check for empty subfolders

Presenting the Report

After running all checks, present results to the user in this format:

Knowledge Base Health Report
=============================

Orphan Pages (not in any index):
- wiki/forgotten_note.md

Broken References:
- _surface.md → wiki/deleted_page.md (file missing)
- wiki/solana_mev.md → [[old_page]] (target missing)

Potential Duplicates:
- wiki/defi_basics.md ↔ wiki/defi_intro.md (overlapping content about DeFi fundamentals)

Stale Pages (not updated in 90+ days):
- wiki/old_research.md (last updated: 2025-12-01)

Index Issues:
- wiki/crypto/ has no _index.md
- _surface.md says crypto/ has 15 pages, actually has 18

✅ No issues: [list any checks that passed clean]

After the report, ask the user: "Want me to fix any of these issues?" Then only act on what the user approves. Fixes may include:

• Adding orphan pages to the index
• Removing broken references or creating the missing pages
• Merging duplicate pages (show a merged draft first)
• Updating the updated date after reviewing stale pages
• Fixing index counts and adding missing _index.md files

Record the lint operation in log.md.

File Format Specifications

Wiki Page Format

---