Data Ingest — Universal Text Source Handler

This applies to all formats — JSON, chat logs, HTML, plaintext, and images alike.

Step 1: Identify the Source Format

Read the file(s) the user points you at. Common formats you'll encounter:

Common Chat Export Formats

ChatGPT export (conversations.json):

[{"title": "...", "mapping": {"node-id": {"message": {"role": "user", "content": {"parts": ["text"]}}}}}]

Slack export (directory of JSON files per channel):

[{"user": "U123", "text": "message", "ts": "1234567890.123456"}]

Generic chat log (timestamped text):

[2024-03-15 10:30] User: message here
[2024-03-15 10:31] Bot: response here

Don't try to handle every format upfront — read the actual data, figure out the structure, and adapt.

Images and visual sources

When the user dumps a folder of screenshots, whiteboard photos, or diagram exports, treat each image as a source:

• Use the Read tool on the image path — it will render the image into context.
• Transcribe any visible text verbatim (this is the only extracted content from an image).
• Describe structure: for diagrams, list nodes/edges; for screenshots, name the app and what's on screen.
• Extract the concepts the image conveys — what's it about? Most of this is ^[inferred].
• Flag anything you can't read, can't identify, or are guessing at with ^[ambiguous].

Image-derived pages will skew heavily inferred — that's expected and the provenance markers will reflect it. Set source_type: "image" in the manifest entry. Skip files with EXIF-only changes (re-saved with no visual diff) — compare via the standard delta logic.

For folders of mixed images (e.g. a screenshot timeline of a debugging session), cluster by visible topic rather than per-file. Twenty screenshots of the same UI bug should produce one wiki page, not twenty.

Step 2: Extract Knowledge

Regardless of format, extract the same things:

• Topics discussed — what subjects come up?
• Decisions made — what was concluded or decided?
• Facts learned — what concrete information is stated?
• Procedures described — how-to knowledge, workflows, steps
• Entities mentioned — people, tools, projects, organizations
• Connections — how do topics relate to each other and to existing wiki content?

For conversation data specifically:

Focus on the substance, not the dialogue. A 50-message debugging session might yield one skills page about the fix. A long brainstorming chat might yield three concept pages.

Skip:

• Greetings, pleasantries, meta-conversation ("can you help me with...")
• Repetitive back-and-forth that doesn't add new information
• Raw code dumps (unless they illustrate a reusable pattern)

Step 3: Cluster and Deduplicate

Before creating pages:

• Group extracted knowledge by topic (not by source file or conversation)
• Check existing wiki pages — does this knowledge belong on an existing page?
• Merge overlapping information from multiple sources
• Note contradictions between sources

Step 4: Distill into Wiki Pages

Follow the wiki-ingest skill's process for creating/updating pages:

• Use correct category directories (concepts/, entities/, skills/, etc.)
• Add YAML frontmatter with title, category, tags, sources
• Use [[wikilinks]] to connect to existing pages
• Attribute claims to their source
• Write a summary: frontmatter field on every new page (1–2 sentences, ≤200 characters) answering "what is this page about?" — this is what downstream skills read to avoid opening the page body.
• Apply provenance markers per the convention in llm-wiki. Conversation, log, and chat data tend to be high-inference — you're often reading between the turns to extract a coherent claim. Be liberal with ^[inferred] for synthesized patterns and with ^[ambiguous] when speakers contradict each other or you're unsure who's right. Write a provenance: frontmatter block on each new/updated page.

Step 5: Update Manifest and Special Files

.manifest.json — Add an entry for each source file processed:

{
  "ingested_at": "TIMESTAMP",
  "size_bytes": FILE_SIZE,
  "modified_at": FILE_MTIME,
  "source_type": "data",  // or "image" for png/jpg/webp/gif sources
  "project": "project-name-or-null",
  "pages_created": ["list/of/pages.md"],
  "pages_updated": ["list/of/pages.md"]
}

index.md and log.md:

- [TIMESTAMP] DATA_INGEST source="path/to/data" format=FORMAT pages_updated=X pages_created=Y

Tips

• When in doubt about format, just read it. The Read tool will show you what you're dealing with.
• Large files: Read in chunks using offset/limit. Don't try to load a 10MB JSON in one go.
• Multiple files: Process them in order, building up wiki pages incrementally.
• Binary files: Skip them, except images — those are first-class sources via the Read tool's vision support.
• Encoding issues: If you see garbled text, mention it to the user and move on.