Graphify

What graphify is for

graphify is built around Andrej Karpathy's /raw folder workflow: drop anything into a folder - papers, tweets, screenshots, code, notes - and get a structured knowledge graph that shows you what you didn't know was connected.

Three things it does that Claude alone cannot:

1. Persistent graph - relationships are stored in graphify-out/graph.json and survive across sessions. Ask questions weeks later without re-reading everything.
2. Honest audit trail - every edge is tagged EXTRACTED, INFERRED, or AMBIGUOUS. You know what was found vs invented.
3. Cross-document surprise - community detection finds connections between concepts in different files that you would never think to ask about directly.

Use it for:

• A codebase you're new to (understand architecture before touching anything)
• A reading list (papers + tweets + notes → one navigable graph)
• A research corpus (citation graph + concept graph in one)
• Your personal /raw folder (drop everything in, let it grow, query it)

What You Must Do When Invoked

If no path was given, use . (current directory). Do not ask the user for a path.

Follow these steps in order. Do not skip steps.

Step 1 - Ensure graphify is installed

# Detect the correct Python interpreter (handles pipx, venv, system installs)
GRAPHIFY_BIN=$(which graphify 2>/dev/null)
if [ -n "$GRAPHIFY_BIN" ]; then
    PYTHON=$(head -1 "$GRAPHIFY_BIN" | tr -d '#!')
    case "$PYTHON" in
        *[!a-zA-Z0-9/_.-]*) PYTHON="python3" ;;
    esac
else
    PYTHON="python3"
fi
"$PYTHON" -c "import graphify" 2>/dev/null || "$PYTHON" -m pip install graphifyy -q 2>/dev/null || "$PYTHON" -m pip install graphifyy -q --break-system-packages 2>&1 | tail -3
# Write interpreter path for all subsequent steps (persists across invocations)
mkdir -p graphify-out
"$PYTHON" -c "import sys; open('graphify-out/.graphify_python', 'w').write(sys.executable)"

If the import succeeds, print nothing and move straight to Step 2.

In every subsequent bash block, replace python3 with $(cat graphify-out/.graphify_python) to use the correct interpreter.

Step 2 - Detect files

$(cat graphify-out/.graphify_python) -c "
import json
from graphify.detect import detect
from pathlib import Path
result = detect(Path('INPUT_PATH'))
print(json.dumps(result))
" > graphify-out/.graphify_detect.json

Replace INPUT_PATH with the actual path the user provided. Do NOT cat or print the JSON - read it silently and present a clean summary instead:

Corpus: X files · ~Y words
  code:     N files (.py .ts .go ...)
  docs:     N files (.md .txt ...)
  papers:   N files (.pdf ...)
  images:   N files
  video:    N files (.mp4 .mp3 ...)

Omit any category with 0 files from the summary.

Then act on it:

• If total_files is 0: stop with "No supported files found in [path]."
• If skipped_sensitive is non-empty: mention file count skipped, not the file names.
• If total_words > 2,000,000 OR total_files > 200: show the warning and the top 5 subdirectories by file count, then ask which subfolder to run on. Wait for the user's answer before proceeding.
• Otherwise: proceed directly to Step 2.5 if video files were detected, or Step 3 if not.

Step 2.5 - Transcribe video / audio files (only if video files detected)

Skip this step entirely if detect returned zero video files.

Video and audio files cannot be read directly. Transcribe them to text first, then treat the transcripts as doc files in Step 3.

Strategy: Read the god nodes from graphify-out/.graphify_detect.json (or the analysis file if it exists from a previous run). You are already a language model — write a one-sentence domain hint yourself from those labels. Then pass it to Whisper as the initial prompt. No separate API call needed.

However, if the corpus has only video files and no other docs/code, use the generic fallback prompt: "Use proper punctuation and paragraph breaks."

Step 1 - Write the Whisper prompt yourself.

Read the top god node labels from detect output or analysis, then compose a short domain hint sentence, for example:

• Labels: transformer, attention, encoder, decoder → "Machine learning research on transformer architectures and attention mechanisms. Use proper punctuation and paragraph breaks."
• Labels: kubernetes, deployment, pod, helm → "DevOps discussion about Kubernetes deployments and Helm charts. Use proper punctuation and paragraph breaks."

Set it as WHISPER_PROMPT to use in the next command.

Step 2 - Transcribe:

GRAPHIFY_WHISPER_MODEL=base  # or whatever --whisper-model the user passed
$(cat graphify-out/.graphify_python) -c "
import json, os
from pathlib import Path
from graphify.transcribe import transcribe_all

detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
video_files = detect.get('files', {}).get('video', [])
prompt = os.environ.get('GRAPHIFY_WHISPER_PROMPT', 'Use proper punctuation and paragraph breaks.')

transcript_paths = transcribe_all(video_files, initial_prompt=prompt)
print(json.dumps(transcript_paths))
" > graphify-out/.graphify_transcripts.json

After transcription:

• Read the transcript paths from graphify-out/.graphify_transcripts.json
• Add them to the docs list before dispatching semantic subagents in Step 3B
• Print how many transcripts were created: Transcribed N video file(s) -> treating as docs
• If transcription fails for a file, print a warning and continue with the rest

Whisper model: Default is base. If the user passed --whisper-model <name>, set GRAPHIFY_WHISPER_MODEL=<name> in the environment before running the command above.

Step 3 - Extract entities and relationships

Before starting: note whether --mode deep was given. You must pass DEEP_MODE=true to every subagent in Step B2 if it was. Track this from the original invocation - do not lose it.

This step has two parts: structural extraction (deterministic, free) and semantic extraction (Claude, costs tokens).

Run Part A (AST) and Part B (semantic) in parallel. Dispatch all semantic subagents AND start AST extraction in the same message. Both can run simultaneously since they operate on different file types. Merge results in Part C as before.

Note: Parallelizing AST + semantic saves 5-15s on large corpora. AST is deterministic and fast; start it while subagents are processing docs/papers.

Part A - Structural extraction for code files

For any code files detected, run AST extraction in parallel with Part B subagents:

$(cat graphify-out/.graphify_python) -c "
import sys, json
from graphify.extract import collect_files, extract
from pathlib import Path
import json

code_files = []
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
for f in detect.get('files', {}).get('code', []):
    code_files.extend(collect_files(Path(f)) if Path(f).is_dir() else [Path(f)])

if code_files:
    result = extract(code_files, cache_root=Path('.'))
    Path('graphify-out/.graphify_ast.json').write_text(json.dumps(result, indent=2))
    print(f'AST: {len(result[\"nodes\"])} nodes, {len(result[\"edges\"])} edges')