Index Freshness

Staleness Detection

Indicators of Stale Indexes

1. Search misses - Can't find code you know exists
2. Wrong summaries - Descriptions don't match current code
3. Missing functions - New functions not in index
4. Old health data - Issues already fixed still showing

Automatic Detection

The system checks:

Files modified since last scan:  15
Last health scan:               3 days ago
Last embedding update:          5 days ago
Git commits since index:        12

Freshness Check Workflow

1. Check Last Update Times

memory_health action=status project=gyst

Returns:

• Last scan timestamp
• Files changed since
• Recommended action

2. Identify Changed Files

The system tracks:

• File modification times
• Git commits since last index
• Added/deleted/modified files

3. Decide: Incremental or Full Reindex

Incremental (fast, <20 files changed):

• Update only changed files
• Preserve unchanged embeddings

Full Reindex (slower, but thorough):

• When >20 files changed
• After major refactors
• If indexes seem corrupted

Reindex Triggers

Automatic (via Watcher)

• File saved → summary updated
• New file → added to index
• File deleted → removed from index

Manual Triggers

# Check freshness
python3 ~/.claude-dash/mlx-tools/freshness_checker.py /path/to/project project-id

# Sync embeddings
python3 ~/.claude-dash/mlx-tools/embedding_sync.py project-id

# Full reindex (via watcher)
~/.claude-dash/watcher/start-watcher.sh reindex project-id

When to Reindex

Troubleshooting Stale Indexes

"Search can't find X"

1. Check if file exists
2. Check if file is in summaries.json
3. Check file modification time vs index time
4. Trigger reindex if needed

"Embeddings seem wrong"

1. Check embeddings_v2.json timestamp
2. Check if embedding provider changed
3. Consider full embedding regeneration

"Health shows fixed issues"

1. Check last scan timestamp
2. Run new health scan
3. Clear old health.json and rescan

MCP Tools

# Check health status (includes freshness)
memory_health action=status project=gyst

# Trigger scan
memory_health action=scan project=gyst

# Workers for maintenance
workers_run worker=freshness project=gyst
workers_run worker=consolidate

Best Practices

1. Let the watcher handle routine updates
2. Reindex after major refactors
3. Check freshness when search seems off
4. Don't reindex unnecessarily - it takes resources

Index File Sizes (Reference)

Typical sizes for a medium project:

summaries.json     ~500KB
functions.json     ~200KB
embeddings_v2.json ~5MB
health.json        ~50KB

If files seem too small, index may be incomplete.