Ingesting Rag Content

Level 3 Resources

Scripts

• scripts/validate_ingestion.py: Runs a health check on the Qdrant ebook_library collection.
• scripts/ai/rag/rag_optimized.py: The core ingestion engine logic.

References

• references/rag-architecture.md: Explains the Parent-Child chunking strategy and Qdrant storage schema.

Important Rules

• PDF Only: In the current optimized version, only PDF documents are supported for Parent-Child splitting.
• UTF-8 Only: Ensure environment variables ($env:PYTHONIOENCODING="utf-8") are set.
• Incremental indexing: The system supports adding to existing Qdrant collections.
• Local Embeddings: Uses BAAI/bge-small-en-v1.5 (via FastEmbed) for performance.

Best Practices

• Recursive Splitting: Always use recursive character splitting to maintain paragraph integrity.
• Absolute Paths: Store absolute paths in metadata to ensure reliable document opening.
• Validation: Run scripts/validate_ingestion.py after bulk ingestion.