Gptme-rag

What's New

Features
- ⚡️ Performance optimization: only index new/modified files
- Track file modification times in metadata
- Skip unchanged files during indexing
- Improved indexing efficiency
- 📊 Added progress reporting with tqdm for document indexing
- Better visibility during large indexing operations
- Accurate batch progress tracking
- 🔍 Enhanced search functionality
- Added scoring explanations and custom weights
- Improved document chunking with better overlap
- Better handling of chunk grouping

Improvements
- 🔄 Enhanced file watcher reliability
- Added robust error handling and retries
- Improved file processing logic
- Better logging for debugging
- 📝 Better error handling and logging throughout
- Replaced debug prints with proper logging
- More descriptive error messages
- Improved debugging capabilities

Technical Improvements
- 🧪 Improved test stability and coverage
- Added shared indexer fixture with auto-cleanup
- Use unique collection names per test
- More reliable test cleanup
- 🔧 Refactored indexer for better efficiency
- Split indexing into collection and processing phases
- Better code organization and readability
- Improved error handling

Co-authored-by: Bob <bobsuperuserlabs.org>

What's New

Features
- 🎨 Added rich logging and syntax highlighting for search results
- New --raw flag to optionally disable highlighting
- Improved document display and formatting
- Better indentation and code blocks in results
- 🔍 Enhanced search functionality
- Added path filtering capabilities
- Improved search results formatting
- Better handling of code blocks with language detection
- 📊 Added status command to show index statistics
- 💾 Fixed and enabled persistence in CLI

Improvements
- 📝 Documentation
- Added explanation of RAG for newcomers
- Added quick start section with basic commands
- Clarified Python version requirement
- Expanded gptme integration section
- Simplified command examples
- Added 'built with gptme' badge

Technical Improvements
- ⚡️ Better indexing performance
- Reduced batch size for better progress visibility
- Increased file limit default from 100 to 1000
- Improved logging messages and levels
- Enhanced gitignore pattern handling
- 🐛 Fixed ChromaDB warnings about existing IDs
- 🔧 Improved pre-commit test hook efficiency

Features

- 📚 Document indexing with ChromaDB
- Fast and efficient vector storage
- Semantic search capabilities
- Persistent storage
- 🔍 Semantic search with embeddings
- Relevance scoring
- Token-aware context assembly
- Clean output formatting
- 📄 Smart document processing
- Streaming large file handling
- Automatic document chunking
- Configurable chunk size/overlap
- Document reconstruction
- 👀 File watching and auto-indexing
- Real-time index updates
- Pattern-based file filtering
- Efficient batch processing
- Automatic persistence
- 🛠️ CLI interface for testing and development
- Index management
- Search functionality
- Context assembly
- File watching

Changes

- Fixed collection setup
- Improved indexing robustness and type safety
- Fixed return value of index_directory to return number of documents indexed