Freamon

* Enhanced Polars-optimized supervised deduplication:
* Improved robustness and error handling for production use
* Fixed numpy divide warnings in correlation calculations
* Enhanced feature name handling for ensemble models
* Added adaptive thresholding for improved duplicate detection
* Fixed model compatibility for RandomForest and GradientBoosting models
* Added comprehensive benchmarking tools and examples
* Enhanced feature contribution calculation for better explainability
* Improved chunked processing to handle large datasets more efficiently
* Added support for mixed data types and diverse feature sets
* Fixed edge cases in cross-batch duplicate detection
* Added integration tests for deduplication pipeline with Polars optimization:
* End-to-end testing of LSH and supervised models together
* Cross-validation of results between pandas and Polars implementations
* Mixed dataframe type testing (pandas/Polars interoperability)
* Advanced features integration testing
* Added comprehensive example scripts:
* Performance benchmarking between pandas and Polars implementations
* Detailed supervised deduplication examples with visualization
* End-to-end duplicate detection workflow examples
* Demonstration of advanced features and optimizations

* Added Polars-optimized deduplication for improved performance:
* Optimized LSH deduplication with 2-5x performance improvements
* Memory-efficient processing of large datasets (60-70% less memory usage)
* Streaming deduplication for datasets that don't fit in memory
* Batch processing to handle very large text collections
* Integration with existing deduplication framework
* Added Polars-optimized supervised deduplication:
* Optimized feature generation for record pairs
* Enhanced batch processing for large datasets
* Maintained all advanced features (active learning, incremental learning, etc.)
* Chunked processing for memory efficiency
* Graceful fallback to pandas implementation
* Added Polars-optimized text utilities:
* Batch text processing for large collections
* Optimized text vectorization workflows
* Memory-efficient similarity calculations
* Parallel processing capabilities
* Text column deduplication and processing utilities
* Enhanced dataframe utilities:
* Improved conversion between pandas and Polars
* Better datatype optimization
* Chunked processing for large datasets
* Enhanced datetime detection

* Added supervised deduplication enhancements:
* Active learning for efficient labeling of ambiguous duplicate pairs
* Incremental learning for continually updating models with new data
* Entity resolution framework with blocking strategies for large datasets
* Advanced explainability for duplicate detection decisions
* Ensemble deduplication methods combining multiple modeling approaches
* Automatic threshold optimization with business impact analysis
* Improved deduplication reporting with interactive visualizations

* Fixed serialization issue with optimized topic models
* Added documentation for topic model export functionality
* Enhanced index tracking for consistent processing of large datasets

* Added split EDA reports for improved memory efficiency
* Enhanced automated modeling workflows with better performance metrics
* Fixed export functionality to support various output formats
* Added more comprehensive examples for advanced use cases

* Fixed an issue with date format detection in mixed datasets
* Enhanced support for international date formats
* Added support for Australian date patterns
* Improved memory efficiency for large datasets
* Enhanced reporting capabilities with new visualization options