Allyanonimiser

Added
- Configuration export/import functionality for sharing settings between users
- Added `export_config()` method to export settings to JSON or YAML files
- Enhanced settings manager with functions to create shareable configurations
- Added support for exporting and importing configuration files
- Created example_export_config.py demonstrating configuration sharing
- Added comprehensive test suite for configuration export functionality
- Added PyArrow integration for improved DataFrame performance
- Graceful fallback for environments without PyArrow installed
- Configurable PyArrow usage through settings with sensible defaults

Enhanced
- Modified DataFrameProcessor class to utilize PyArrow when available
- Updated README with comprehensive documentation on configuration sharing
- Improved error handling in configuration import/export

Added
- DataFrameProcessor class for efficient pandas DataFrame processing
- Batch processing for handling large DataFrames with optimized memory usage
- Multi-processing capabilities for improved performance
- Comprehensive DataFrame anonymization functionality
- Statistical analysis tools for entity detection results
- Progress tracking with tqdm for long-running operations
- Extended main interface with DataFrame convenience methods
- Added example_dataframe.py demonstrating DataFrame processing

Enhanced
- Added pandas dependency to package requirements
- Implemented test suite for DataFrame processing
- Added documentation for DataFrame processing in README
- Optimized entity detection algorithms for batched data

Added
- Comprehensive custom pattern creation and management functionality
- PatternManager class for handling pattern collections and application
- PatternRegistry class for saving and loading patterns
- Enhanced CustomPatternDefinition with serialization capabilities
- Support for creating patterns from examples with different generalization levels
- Added pattern creation helpers in main interface
- Implemented pattern persistence with JSON storage
- Added example script demonstrating custom pattern usage

Enhanced
- Improved entity type metadata for custom patterns
- Enhanced pattern explanation capabilities
- Extended public API with pattern management methods
- Improved pattern testing and validation support

Added
- Integrated spaCy's NER system for more accurate PERSON entity detection
- Added entity type mapping between spaCy NER and custom entity types
- Improved detection of names with contextual understanding

Fixed
- Reduced false positives by prioritizing spaCy NER results over pattern matching for PERSON entities
- Fixed issues with conflicting entity types for the same text span
- Addressed false positives for phrases like "Ref Number" being detected as PERSON entities
- Added proper entity type resolution with configurable prioritization rules

Fixed
- Fixed import issue with `create_simple_generalized_regex` function in package
- Corrected relative vs. absolute imports in pattern generation functionality
- Added missing pattern generation functions to package structure
- Fixed syntax errors in regex pattern matchers

Added
- Implemented multi-level pattern generalization system with four levels of flexibility
- Added intelligent format detection for common patterns (dates, emails, phone numbers)
- Created prefix/suffix analysis and character class recognition algorithms
- Added `generalization_level` parameter to `create_pattern_from_examples`
- Added comprehensive docstrings with detailed examples
- Created example script `example_advanced_pattern_generation.py`
- Exposed additional pattern generation helper functions

Enhanced
- Improved pattern matching capability for variations of examples
- Made generated patterns more flexible with character classes
- Enabled automatic detection of common format patterns

Fixed
- Fixed imports in the pattern generator modules
- Improved error handling in pattern generation functions
- Fixed path issues in the pattern creation utility functions