We are thrilled to announce the release of **HypEx v0.1.4**! This update brings significant improvements in matching speed, especially when dealing with large datasets.
Key Features and Improvements
- **Enhanced Matching Performance**:
- We have optimized the matching process, achieving a performance boost by an order of magnitude. This enhancement will be particularly beneficial for datasets exceeding one million entries.
- **Introduction of FAISS Algorithm**:
- For datasets larger than one million entries, we've implemented the [FAISS](https://github.com/facebookresearch/faiss) algorithm. FAISS (Facebook AI Similarity Search) provides a substantial increase in speed by using efficient nearest neighbor search techniques.
- **Trade-offs**:
- **Speed vs. Accuracy**: The new algorithm improves speed but slightly compromises on accuracy and reproducibility. This is because FAISS first performs random clustering before searching for nearest neighbors, which leads to faster results compared to the exhaustive search used in previous versions.
Benefits
- **Faster Processing**: Users with large-scale data can now experience significantly reduced computation times, enabling faster data processing and analysis.
- **Scalability**: The adoption of FAISS makes HypEx more scalable, allowing users to work with larger datasets efficiently.
Potential Considerations
- **Accuracy**: While the speed improvements are substantial, users should be aware of the trade-off in accuracy. In scenarios where precise matching is crucial, users might want to evaluate the results to ensure they meet their needs.
- **Reproducibility**: Due to the nature of the FAISS algorithm, results may vary slightly between runs. Users should consider this when reproducibility is critical.