Highlights
- New model topology feature: ISQ and device mapping
- 🔥Faster FlashAttention support when batching
- Removed `plotly` and associated JS dependencies
- φ³ Support Phi 3.5, Phi 3.5 vision, Phi 3.5 MoE
- Improved Rust API ergonomics
- Support multiple (shaded) GGUF files
MSRV
The Rust MSRV of this version is 1.79.0
What's Changed
* Fixes for auto dtype selection with RUST_BACKTRACE=1 by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/690
* Add support multiple GGUF files by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/692
* Refactor normal and vision loaders by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/693
* Fix `split.count` GGUF duplication handling by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/695
* Batching example by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/694
* Some fixes by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/697
* Improve vision rust examples by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/698
* Add ISQ topology by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/701
* Add custom logits processor API by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/702
* Add Gemma 2 PagedAttention support by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/704
* Faster RmsNorm in Gemma/Gemma2 by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/703
* Fix bug in Metal ISQ by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/706
* Support GGUF BF16 tensors by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/691
* Better support for FlashAttention: real batching + sliding window + softcap by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/707
* Remove some usages of `pub` in models by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/708
* Support the Phi 3.5 V model by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/710
* Implement the Phi 3.5 MoE model by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/709
* Device map topology by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/717
* Implement DRY penalty by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/637
* Remove plotly and just output CSV loss file by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/700
* Using once_cell to reduce MSRV by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/724
* Fixes for Windows build by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/729
* Even more phi3.5moe fix attempts by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/731
* Add example for Phi 3.5 MoE by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/733
* Add Phi 3.5 chat template by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/734
* Patch ISQ for Mixtral by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/730
* Gracefully handle Engine Drop with termination request by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/735
* feat(vision): add support for proper file and data image URLs by Schuwi in https://github.com/EricLBuehler/mistral.rs/pull/727
* Add new parsing to Python API by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/737
* Remove test and add custom error type to Python API by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/738
* Update kernels for metal bf16 by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/719
* Better `Response` Result API by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/739
* More Metal quantized kernel fixes by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/740
* [Breaking] Bump version to v0.3.0 by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/736
* Final changes for v0.3.0 by EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/741
New Contributors
* Schuwi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/727
**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.2.5...v0.3.0