This release contains new features and fixes for distributed training.
**Important:** This release fixes hangs in distributed training by ensuring the same number of batches are returned on each rank (237). However, this and other fixes change how samples are assigned to ranks and is therefore a breaking change. Resuming from checkpoints created with an older version of LitData will not be valid (if you are using the stateful data loader feature).
What's Changed
* Feat: Updates readme and a few nitpicks by deependujha in https://github.com/Lightning-AI/litdata/pull/223
* docs: add `Specify cache directory` by csy1204 in https://github.com/Lightning-AI/litdata/pull/229
* Enable compatibility with Numpy 2.0 by weiji14 in https://github.com/Lightning-AI/litdata/pull/230
* Fix typo in resolver.py by lud-ds in https://github.com/Lightning-AI/litdata/pull/239
* Feature: Add support for encryption and decryption of data at chunk/sample level by bhimrazy in https://github.com/Lightning-AI/litdata/pull/219
* Fix uneven batches in distributed dataloading by awaelchli in https://github.com/Lightning-AI/litdata/pull/237
* feat: add a custom storage options param by csy1204 in https://github.com/Lightning-AI/litdata/pull/246
* Fix index errors on world size > 0 by awaelchli in https://github.com/Lightning-AI/litdata/pull/252
New Contributors
* csy1204 made their first contribution in https://github.com/Lightning-AI/litdata/pull/229
* weiji14 made their first contribution in https://github.com/Lightning-AI/litdata/pull/230
* lud-ds made their first contribution in https://github.com/Lightning-AI/litdata/pull/239
**Full Changelog**: https://github.com/Lightning-AI/litdata/compare/v0.2.16...v0.2.17