Megatron-energon

New Features
* New dataset joining and pre-indexed joined tar iteration (https://github.com/NVIDIA/Megatron-Energon/pull/51, philipp-fischer, voegtlel)
* Allow restore with different worker configuration (refactor save/restore concept, worker dimension as outer dimension) (https://github.com/NVIDIA/Megatron-Energon/pull/80, philipp-fischer, voegtlel)
* Use EPath for all paths, removes fsspec (https://github.com/NVIDIA/Megatron-Energon/pull/62, voegtlel)
* Simplify savable loader implementation (https://github.com/NVIDIA/Megatron-Energon/pull/87, philipp-fischer, voegtlel)
* Efficient Audio and Video decoding (https://github.com/NVIDIA/Megatron-Energon/pull/38, https://github.com/NVIDIA/Megatron-Energon/pull/93, jon-barker, voegtlel)
* Expose `prefetch_factor` arg for loader (https://github.com/NVIDIA/Megatron-Energon/pull/83, philipp-fischer)

Fixes
* Fix len for `RepeatDataset` with float repeats (https://github.com/NVIDIA/Megatron-Energon/pull/89, voegtlel)
* Fix `EPath` for relative local filesystem string paths (https://github.com/NVIDIA/Megatron-Energon/pull/95, voegtlel)
* Fix `EPath.open` in `ITarReader` (https://github.com/NVIDIA/Megatron-Energon/pull/88, shunjiad)
* Fix a rare bug in save/restore (https://github.com/NVIDIA/Megatron-Energon/pull/79, philipp-fischer)

Internal Changes
* Toolchain upgrade: Drop black, isort and introduce ruff, uv and just (https://github.com/NVIDIA/Megatron-Energon/pull/67, philipp-fischer)

**Full Changelog**: https://github.com/NVIDIA/Megatron-Energon/compare/5.2.0...6.0.1

What's Changed

New Features
* Add ability to use [Multi-Storage-Client](https://github.com/NVIDIA/multi-storage-client) in Energon, new dependencies (shunjiad, #50)
* Allow float for `epochized_blend` repetitions (philipp-fischer, 56)
* Extend docs with example for interleaved data samples (voegtlel, philipp-fischer, 46)

Fixes
* `encode_batch` now called for grouped batching (nvnbagrov, 64)
* GC: make `gc_collect_every_n_steps` configurable and add a new default for optimized speed (philipp-fischer, voegtlel, 66)
* Create docs for usage with parallelism and fix `save_state_global` and `restore_state_global` for Tensor Parallelism (philipp-fischer, voegtlel, 72)
* Preparing crude (nvnbagrov, 55)
* Packing pause/burst when buffer is empty. Restore ability to fill the buffer as it is emitted (voegtlel, 65)

Internal Changes
* Rename `prepare()` in metadatasets to `post_initialize()` (philipp-fischer, 45)
* Refactored `prepare` (nvnbagrov, 57)
* Fix restore key if there's an empty worker (philipp-fischer, 69)

**Full Changelog**: https://github.com/NVIDIA/Megatron-Energon/compare/5.1.1...5.2.0

What's Changed
* Hotfix: Fix restore key if there's an empty worker in https://github.com/NVIDIA/Megatron-Energon/pull/68

**Full Changelog**: https://github.com/NVIDIA/Megatron-Energon/compare/5.1.0...5.1.1

Whats Changed

* Reduce checkpoint size, by reproducing the pending shards instead of storing them (voegtlel, https://github.com/NVIDIA/Megatron-Energon/pull/49, CHECKPOINT BREAKING CHANGE)

What's Changed
* Implement epochs for blending (optionally giving the number of repetitions for each dataset for an epoch instead of sampling weight) (voegtlel, philipp-fischer, 41, **CHECKPOINT BREAKING CHANGE**)
* Implement grouped batching (e.g. for Open-Sora) (voegtlel, 31)
* Fix distribution of samples to workers if using lots of small datasets (philipp-fischer, 32, **ITERATION ORDER BREAKING CHANGE**)
* Improve and restructure documentation (philipp-fischer, voegtlel, 37)
* Activating `gc.freeze()` in workers on init to improve `gc.collect()` speed by a lot (voegtlel, 40)
* Deprecated `SavableLoader.save_state` and `SavableLoader.restore_state`: Renamed to `save_state_global` and `restore_state_global`, and removed the option to not specify the `dst_rank` for saving (this is breaking but had no real use-case). Added docs for the scenarios. (voegtlel, philipp-fischer, 43)
* Fix size print for >1PiB (nvnbagrov, 39)

Internal Changes
* All dataset wrappers now have the worker config (philipp-fischer, 36)
* Check black, isort, license headers (voegtlel, philipp-fischer, 25)

What's Changed
* Enable adding of additional data by joining another dataset by voegtlel and philipp-fischer in https://github.com/NVIDIA/Megatron-Energon/pull/20
* Replace the dataset type in the dataset.yaml by sample type directly by voegtlel and philipp-fischer in https://github.com/NVIDIA/Megatron-Energon/pull/29

Breaking Changes
* Dataset checkpoints from <4.0.0 will not be compatible due to the structural simplification. Everything else (e.g. randomness and the interface compatibility) should remain the same.

**Full Changelog**: https://github.com/NVIDIA/Megatron-Energon/compare/3.0.1...4.0.0