Accelerate

Dependency Changes
- **Updated Safetensors Requirement:** The library now requires `safetensors` version 0.4.3.
- **Added support for Numpy 2.0:** The library now fully supports `numpy` 2.0.0

Core

New Script Behavior Changes
- **Process Group Management:** PyTorch now requires users to destroy process groups after training. The `accelerate` library will handle this automatically with `accelerator.end_training()`, or you can do it manually using `PartialState().destroy_process_group()`.
- **MLU Device Support:** Added support for saving and loading RNG states on MLU devices by huismiling
- **NPU Support:** Corrected backend and distributed settings when using `transfer_to_npu`, ensuring better performance and compatibility.

DataLoader Enhancements
- **Stateful DataDataLoader:** We are excited to announce that early support has been added for the `StatefulDataLoader` from `torchdata`, allowing better handling of data loading states. Enable by passing `use_stateful_dataloader=True` to the `DataLoaderConfiguration`, and when calling `load_state()` the `DataLoader` will automatically be resumed from its last step, no more having to iterate through passed batches.
- **Decoupled Data Loader Preparation:** The `prepare_data_loader()` function is now **independent** of the `Accelerator`, giving you more flexibility towards which API levels you would like to use.
- **XLA Compatibility:** Added support for skipping initial batches when using XLA.
- **Improved State Management:** Bug fixes and enhancements for saving/loading `DataLoader` states, ensuring smoother training sessions.
- **Epoch Setting:** Introduced the `set_epoch` function for `MpDeviceLoaderWrapper`.

FP8 Training Improvements
- **Enhanced FP8 Training:** Fully Sharded Data Parallelism (FSDP) and DeepSpeed support now work seamlessly with `TransformerEngine` FP8 training, including better defaults for the quantized FP8 weights.
- **Integration baseline**: We've added a new suite of examples and benchmarks to ensure that our `TransformerEngine` integration works exactly as intended. These scripts run one half using 🤗 Accelerate's integration, the other with raw `TransformersEngine`, providing users with a nice example of what we do under the hood with accelerate, and a good sanity check to make sure nothing breaks down over time. Find them [here](https://github.com/huggingface/accelerate/tree/main/benchmarks/fp8)
- **Import Fixes:** Resolved issues with import checks for the Transformers Engine that has downstream issues.
- **FP8 Docker Images:** We've added new docker images for `TransformerEngine` and `accelerate` as well. Use `docker pull huggingface/accelerategpu-fp8-transformerengine` to quickly get an environment going.


`torchpippy` no more, long live `torch.distributed.pipelining`
- With the latest PyTorch release, `torchpippy` is now fully integrated into torch core, and as a result we are **exclusively supporting the PyTorch implementation from now on**
- There are breaking examples and changes that comes from this shift. Namely:
- Tracing of inputs is done with a shape *each GPU will see*, rather than the size of the total batch. So for 2 GPUs, one should pass in an input of `[1, n, n]` rather than `[2, n, n]` as before.
- **We no longer support Encoder/Decoder models**. PyTorch tracing for `pipelining` no longer supports encoder/decoder models, so the `t5` example has been removed.
- **Computer vision model support currently does not work**: There are some tracing issues regarding resnet's we are actively looking into.
- **If either of these changes are too breaking, we recommend pinning your accelerate version**. If the encoder/decoder model support is **actively blocking your inference using pippy**, please open an issue and let us know. We can look towards adding in the old support for `torchpippy` potentially if needed.

Fully Sharded Data Parallelism (FSDP)
- **Environment Flexibility:** Environment variables are now **fully optional** for FSDP, simplifying configuration. You can now fully create a `FullyShardedDataParallelPlugin` yourself manually *with no need for environment patching*:
python
from accelerate import FullyShardedDataParallelPlugin
fsdp_plugin = FullyShardedDataParallelPlugin(...)

- **FSDP RAM efficient loading:** Added a utility to enable RAM-efficient model loading (by setting the proper environmental variable). **This is generally needed if not using `accelerate launch` and need to ensure the env variables are setup properly for model loading**:
python
from accelerate.utils import enable_fsdp_ram_efficient_loading, disable_fsdp_ram_efficient_loading
enable_fsdp_ram_efficient_loading()

- **Model State Dict Management:** Enhanced support for unwrapping model state dicts in FSDP, making it easier to manage distributed models.

New Examples
- **Configuration and Models:** Improved configuration handling and introduced a configuration zoo for easier experimentation. You can learn more [here](https://github.com/huggingface/accelerate/tree/main/examples/config_yaml_templates). This was largely inspired by the `axolotl` library, so very big kudos to their wonderful [work](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples)
- **FSDP + SLURM Example:** Added a [minimal configuration example for running jobs with SLURM and using FSDP](https://github.com/huggingface/accelerate/blob/main/examples/slurm/submit_multinode_fsdp.sh)

Bug Fixes
* Fix bug of clip_grad_norm_ for xla fsdp by hanwen-sun in https://github.com/huggingface/accelerate/pull/2941
* Explicit check for `step` when loading the state by muellerzr in https://github.com/huggingface/accelerate/pull/2992
* Fix `find_tied_params` for models with shared layers by qubvel in https://github.com/huggingface/accelerate/pull/2986
* clear memory after offload by SunMarc in https://github.com/huggingface/accelerate/pull/2994
* fix default value for rank size in cpu threads_per_process assignment logic by rbrugaro in https://github.com/huggingface/accelerate/pull/3009
* Fix batch_sampler maybe None error by candlewill in https://github.com/huggingface/accelerate/pull/3025
* Do not import `transformer_engine` on import by oraluben in https://github.com/huggingface/accelerate/pull/3056
* Fix torchvision to be compatible with torch version in CI by SunMarc in https://github.com/huggingface/accelerate/pull/2982
* Fix gated test by muellerzr in https://github.com/huggingface/accelerate/pull/2993
* Fix typo on warning str: "on the meta device device" -> "on the meta device" by HeAndres in https://github.com/huggingface/accelerate/pull/2997
* Fix deepspeed tests by muellerzr in https://github.com/huggingface/accelerate/pull/3003
* Fix torch version check by muellerzr in https://github.com/huggingface/accelerate/pull/3024
* Fix fp8 benchmark on single GPU by muellerzr in https://github.com/huggingface/accelerate/pull/3032
* Fix typo in comment by zmoki688 in https://github.com/huggingface/accelerate/pull/3045
* Speed up tests by shaving off subprocess when not needed by muellerzr in https://github.com/huggingface/accelerate/pull/3042
* Remove `skip_first_batches` support for StatefulDataloader and fix all the tests by muellerzr in https://github.com/huggingface/accelerate/pull/3068

New Contributors
* byi8220 made their first contribution in https://github.com/huggingface/accelerate/pull/2957
* alex-jw-brooks made their first contribution in https://github.com/huggingface/accelerate/pull/2959
* XciD made their first contribution in https://github.com/huggingface/accelerate/pull/2981
* hanwen-sun made their first contribution in https://github.com/huggingface/accelerate/pull/2941
* HeAndres made their first contribution in https://github.com/huggingface/accelerate/pull/2997
* yitongh made their first contribution in https://github.com/huggingface/accelerate/pull/2966
* qubvel made their first contribution in https://github.com/huggingface/accelerate/pull/2986
* rbrugaro made their first contribution in https://github.com/huggingface/accelerate/pull/3009
* candlewill made their first contribution in https://github.com/huggingface/accelerate/pull/3025
* siddk made their first contribution in https://github.com/huggingface/accelerate/pull/3047
* oraluben made their first contribution in https://github.com/huggingface/accelerate/pull/3056
* tmm1 made their first contribution in https://github.com/huggingface/accelerate/pull/3055
* zmoki688 made their first contribution in https://github.com/huggingface/accelerate/pull/3045

Full Changelog:
* Require safetensors>=0.4.3 by byi8220 in https://github.com/huggingface/accelerate/pull/2957
* Fix torchvision to be compatible with torch version in CI by SunMarc in https://github.com/huggingface/accelerate/pull/2982
* Enable Unwrapping for Model State Dicts (FSDP) by alex-jw-brooks in https://github.com/huggingface/accelerate/pull/2959
* chore: Update runs-on configuration for CI workflows by XciD in https://github.com/huggingface/accelerate/pull/2981
* add MLU devices for rng state saving and loading. by huismiling in https://github.com/huggingface/accelerate/pull/2940
* remove .md to allow proper linking by nbroad1881 in https://github.com/huggingface/accelerate/pull/2977
* Fix bug of clip_grad_norm_ for xla fsdp by hanwen-sun in https://github.com/huggingface/accelerate/pull/2941
* Fix gated test by muellerzr in https://github.com/huggingface/accelerate/pull/2993
* Explicit check for `step` when loading the state by muellerzr in https://github.com/huggingface/accelerate/pull/2992
* Fix typo on warning str: "on the meta device device" -> "on the meta device" by HeAndres in https://github.com/huggingface/accelerate/pull/2997
* Support skip_first_batches for XLA by yitongh in https://github.com/huggingface/accelerate/pull/2966
* clear memory after offload by SunMarc in https://github.com/huggingface/accelerate/pull/2994
* Fix deepspeed tests by muellerzr in https://github.com/huggingface/accelerate/pull/3003
* Make env variables optional for FSDP by muellerzr in https://github.com/huggingface/accelerate/pull/2998
* Add small util to enable FSDP offloading quickly by muellerzr in https://github.com/huggingface/accelerate/pull/3006
* update version to 0.34.dev0 by SunMarc in https://github.com/huggingface/accelerate/pull/3007
* Fix `find_tied_params` for models with shared layers by qubvel in https://github.com/huggingface/accelerate/pull/2986
* Enable FSDP & Deepspeed + FP8 by muellerzr in https://github.com/huggingface/accelerate/pull/2983
* fix default value for rank size in cpu threads_per_process assignment logic by rbrugaro in https://github.com/huggingface/accelerate/pull/3009
* Wrong import check for TE by muellerzr in https://github.com/huggingface/accelerate/pull/3016
* destroy process group in `end_training` by SunMarc in https://github.com/huggingface/accelerate/pull/3012
* Tweak defaults for quantized-typed FP8 TE weights by muellerzr in https://github.com/huggingface/accelerate/pull/3018
* Set correct NPU backend and distributed_type when using transfer_to_npu by ArthurinRUC in https://github.com/huggingface/accelerate/pull/3021
* Fix torch version check by muellerzr in https://github.com/huggingface/accelerate/pull/3024
* Add end_training/destroy_pg to everything and unpin numpy by muellerzr in https://github.com/huggingface/accelerate/pull/3030
* Improve config handling and add a zoo by muellerzr in https://github.com/huggingface/accelerate/pull/3029
* Add early support for `torchdata.stateful_dataloader.StatefulDataLoader` within the `Accelerator` by byi8220 in https://github.com/huggingface/accelerate/pull/2895
* Fix fp8 benchmark on single GPU by muellerzr in https://github.com/huggingface/accelerate/pull/3032
* Fix batch_sampler maybe None error by candlewill in https://github.com/huggingface/accelerate/pull/3025
* Fixup dataloader state dict bugs + incorporate load/save_state API by muellerzr in https://github.com/huggingface/accelerate/pull/3034
* Decouple `prepare_data_loader()` from Accelerator by siddk in https://github.com/huggingface/accelerate/pull/3047
* Update CONTRIBUTING.md Setup Instructions by siddk in https://github.com/huggingface/accelerate/pull/3046
* Add a SLURM example with minimal config by muellerzr in https://github.com/huggingface/accelerate/pull/2950
* Add FP8 docker images by muellerzr in https://github.com/huggingface/accelerate/pull/3048
* Update torchpippy by muellerzr in https://github.com/huggingface/accelerate/pull/2938
* Do not import `transformer_engine` on import by oraluben in https://github.com/huggingface/accelerate/pull/3056
* use duck-typing to ensure underlying optimizer supports schedulefree hooks by tmm1 in https://github.com/huggingface/accelerate/pull/3055
* Fix typo in comment by zmoki688 in https://github.com/huggingface/accelerate/pull/3045
* add set_epoch for MpDeviceLoaderWrapper by hanwen-sun in https://github.com/huggingface/accelerate/pull/3053
* Speed up tests by shaving off subprocess when not needed by muellerzr in https://github.com/huggingface/accelerate/pull/3042
* Remove `skip_first_batches` support for StatefulDataloader and fix all the tests by muellerzr in https://github.com/huggingface/accelerate/pull/3068


Detailed Full Changelog:
* https://github.com/huggingface/accelerate/compare/v0.33.0...v0.34.0

MUSA backend support and bugfixes

Small release this month, with key focuses on some added support for backends and bugs:
* Support MUSA (Moore Threads GPU) backend in accelerate by fmo-mt in https://github.com/huggingface/accelerate/pull/2917
* Allow multiple process per device by cifkao in https://github.com/huggingface/accelerate/pull/2916
* Add `torch.float8_e4m3fn` format `dtype_byte_size` by SunMarc in https://github.com/huggingface/accelerate/pull/2945
* Properly handle Params4bit in set_module_tensor_to_device by matthewdouglas in https://github.com/huggingface/accelerate/pull/2934

What's Changed
* [tests] fix bug in torch_device by faaany in https://github.com/huggingface/accelerate/pull/2909
* Fix slowdown on init with `device_map="auto"` by muellerzr in https://github.com/huggingface/accelerate/pull/2914
* fix: bug where `multi_gpu` was being set and warning being printed even with `num_processes=1` by HarikrishnanBalagopal in https://github.com/huggingface/accelerate/pull/2921
* Better error when a bad directory is given for weight merging by muellerzr in https://github.com/huggingface/accelerate/pull/2852
* add xpu device check before moving tensor directly to xpu device by faaany in https://github.com/huggingface/accelerate/pull/2928
* Add huggingface_hub version to setup.py by nullquant in https://github.com/huggingface/accelerate/pull/2932
* Correct loading of models with shared tensors when using accelerator.load_state() by jkuntzer in https://github.com/huggingface/accelerate/pull/2875
* Hotfix PyTorch Version Installation in CI Workflow for Minimum Version Matrix by yhna940 in https://github.com/huggingface/accelerate/pull/2889
* Fix import test by muellerzr in https://github.com/huggingface/accelerate/pull/2931
* Consider pynvml available when installed through the nvidia-ml-py distribution by matthewdouglas in https://github.com/huggingface/accelerate/pull/2936
* Improve test reliability for Accelerator.free_memory() by matthewdouglas in https://github.com/huggingface/accelerate/pull/2935
* delete CCL env var setting by Liangliang-Ma in https://github.com/huggingface/accelerate/pull/2927
* feat(ci): add `pip` caching in CI by SauravMaheshkar in https://github.com/huggingface/accelerate/pull/2952

New Contributors
* HarikrishnanBalagopal made their first contribution in https://github.com/huggingface/accelerate/pull/2921
* fmo-mt made their first contribution in https://github.com/huggingface/accelerate/pull/2917
* nullquant made their first contribution in https://github.com/huggingface/accelerate/pull/2932
* cifkao made their first contribution in https://github.com/huggingface/accelerate/pull/2916
* jkuntzer made their first contribution in https://github.com/huggingface/accelerate/pull/2875
* matthewdouglas made their first contribution in https://github.com/huggingface/accelerate/pull/2936
* Liangliang-Ma made their first contribution in https://github.com/huggingface/accelerate/pull/2927
* SauravMaheshkar made their first contribution in https://github.com/huggingface/accelerate/pull/2952

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.32.1...v0.33.0

Core
* Utilize shard saving from the `huggingface_hub` rather than our own implementation (https://github.com/huggingface/accelerate/pull/2795)
* Refactor logging to use logger in `dispatch_model` (https://github.com/huggingface/accelerate/pull/2855)
* The `Accelerator.step` number is now restored when using `save_state` and `load_state` (https://github.com/huggingface/accelerate/pull/2765)
* A new profiler has been added allowing users to collect performance metrics during model training and inference, including detailed analysis of execution time and memory consumption. These can then be generated in Chrome's tracing tool. Read more about it [here](https://huggingface.co/docs/accelerate/usage_guides/profiler) (https://github.com/huggingface/accelerate/pull/2883)
* Reduced import times for doing `import accelerate` and any other major core import by 68%, now should be only slightly longer than doing `import torch` (https://github.com/huggingface/accelerate/pull/2845)
* Fixed a bug in `get_backend` and added a `clear_device_cache` utility (https://github.com/huggingface/accelerate/pull/2857)

Distributed Data Parallelism
* Introduce DDP communication hooks to have more flexibility in how gradients are communicated across workers, overriding the standard `allreduce`. (https://github.com/huggingface/accelerate/pull/2841)
* Make `log_line_prefix_template` optional the `notebook_launcher` (https://github.com/huggingface/accelerate/pull/2888)
FSDP
* If the output directory doesn't exist when using `accelerate merge-weights`, one will be automatically created (https://github.com/huggingface/accelerate/pull/2854)
* When merging weights, the default is now `.safetensors` (https://github.com/huggingface/accelerate/pull/2853)

XPU
* Migrate to pytorch's native XPU backend on `torch>=2.4` (https://github.com/huggingface/accelerate/pull/2825)
* Add `require_triton` test decorator and enable `test_dynamo` work on xpu (https://github.com/huggingface/accelerate/pull/2878)
* Fixed `load_state_dict` not working on `xpu` and refine xpu `safetensors` version check (https://github.com/huggingface/accelerate/pull/2879)
XLA
* Added support for XLA Dynamo backends for both training and inference (https://github.com/huggingface/accelerate/pull/2892)

Examples
* Added a new multi-cpu SLURM example using `accelerate launch` (https://github.com/huggingface/accelerate/pull/2902)

Full Changelog
* Use shard saving from huggingface_hub by SunMarc in https://github.com/huggingface/accelerate/pull/2795
* doc: fix link by imba-tjd in https://github.com/huggingface/accelerate/pull/2844
* Revert "Slight rename" by SunMarc in https://github.com/huggingface/accelerate/pull/2850
* remove warning hook addede during dispatch_model by SunMarc in https://github.com/huggingface/accelerate/pull/2843
* Remove underlines between badges by novialriptide in https://github.com/huggingface/accelerate/pull/2851
* Auto create dir when merging FSDP weights by helloworld1 in https://github.com/huggingface/accelerate/pull/2854
* Add DDP Communication Hooks by yhna940 in https://github.com/huggingface/accelerate/pull/2841
* Refactor logging to use logger in `dispatch_model` by panjd123 in https://github.com/huggingface/accelerate/pull/2855
* xpu: support xpu backend from stock pytorch (>=2.4) by dvrogozh in https://github.com/huggingface/accelerate/pull/2825
* Drop torch re-imports in npu and mlu paths by dvrogozh in https://github.com/huggingface/accelerate/pull/2856
* Default FSDP weights merge to safetensors by helloworld1 in https://github.com/huggingface/accelerate/pull/2853
* [tests] fix bug in `test_tracking.ClearMLTest` by faaany in https://github.com/huggingface/accelerate/pull/2863
* [tests] use `torch_device` instead of `0` for device check by faaany in https://github.com/huggingface/accelerate/pull/2861
* [tests] skip bnb-related tests instead of failing on xpu by faaany in https://github.com/huggingface/accelerate/pull/2860
* Potentially fix tests by muellerzr in https://github.com/huggingface/accelerate/pull/2862
* [tests] enable XPU backend for `test_zero3_integration` by faaany in https://github.com/huggingface/accelerate/pull/2864
* Support saving and loading of step while saving and loading state by bipinKrishnan in https://github.com/huggingface/accelerate/pull/2765
* Add Profiler Support for Performance Analysis by yhna940 in https://github.com/huggingface/accelerate/pull/2883
* Speed up imports and add a CI by muellerzr in https://github.com/huggingface/accelerate/pull/2845
* Make `log_line_prefix_template` Optional in Elastic Launcher for Backward Compatibility by yhna940 in https://github.com/huggingface/accelerate/pull/2888
* Add XLA Dynamo backends for training and inference by johnsutor in https://github.com/huggingface/accelerate/pull/2892
* Added a MultiCPU SLURM example using Accelerate Launch and MPIRun by okhleif-IL in https://github.com/huggingface/accelerate/pull/2902
* make more cuda-only tests device-agnostic by faaany in https://github.com/huggingface/accelerate/pull/2876
* fix mlu device longTensor bugs by huismiling in https://github.com/huggingface/accelerate/pull/2887
* add `require_triton` and enable `test_dynamo` work on xpu by faaany in https://github.com/huggingface/accelerate/pull/2878
* fix `load_state_dict` for xpu and refine xpu safetensor version check by faaany in https://github.com/huggingface/accelerate/pull/2879
* Fix get_backend bug and add clear_device_cache function by NurmaU in https://github.com/huggingface/accelerate/pull/2857

New Contributors
* McPatate made their first contribution in https://github.com/huggingface/accelerate/pull/2836
* imba-tjd made their first contribution in https://github.com/huggingface/accelerate/pull/2844
* novialriptide made their first contribution in https://github.com/huggingface/accelerate/pull/2851
* panjd123 made their first contribution in https://github.com/huggingface/accelerate/pull/2855
* dvrogozh made their first contribution in https://github.com/huggingface/accelerate/pull/2825
* johnsutor made their first contribution in https://github.com/huggingface/accelerate/pull/2892
* okhleif-IL made their first contribution in https://github.com/huggingface/accelerate/pull/2902
* NurmaU made their first contribution in https://github.com/huggingface/accelerate/pull/2857

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.31.0...v0.32.0

Core
* Set `timeout` default to PyTorch defaults based on backend by muellerzr in https://github.com/huggingface/accelerate/pull/2758
* fix duplicate elements in split_between_processes by hkunzhe in https://github.com/huggingface/accelerate/pull/2781
* Add Elastic Launch Support to `notebook_launcher` by yhna940 in https://github.com/huggingface/accelerate/pull/2788
* Fix Wrong use of sync_gradients used to implement sync_each_batch by fabianlim in https://github.com/huggingface/accelerate/pull/2790

FSDP
* Introduce shard-merging util for FSDP by muellerzr in https://github.com/huggingface/accelerate/pull/2772
* Enable sharded state dict + offload to cpu resume by muellerzr in https://github.com/huggingface/accelerate/pull/2762
* Enable config for fsdp activation checkpointing by helloworld1 in https://github.com/huggingface/accelerate/pull/2779

Megatron
* Upgrade huggingface's megatron to nvidia's megatron when use MegatronLMPlugin by zhangsheng377 in https://github.com/huggingface/accelerate/pull/2501

What's Changed
* Add feature to allow redirecting std streams into log files when using torchrun as the launcher. by lyuwen in https://github.com/huggingface/accelerate/pull/2740
* Update modeling.py by adding try-catch section to skip the unavailable devices by MeVeryHandsome in https://github.com/huggingface/accelerate/pull/2681
* Fixed the problem of incorrect conditional judgment statement when configuring enable_cpu_affinity by statelesshz in https://github.com/huggingface/accelerate/pull/2748
* Fix stacklevel in `logging` to log the actual user call site (instead of the call site inside the logger wrapper) of log functions by luowyang in https://github.com/huggingface/accelerate/pull/2730
* LOMO / FIX: Support multiple optimizers by younesbelkada in https://github.com/huggingface/accelerate/pull/2745
* Fix max_memory assignment by SunMarc in https://github.com/huggingface/accelerate/pull/2751
* Fix duplicate environment variable check in multi-cpu condition by yhna940 in https://github.com/huggingface/accelerate/pull/2752
* Simplify CLI args validation and ensure CLI args take precedence over config file. by Iain-S in https://github.com/huggingface/accelerate/pull/2757
* Fix sagemaker config by muellerzr in https://github.com/huggingface/accelerate/pull/2753
* fix cpu omp num threads set by jiqing-feng in https://github.com/huggingface/accelerate/pull/2755
* Revert "Simplify CLI args validation and ensure CLI args take precedence over config file." by muellerzr in https://github.com/huggingface/accelerate/pull/2763
* Enable sharded cpu resume by muellerzr in https://github.com/huggingface/accelerate/pull/2762
* Sets default to PyTorch defaults based on backend by muellerzr in https://github.com/huggingface/accelerate/pull/2758
* optimize get_module_leaves speed by BBuf in https://github.com/huggingface/accelerate/pull/2756
* fix minor typo by TemryL in https://github.com/huggingface/accelerate/pull/2767
* Fix small edge case in get_module_leaves by SunMarc in https://github.com/huggingface/accelerate/pull/2774
* Skip deepspeed test by SunMarc in https://github.com/huggingface/accelerate/pull/2776
* Enable config for fsdp activation checkpointing by helloworld1 in https://github.com/huggingface/accelerate/pull/2779
* Add arg from CLI to fix failing test by muellerzr in https://github.com/huggingface/accelerate/pull/2783
* Skip tied weights disk offload test by SunMarc in https://github.com/huggingface/accelerate/pull/2782
* Introduce shard-merging util for FSDP by muellerzr in https://github.com/huggingface/accelerate/pull/2772
* FIX / FSDP : Guard fsdp utils for earlier PyTorch versions by younesbelkada in https://github.com/huggingface/accelerate/pull/2794
* Upgrade huggingface's megatron to nvidia's megatron when use MegatronLMPlugin by zhangsheng377 in https://github.com/huggingface/accelerate/pull/2501
* Fixup CLI test by muellerzr in https://github.com/huggingface/accelerate/pull/2796
* fix duplicate elements in split_between_processes by hkunzhe in https://github.com/huggingface/accelerate/pull/2781
* Add Elastic Launch Support to `notebook_launcher` by yhna940 in https://github.com/huggingface/accelerate/pull/2788
* Fix Wrong use of sync_gradients used to implement sync_each_batch by fabianlim in https://github.com/huggingface/accelerate/pull/2790
* Fix type in accelerator.py by qgallouedec in https://github.com/huggingface/accelerate/pull/2800
* fix comet ml test by SunMarc in https://github.com/huggingface/accelerate/pull/2804
* New template by muellerzr in https://github.com/huggingface/accelerate/pull/2808
* Fix access error for torch.mps when using torch==1.13.1 on macOS by SunMarc in https://github.com/huggingface/accelerate/pull/2806
* 4-bit quantization meta device bias loading bug by SunMarc in https://github.com/huggingface/accelerate/pull/2805
* State dictionary retrieval from offloaded modules by blbadger in https://github.com/huggingface/accelerate/pull/2619
* add cuda dep for a test by SunMarc in https://github.com/huggingface/accelerate/pull/2820
* Remove out-dated xpu device check code in `get_balanced_memory` by faaany in https://github.com/huggingface/accelerate/pull/2826
* Fix DeepSpeed config validation error by changing `stage3_prefetch_bucket_size` value to an integer by adk9 in https://github.com/huggingface/accelerate/pull/2814
* Improve test speeds by up to 30% in multi-gpu settings by muellerzr in https://github.com/huggingface/accelerate/pull/2830
* monitor-interval, take 2 by muellerzr in https://github.com/huggingface/accelerate/pull/2833
* Optimize the megatron plugin by zhangsheng377 in https://github.com/huggingface/accelerate/pull/2822
* fix fstr format by Jintao-Huang in https://github.com/huggingface/accelerate/pull/2810

New Contributors
* lyuwen made their first contribution in https://github.com/huggingface/accelerate/pull/2740
* MeVeryHandsome made their first contribution in https://github.com/huggingface/accelerate/pull/2681
* luowyang made their first contribution in https://github.com/huggingface/accelerate/pull/2730
* Iain-S made their first contribution in https://github.com/huggingface/accelerate/pull/2757
* BBuf made their first contribution in https://github.com/huggingface/accelerate/pull/2756
* TemryL made their first contribution in https://github.com/huggingface/accelerate/pull/2767
* helloworld1 made their first contribution in https://github.com/huggingface/accelerate/pull/2779
* hkunzhe made their first contribution in https://github.com/huggingface/accelerate/pull/2781
* adk9 made their first contribution in https://github.com/huggingface/accelerate/pull/2814
* Jintao-Huang made their first contribution in https://github.com/huggingface/accelerate/pull/2810

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.30.1...v0.31.0

Patchfix
* Fix duplicate environment variable check in multi-cpu condition thanks to yhna940 in https://github.com/huggingface/accelerate/pull/2752
* Fix issue with missing values in the SageMaker config leading to not being able to launch in https://github.com/huggingface/accelerate/pull/2753
* Fix CPU OMP num threads setting thanks to jiqing-feng in https://github.com/huggingface/accelerate/pull/2755
* Fix FSDP checkpoint unable to resume when using offloading and sharded weights due to CUDA OOM when loading the optimizer and model https://github.com/huggingface/accelerate/pull/2762
* Fixed the problem of incorrect conditional judgment statement when configuring enable_cpu_affinity thanks to statelesshz in https://github.com/huggingface/accelerate/pull/2748
* Fix stacklevel in logging to log the actual user call site (instead of the call site inside the logger wrapper) of log functions thanks to luowyang in https://github.com/huggingface/accelerate/pull/2730
* Fix support for multiple optimizers when using LOMO thanks to younesbelkada in https://github.com/huggingface/accelerate/pull/2745

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.30.0...v0.30.1

Core
* We've simplified the `tqdm` wrapper to make it fully passthrough, no need to have `tqdm(main_process_only, *args)`, it is now just `tqdm(*args)` and you can pass in `is_main_process` as a kwarg.
* We've added support for advanced optimizer usage:
* Schedule free optimizer introduced by [Meta](https://github.com/facebookresearch/schedule_free/tree/main) by muellerzr in https://github.com/huggingface/accelerate/pull/2631
* LOMO optimizer introduced by [OpenLMLab](https://github.com/OpenLMLab/LOMO) by younesbelkada in https://github.com/huggingface/accelerate/pull/2695
* Enable BF16 autocast to everything during FP8 and enable FSDP by muellerzr in https://github.com/huggingface/accelerate/pull/2655
* Support dataloader send_to_device calls to use non-blocking by drhead in https://github.com/huggingface/accelerate/pull/2685
* allow gather_for_metrics to be more flexible by SunMarc in https://github.com/huggingface/accelerate/pull/2710
* Add `cann` version info to command accelerate env for NPU by statelesshz in https://github.com/huggingface/accelerate/pull/2689
* Add MLU rng state setter by ArthurinRUC in https://github.com/huggingface/accelerate/pull/2664
* device agnostic testing for hooks&utils&big_modeling by statelesshz in https://github.com/huggingface/accelerate/pull/2602

Documentation
* Through collaboration between fabianlim (lead contribuitor), stas00, pacman100, and muellerzr we have a new concept guide out for FSDP and DeepSpeed explicitly detailing how each interop and explaining fully and clearly how each of those work. This was a momumental effort by fabianlim to ensure that everything can be as accurate as possible to users. I highly recommend visiting this new documentation, available [here](https://huggingface.co/docs/accelerate/concept_guides/fsdp_and_deepspeed)
* New distributed inference examples have been added thanks to SunMarc in https://github.com/huggingface/accelerate/pull/2672
* Fixed some docs for using internal trackers by brentyi in https://github.com/huggingface/accelerate/pull/2650

DeepSpeed
* Accelerate can now handle MoE models when using deepspeed, thanks to pacman100 in https://github.com/huggingface/accelerate/pull/2662
* Allow "auto" for gradient clipping in YAML by regisss in https://github.com/huggingface/accelerate/pull/2649
* Introduce a `deepspeed`-specific Docker image by muellerzr in https://github.com/huggingface/accelerate/pull/2707. To use, pull the `gpu-deepspeed` tag `docker pull huggingface/accelerate:cuda-deepspeed-nightly`

Megatron
* Megatron plugin can support NPU by zhangsheng377 in https://github.com/huggingface/accelerate/pull/2667


Big Modeling
* Add strict arg to load_checkpoint_and_dispatch by SunMarc in https://github.com/huggingface/accelerate/pull/2641

Bug Fixes
* Fix up state with xla + performance regression by muellerzr in https://github.com/huggingface/accelerate/pull/2634
* Parenthesis on xpu_available by muellerzr in https://github.com/huggingface/accelerate/pull/2639
* Fix `is_train_batch_min` type in DeepSpeedPlugin by yhna940 in https://github.com/huggingface/accelerate/pull/2646
* Fix backend check by jiqing-feng in https://github.com/huggingface/accelerate/pull/2652
* Fix the rng states of sampler's generator to be synchronized for correct sharding of dataset across GPUs by pacman100 in https://github.com/huggingface/accelerate/pull/2694
* Block AMP for MPS device by SunMarc in https://github.com/huggingface/accelerate/pull/2699
* Fixed issue when doing multi-gpu training with bnb when the first gpu is not used by SunMarc in https://github.com/huggingface/accelerate/pull/2714
* Fixup `free_memory` to deal with garbage collection by muellerzr in https://github.com/huggingface/accelerate/pull/2716
* Fix sampler serialization failing by SunMarc in https://github.com/huggingface/accelerate/pull/2723
* Fix deepspeed offload device type in the arguments to be more accurate by yhna940 in https://github.com/huggingface/accelerate/pull/2717

Full Changelog
* Schedule free optimizer support by muellerzr in https://github.com/huggingface/accelerate/pull/2631
* Fix up state with xla + performance regression by muellerzr in https://github.com/huggingface/accelerate/pull/2634
* Parenthesis on xpu_available by muellerzr in https://github.com/huggingface/accelerate/pull/2639
* add third-party device prefix to `execution_device` by faaany in https://github.com/huggingface/accelerate/pull/2612
* add strict arg to load_checkpoint_and_dispatch by SunMarc in https://github.com/huggingface/accelerate/pull/2641
* device agnostic testing for hooks&utils&big_modeling by statelesshz in https://github.com/huggingface/accelerate/pull/2602
* Docs fix for using internal trackers by brentyi in https://github.com/huggingface/accelerate/pull/2650
* Allow "auto" for gradient clipping in YAML by regisss in https://github.com/huggingface/accelerate/pull/2649
* Fix `is_train_batch_min` type in DeepSpeedPlugin by yhna940 in https://github.com/huggingface/accelerate/pull/2646
* Don't use deprecated `Repository` anymore by Wauplin in https://github.com/huggingface/accelerate/pull/2658
* Fix test_from_pretrained_low_cpu_mem_usage_measured failure by yuanwu2017 in https://github.com/huggingface/accelerate/pull/2644
* Add MLU rng state setter by ArthurinRUC in https://github.com/huggingface/accelerate/pull/2664
* fix backend check by jiqing-feng in https://github.com/huggingface/accelerate/pull/2652
* Megatron plugin can support NPU by zhangsheng377 in https://github.com/huggingface/accelerate/pull/2667
* Revert "fix backend check" by muellerzr in https://github.com/huggingface/accelerate/pull/2669
* `tqdm`: `*args` should come ahead of `main_process_only` by rb-synth in https://github.com/huggingface/accelerate/pull/2654
* Handle MoE models with DeepSpeed by pacman100 in https://github.com/huggingface/accelerate/pull/2662
* Fix deepspeed moe test with version check by pacman100 in https://github.com/huggingface/accelerate/pull/2677
* Pin DS...again.. by muellerzr in https://github.com/huggingface/accelerate/pull/2679
* fix backend check by jiqing-feng in https://github.com/huggingface/accelerate/pull/2670
* Deprecate tqdm args + slight logic tweaks by muellerzr in https://github.com/huggingface/accelerate/pull/2673
* Enable BF16 autocast to everything during FP8 + some tweaks to enable FSDP by muellerzr in https://github.com/huggingface/accelerate/pull/2655
* Fix the rng states of sampler's generator to be synchronized for correct sharding of dataset across GPUs by pacman100 in https://github.com/huggingface/accelerate/pull/2694
* Simplify test logic by pacman100 in https://github.com/huggingface/accelerate/pull/2697
* Add source code for DataLoader Animation by muellerzr in https://github.com/huggingface/accelerate/pull/2696
* Block AMP for MPS device by SunMarc in https://github.com/huggingface/accelerate/pull/2699
* Do a pip freeze during workflows by muellerzr in https://github.com/huggingface/accelerate/pull/2704
* add cann version info to command accelerate env by statelesshz in https://github.com/huggingface/accelerate/pull/2689
* Add version checks for the import of DeepSpeed moe utils by pacman100 in https://github.com/huggingface/accelerate/pull/2705
* Change dataloader send_to_device calls to non-blocking by drhead in https://github.com/huggingface/accelerate/pull/2685
* add distributed examples by SunMarc in https://github.com/huggingface/accelerate/pull/2672
* Add diffusers to req by muellerzr in https://github.com/huggingface/accelerate/pull/2711
* fix bnb multi gpu training by SunMarc in https://github.com/huggingface/accelerate/pull/2714
* allow gather_for_metrics to be more flexible by SunMarc in https://github.com/huggingface/accelerate/pull/2710
* Add Upcasting for FSDP in Mixed Precision. Add Concept Guide for FSPD and DeepSpeed. by fabianlim in https://github.com/huggingface/accelerate/pull/2674
* Segment out a deepspeed docker image by muellerzr in https://github.com/huggingface/accelerate/pull/2707
* Fixup `free_memory` to deal with garbage collection by muellerzr in https://github.com/huggingface/accelerate/pull/2716
* fix sampler serialization by SunMarc in https://github.com/huggingface/accelerate/pull/2723
* Fix sampler failing test by SunMarc in https://github.com/huggingface/accelerate/pull/2728
* Docs: Fix build main documentation by SunMarc in https://github.com/huggingface/accelerate/pull/2729
* Fix Documentation in FSDP and DeepSpeed Concept Guide by fabianlim in https://github.com/huggingface/accelerate/pull/2725
* Fix deepspeed offload device type by yhna940 in https://github.com/huggingface/accelerate/pull/2717
* FEAT: Add LOMO optimizer by younesbelkada in https://github.com/huggingface/accelerate/pull/2695
* Fix tests on main by muellerzr in https://github.com/huggingface/accelerate/pull/2739

New Contributors
* brentyi made their first contribution in https://github.com/huggingface/accelerate/pull/2650
* regisss made their first contribution in https://github.com/huggingface/accelerate/pull/2649
* yhna940 made their first contribution in https://github.com/huggingface/accelerate/pull/2646
* Wauplin made their first contribution in https://github.com/huggingface/accelerate/pull/2658
* ArthurinRUC made their first contribution in https://github.com/huggingface/accelerate/pull/2664
* jiqing-feng made their first contribution in https://github.com/huggingface/accelerate/pull/2652
* zhangsheng377 made their first contribution in https://github.com/huggingface/accelerate/pull/2667
* rb-synth made their first contribution in https://github.com/huggingface/accelerate/pull/2654
* drhead made their first contribution in https://github.com/huggingface/accelerate/pull/2685

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.29.3...v0.30.0