Accelerate

As it's been ~2 years since torch 2.0 was first released, we are now requiring this as the **minimum version for Accelerate**, which similarly was done in `transformers` as of its last release.

Core
* [docs] no hard-coding cuda by faaany in https://github.com/huggingface/accelerate/pull/3270
* fix load_state_dict for npu by ji-huazhong in https://github.com/huggingface/accelerate/pull/3211
* Add `keep_torch_compile` param to `unwrap_model` and `extract_model_from_parallel` for distributed compiled model. by ggoggam in https://github.com/huggingface/accelerate/pull/3282
* [tests] make cuda-only test case device-agnostic by faaany in https://github.com/huggingface/accelerate/pull/3340
* latest bnb no longer has optim_args attribute on optimizer by winglian in https://github.com/huggingface/accelerate/pull/3311
* add torchdata version check to avoid "in_order" error by faaany in https://github.com/huggingface/accelerate/pull/3344
* [docs] fix typo, change "backoff_filter" to "backoff_factor" by suchot in https://github.com/huggingface/accelerate/pull/3296
* dataloader: check that in_order is in kwargs before trying to drop it by dvrogozh in https://github.com/huggingface/accelerate/pull/3346
* feat(tpu): remove nprocs from xla.spawn by tengomucho in https://github.com/huggingface/accelerate/pull/3324

Big Modeling
* Fix test_nested_hook by SunMarc in https://github.com/huggingface/accelerate/pull/3289
* correct the return statement of _init_infer_auto_device_map by Nech-C in https://github.com/huggingface/accelerate/pull/3279
* Use torch.xpu.mem_get_info for XPU by dvrogozh in https://github.com/huggingface/accelerate/pull/3275
* Ensure that tied parameter is children of module by pablomlago in https://github.com/huggingface/accelerate/pull/3327
* Fix for offloading when using TorchAO >= 0.7.0 by a-r-r-o-w in https://github.com/huggingface/accelerate/pull/3332
* Fix offload generate tests by SunMarc in https://github.com/huggingface/accelerate/pull/3334

Examples
* Give example on how to handle gradient accumulation with cross-entropy by ylacombe in https://github.com/huggingface/accelerate/pull/3193

Full Changelog
What's Changed
* [docs] no hard-coding cuda by faaany in https://github.com/huggingface/accelerate/pull/3270
* fix load_state_dict for npu by ji-huazhong in https://github.com/huggingface/accelerate/pull/3211
* Fix test_nested_hook by SunMarc in https://github.com/huggingface/accelerate/pull/3289
* correct the return statement of _init_infer_auto_device_map by Nech-C in https://github.com/huggingface/accelerate/pull/3279
* Give example on how to handle gradient accumulation with cross-entropy by ylacombe in https://github.com/huggingface/accelerate/pull/3193
* Use torch.xpu.mem_get_info for XPU by dvrogozh in https://github.com/huggingface/accelerate/pull/3275
* Add `keep_torch_compile` param to `unwrap_model` and `extract_model_from_parallel` for distributed compiled model. by ggoggam in https://github.com/huggingface/accelerate/pull/3282
* Ensure that tied parameter is children of module by pablomlago in https://github.com/huggingface/accelerate/pull/3327
* Bye bye torch <2 by muellerzr in https://github.com/huggingface/accelerate/pull/3331
* Fixup docker build err by muellerzr in https://github.com/huggingface/accelerate/pull/3333
* feat(tpu): remove nprocs from xla.spawn by tengomucho in https://github.com/huggingface/accelerate/pull/3324
* Fix offload generate tests by SunMarc in https://github.com/huggingface/accelerate/pull/3334
* [tests] make cuda-only test case device-agnostic by faaany in https://github.com/huggingface/accelerate/pull/3340
* latest bnb no longer has optim_args attribute on optimizer by winglian in https://github.com/huggingface/accelerate/pull/3311
* Fix for offloading when using TorchAO >= 0.7.0 by a-r-r-o-w in https://github.com/huggingface/accelerate/pull/3332
* add torchdata version check to avoid "in_order" error by faaany in https://github.com/huggingface/accelerate/pull/3344
* [docs] fix typo, change "backoff_filter" to "backoff_factor" by suchot in https://github.com/huggingface/accelerate/pull/3296
* dataloader: check that in_order is in kwargs before trying to drop it by dvrogozh in https://github.com/huggingface/accelerate/pull/3346

New Contributors
* ylacombe made their first contribution in https://github.com/huggingface/accelerate/pull/3193
* ggoggam made their first contribution in https://github.com/huggingface/accelerate/pull/3282
* pablomlago made their first contribution in https://github.com/huggingface/accelerate/pull/3327
* tengomucho made their first contribution in https://github.com/huggingface/accelerate/pull/3324
* suchot made their first contribution in https://github.com/huggingface/accelerate/pull/3296

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.2.1...v1.3.0

**Bug Fixes**:
* Fixed an issue with `torch.get_default_device()` requiring a higher version than what we support
* Fixed a broken `pytest` import in prod

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.5.0...v1.5.2

HPU Support
* Adds in HPU accelerator support for 🤗 Accelerate


What's Changed
* [bug] fix device index bug for model training loaded with bitsandbytes by faaany in https://github.com/huggingface/accelerate/pull/3408
* [docs] add the missing `import torch` by faaany in https://github.com/huggingface/accelerate/pull/3396
* minor doc fixes by nbroad1881 in https://github.com/huggingface/accelerate/pull/3365
* fix: ensure CLI args take precedence over config file. by cyr0930 in https://github.com/huggingface/accelerate/pull/3409
* fix: Add `device=torch.get_default_device()` in `torch.Generator`s by saforem2 in https://github.com/huggingface/accelerate/pull/3420
* Add Tecorigin SDAA accelerator support by siqi654321 in https://github.com/huggingface/accelerate/pull/3330
* fix typo : thier -> their by hackty in https://github.com/huggingface/accelerate/pull/3423
* Fix quality by muellerzr in https://github.com/huggingface/accelerate/pull/3424
* Distributed inference example for llava_next by VladOS95-cyber in https://github.com/huggingface/accelerate/pull/3417
* HPU support by IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3378

New Contributors
* cyr0930 made their first contribution in https://github.com/huggingface/accelerate/pull/3409
* saforem2 made their first contribution in https://github.com/huggingface/accelerate/pull/3420
* siqi654321 made their first contribution in https://github.com/huggingface/accelerate/pull/3330
* hackty made their first contribution in https://github.com/huggingface/accelerate/pull/3423
* VladOS95-cyber made their first contribution in https://github.com/huggingface/accelerate/pull/3417
* IlyasMoutawwakil made their first contribution in https://github.com/huggingface/accelerate/pull/3378

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.4.0...v1.5.0

`torchao` FP8, initial Tensor Parallel support, and memory leak fixes

`torchao` FP8

This release introduces a new FP8 API and brings in a new backend: [`torchao`](https://github.com/pytorch/ao/tree/main/torchao/float8). To use, pass in `AORecipeKwargs` to the `Accelerator` while setting `mixed_precision="fp8"`. This is initial support, as it matures we will incorporate more into it (such as `accelerate config`/yaml) in future releases. See our benchmark examples [here](https://github.com/huggingface/accelerate/tree/main/benchmarks/fp8/torchao)

TensorParallel

We have intial support for an in-house solution to TP when working with accelerate dataloaders. check out the PR [here](https://github.com/huggingface/accelerate/pull/3173)

Bug fixes
* fix triton version check by faaany in https://github.com/huggingface/accelerate/pull/3345
* fix torch_dtype in estimate memory by SunMarc in https://github.com/huggingface/accelerate/pull/3383
* works for fp8 with deepspeed by XiaobingSuper in https://github.com/huggingface/accelerate/pull/3361
* [`memory leak`] Replace GradientState -> DataLoader reference with weakrefs by tomaarsen in https://github.com/huggingface/accelerate/pull/3391

What's Changed
* fix triton version check by faaany in https://github.com/huggingface/accelerate/pull/3345
* [tests] enable BNB test cases in `tests/test_quantization.py` on XPU by faaany in https://github.com/huggingface/accelerate/pull/3349
* [Dev] Update release directions by muellerzr in https://github.com/huggingface/accelerate/pull/3352
* [tests] make cuda-only test work on other hardware accelerators by faaany in https://github.com/huggingface/accelerate/pull/3302
* [tests] remove `require_non_xpu` test markers by faaany in https://github.com/huggingface/accelerate/pull/3301
* Support more functionalities for MUSA backend by fmo-mt in https://github.com/huggingface/accelerate/pull/3359
* [tests] enable more bnb tests on XPU by faaany in https://github.com/huggingface/accelerate/pull/3350
* feat: support tensor parallel & Data loader by kmehant in https://github.com/huggingface/accelerate/pull/3173
* DeepSpeed github repo move sync by stas00 in https://github.com/huggingface/accelerate/pull/3376
* [tests] Fix bnb cpu error by faaany in https://github.com/huggingface/accelerate/pull/3351
* fix torch_dtype in estimate memory by SunMarc in https://github.com/huggingface/accelerate/pull/3383
* works for fp8 with deepspeed by XiaobingSuper in https://github.com/huggingface/accelerate/pull/3361
* fix: typos in documentation files by maximevtush in https://github.com/huggingface/accelerate/pull/3388
* [examples] upgrade code for seed setting by faaany in https://github.com/huggingface/accelerate/pull/3387
* [`memory leak`] Replace GradientState -> DataLoader reference with weakrefs by tomaarsen in https://github.com/huggingface/accelerate/pull/3391
* add xpu check in `get_quantized_model_device_map` by faaany in https://github.com/huggingface/accelerate/pull/3397
* Torchao float8 training by muellerzr in https://github.com/huggingface/accelerate/pull/3348

New Contributors
* kmehant made their first contribution in https://github.com/huggingface/accelerate/pull/3173
* XiaobingSuper made their first contribution in https://github.com/huggingface/accelerate/pull/3361
* maximevtush made their first contribution in https://github.com/huggingface/accelerate/pull/3388

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.3.0...v1.4.0

* fix: add max_memory to _init_infer_auto_device_map's return statement in https://github.com/huggingface/accelerate/pull/3279 by Nech-C
* fix load_state_dict for npu in https://github.com/huggingface/accelerate/pull/3211 by statelesshz

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.2.0...v1.2.1