Accelerate

This release offers no significant new API, it is just needed to have access to some utils in Transformers.

* Handle deprication errors in launch by muellerzr in 360
* Update launchers.py by tmabraham in 363
* fix tracking by pacman100 in 361
* Remove tensor call by muellerzr in 365
* Add a utility for writing a barebones config file by muellerzr in 371
* fix deepspeed model saving by pacman100 in 370
* deepspeed save model temp fix by pacman100 in 374
* Refactor tests to use accelerate launch by muellerzr in 373
* fix zero stage-1 by pacman100 in 378
* fix shuffling for ShufflerIterDataPipe instances by loubnabnl in 376
* Better check for deepspeed availability by sgugger in 379
* Refactor some parts in utils by sgugger in 380

Big model inference

To handle very large models, new functionality has been added in Accelerate:
- a context manager to initalize empty models
- a function to load a sharded checkpoint directly on the right devices
- a set of custom hooks that allow execution of a model split on different devices, as well as CPU or disk offload
- a magic method that auto-determines a device map for a given model, maximizing the GPU spaces, available RAM before using disk offload as a last resort.
- a function that wraps the last three blocks in one simple call (`load_checkpoint_and_dispatch`)

See more in the [documentation](https://huggingface.co/docs/accelerate/main/en/big_modeling)

* Big model inference by sgugger in 345

What's new

* Create peak_memory_uasge_tracker.py by pacman100 in 336
* Fixed a typo to enable running accelerate correctly by Idodox in 339
* Introduce multiprocess logger by muellerzr in 337
* Refactor utils into its own module by muellerzr in 340
* Improve num_processes question in CLI by muellerzr in 343
* Handle Manual Wrapping in FSDP. Minor fix of fsdp example. by pacman100 in 342
* Better prompt for number of training devices by muellerzr in 344
* Fix prompt for num_processes by pacman100 in 347
* Fix sample calculation in examples by muellerzr in 352
* Fixing metric eval in distributed setup by pacman100 in 355
* DeepSpeed and FSDP plugin support through script by pacman100 in 356

- Fix fdsp config in cluster [331](https://github.com/huggingface/accelerate/pull/331)
- Add guards for batch size finder [334](https://github.com/huggingface/accelerate/pull/334)
- Patchfix infinite loop [335](https://github.com/huggingface/accelerate/pull/335)

Logging API

Use any of your favorite logging libraries (TensorBoard, Wandb, CometML...) with just a few lines of code inside your training scripts with Accelerate. All details are in the [documentation](https://huggingface.co/docs/accelerate/tracking).

* Add logging capabilities by muellerzr in https://github.com/huggingface/accelerate/pull/293

Support for FSDP (fully sharded DataParallel)

PyTorch recently released a new model wrapper for sharded DDP training called [FSDP](https://pytorch.org/docs/stable/fsdp.html). This release adds support for it (note that it doesn't work with mixed precision yet). See all caveats in the [documentation](https://huggingface.co/docs/accelerate/fsdp).

* PyTorch FSDP Feature Incorporation by pacman100 in https://github.com/huggingface/accelerate/pull/321

Batch size finder

Say goodbye to the CUDA OOM errors with the new `find_executable_batch_size` decorator. Just decorate your training function and pick a starting batch size, then let Accelerate do the rest.

* Add a memory-aware decorator for CUDA OOM avoidance by muellerzr in https://github.com/huggingface/accelerate/pull/324

Examples revamp

The [Accelerate examples](https://github.com/huggingface/accelerate/tree/main/examples) are now split in two: you can find in the base folder a very simple nlp and computer vision examples, as well as complete versions incorporating all features. But you can also browse the examples in the `by_feature` subfolder, which will show you exactly what code to add for each given feature (checkpointing, tracking, cross-validation etc.)

* Refactor Examples by Feature by muellerzr in https://github.com/huggingface/accelerate/pull/312

What's Changed
* Document save/load state by muellerzr in https://github.com/huggingface/accelerate/pull/290
* Refactor precisions to its own enum by muellerzr in https://github.com/huggingface/accelerate/pull/292
* Load model and optimizet states on CPU to void OOMs by sgugger in https://github.com/huggingface/accelerate/pull/299
* Fix example for datasets v2 by sgugger in https://github.com/huggingface/accelerate/pull/298
* Leave default as None in `mixed_precision` for launch command by sgugger in https://github.com/huggingface/accelerate/pull/300
* Pass `lr_scheduler` to `Accelerator.prepare` by sgugger in https://github.com/huggingface/accelerate/pull/301
* Create new TestCase classes and clean up W&B tests by muellerzr in https://github.com/huggingface/accelerate/pull/304
* Have custom trackers work with the API by muellerzr in https://github.com/huggingface/accelerate/pull/305
* Write tests for comet_ml by muellerzr in https://github.com/huggingface/accelerate/pull/306
* Fix training in DeepSpeed by sgugger in https://github.com/huggingface/accelerate/pull/308
* Update example scripts by muellerzr in https://github.com/huggingface/accelerate/pull/307
* Use --no_local_rank for DeepSpeed launch by sgugger in https://github.com/huggingface/accelerate/pull/309
* Fix Accelerate CLI CPU option + small fix for W&B tests by muellerzr in https://github.com/huggingface/accelerate/pull/311
* Fix DataLoader sharding for deepspeed in accelerate by m3rlin45 in https://github.com/huggingface/accelerate/pull/315
* Create a testing framework for example scripts and fix current ones by muellerzr in https://github.com/huggingface/accelerate/pull/313
* Refactor Tracker logic and write guards for logging_dir by muellerzr in https://github.com/huggingface/accelerate/pull/316
* Create Cross-Validation example by muellerzr in https://github.com/huggingface/accelerate/pull/317
* Create alias for Accelerator.free_memory by muellerzr in https://github.com/huggingface/accelerate/pull/318
* fix typo in docs of accelerate tracking by loubnabnl in https://github.com/huggingface/accelerate/pull/320
* Update examples to show how to deal with extra validation copies by muellerzr in https://github.com/huggingface/accelerate/pull/319
* Fixup all checkpointing examples by muellerzr in https://github.com/huggingface/accelerate/pull/323
* Introduce reduce operator by muellerzr in https://github.com/huggingface/accelerate/pull/326

New Contributors
* m3rlin45 made their first contribution in https://github.com/huggingface/accelerate/pull/315
* loubnabnl made their first contribution in https://github.com/huggingface/accelerate/pull/320
* pacman100 made their first contribution in https://github.com/huggingface/accelerate/pull/321

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.6.0...v0.7.0

The launcher was ignoring the mixed precision attribute of the config since v0.6.0. This patch fixes that.

Patches an issue with mixed precision (see 286)