Accelerate

Model quantization with bitsandbytes

You can now quantize any model (no just Transformer models) using Accelerate. This is mainly for models having a lot of linear layers. See the [documentation](https://huggingface.co/docs/accelerate/usage_guides/quantization) for more information!

* Bnb quantization by SunMarc in 1626

Support for Ascend NPUs

Accelerate now supports Ascend NPUs.

* Add Ascend NPU accelerator support by statelesshz in 1676

What's new?

Accelerate now requires Python 3.8+ and PyTorch 1.10+ :

* 🚨🚨🚨 Spring cleaning: Python 3.8 🚨🚨🚨 by muellerzr in 1661
* 🚨🚨🚨 Spring cleaning: PyTorch 1.10 🚨🚨🚨 by muellerzr in 1662


* [doc build] Use secrets by mishig25 in 1551
* Update launch.mdx by LiamSwayne in 1553
* Avoid double wrapping of all accelerate.prepare objects by muellerzr in 1555
* Update README.md by LiamSwayne in 1556
* Fix load_state_dict when there is one device and disk by sgugger in 1557
* Fix tests not being ran on multi-GPU nightly by muellerzr in 1558
* fix the typo when setting the "_accelerator_prepared" attribute by Yura52 in 1560
* [`core`] Fix possibility to pass`NoneType` objects in `prepare` by younesbelkada in 1561
* Reset dataloader end_of_datalaoder at each iter by sgugger in 1562
* Update big_modeling.mdx by LiamSwayne in 1564
* [`bnb`] Fix failing int8 tests by younesbelkada in 1567
* Update gradient sync docs to reflect importance of `optimizer.step()` by dleve123 in 1565
* Update mixed precision integrations in README by sgugger in 1569
* Raise error instead of warn by muellerzr in 1568
* Introduce listify, fix tensorboard silently failing by muellerzr in 1570
* Check for bak and expand docs on directory structure by muellerzr in 1571
* Perminant solution by muellerzr in 1577
* fix the bug in xpu by mingxiaoh in 1508
* Make sure that we only set is_accelerator_prepared on items accelerate actually prepares by muellerzr in 1578
* Expand `prepare()` doc by muellerzr in 1580
* Get Torch version using importlib instead of pkg_resources by catwell in 1585
* improve oob performance when use mpirun to start DDP finetune without `accelerate launch` by sywangyi in 1575
* Update training_tpu.mdx by LiamSwayne in 1582
* Return false if CUDA available by muellerzr in 1581
* fix logger level by caopulan in 1579
* Fix test by muellerzr in 1586
* Update checkpoint.mdx by LiamSwayne in 1587
* FSDP updates by pacman100 in 1576
* Update modeling.py by ain-soph in 1595
* Integration tests by muellerzr in 1593
* Add triggers for CI workflow by muellerzr in 1597
* Remove asking xpu plugin for non xpu devices by abhilash1910 in 1594
* Remove GPU safetensors env variable by sgugger in 1603
* reset end_of_dataloader for dataloader_dispatcher by megavaz in 1609
* fix for arc gpus by abhilash1910 in 1615
* Ignore low_zero option when only device is available by sgugger in 1617
* Fix failing multinode tests by muellerzr in 1616
* Doc to md by sgugger in 1618
* Fix tb issue by muellerzr in 1623
* Fix workflow by muellerzr in 1625
* Fix transformers sync bug with accumulate by muellerzr in 1624
* fixes offload dtype by SunMarc in 1631
* fix: Megatron is not installed. please build it from source. by yuanwu2017 in 1636
* deepspeed z2/z1 state_dict bloating fix by pacman100 in 1638
* Swap disable rich by muellerzr in 1640
* fix autocasting bug by pacman100 in 1637
* fix modeling low zero by abhilash1910 in 1634
* Add skorch to runners by muellerzr in 1646
* add save model by SunMarc in 1641
* Change dispatch_model when we have only one device by SunMarc in 1648
* Doc save model by SunMarc in 1650
* Fix device_map by SunMarc in 1651
* Check for port usage before launch by muellerzr in 1656
* [`BigModeling`] Add missing check for quantized models by younesbelkada in 1652
* Bump integration by muellerzr in 1658
* TIL by muellerzr in 1657
* docker cpu py version by muellerzr in 1659
* [`BigModeling`] Final fix for dispatch int8 and fp4 models by younesbelkada in 1660
* remove safetensor dep on shard_checkpoint by SunMarc in 1664
* change the import place to avoid import error by pacman100 in 1653
* Update broken Runhouse link in examples/README.md by dongreenberg in 1668
* Bnb quantization by SunMarc in 1626
* replace save funct in doc by SunMarc in 1672
* Doc big model inference by SunMarc in 1670
* Add docs for saving Transformers models by deppen8 in 1671
* fix bnb tests by SunMarc in 1679
* Fix workflow CI by muellerzr in 1690
* remove duplicate class by SunMarc in 1691
* update readme in examples by statelesshz in 1678
* Fix nightly tests by muellerzr in 1696
* Fixup docs by muellerzr in 1697
* Improve quality errors by muellerzr in 1698
* Move mixed precision wrapping ahead of DDP/FSDP wrapping by ChenWu98 in 1682
* Add offload for 8-bit model by SunMarc in 1699
* Deepcopy on Accelerator to return self by muellerzr in 1694
* Update tracking.md by stevhliu in 1702
* Skip tests when bnb isn't available by muellerzr in 1706
* Fix launcher validation by abhilash1910 in 1705
* Fixes for issue 1683: failed to run accelerate config in colab by Erickrus in 1692
* Fix the bug where DataLoaderDispatcher gets stuck in an infinite wait when the dataset is an IterDataPipe during multi-process training. by yuxinyuan in 1709
* add multi_gpu decorator by SunMarc in 1712
* Modify loading checkpoint behavior by SunMarc in 1715
* fix version by SunMarc in 1701
* Keep old behavior by muellerzr in 1716
* Optimize `get_scale` to reduce async calls by muellerzr in 1718
* Remove duplicate code by muellerzr in 1717
* New tactic by muellerzr in 1719
* add Comfy-UI by pacman100 in 1723
* add compatibility with peft by SunMarc in 1725

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* LiamSwayne
* Update launch.mdx (1553)
* Update README.md (1556)
* Update big_modeling.mdx (1564)
* Update training_tpu.mdx (1582)
* Update checkpoint.mdx (1587)
* mingxiaoh
* fix the bug in xpu (1508)
* statelesshz
* update readme in examples (1678)
* Add Ascend NPU accelerator support (1676)
* ChenWu98
* Move mixed precision wrapping ahead of DDP/FSDP wrapping (1682)

- Reset dataloader end_of_datalaoder at each iter in 1562 by sgugger

- fix the typo when setting the "_accelerator_prepared" attribute in 1560 by Yura52
- [core] Fix possibility to pass] `NoneType` objects in `prepare` in 1561 by younesbelkada

- Avoid double wrapping of all accelerate.prepare objects by muellerzr in 1555
- Fix load_state_dict when there is one device and disk by sgugger in 1557

Big model inference

Support has been added to run `device_map="auto"` on the MPS device. Big model inference also work with models loaded in 4 bits in Transformers.

* Add mps support to big inference modeling by SunMarc in 1545
* Adds fp4 support for model dispatching by younesbelkada in 1505

4-bit QLoRA Support

* 4-bit QLoRA via bitsandbytes (4-bit base model + LoRA) by TimDettmers in 1458

Distributed Inference Utilities

This version introduces a new `Accelerator.split_between_processes` utility to help with performing distributed infernece with non-tensorized or non-dataloader workflows. Read more [here](https://huggingface.co/docs/accelerate/usage_guides/distributed_inference)

Introduce XPU support for Intel GPU

* Intel GPU support initialization by abhilash1910 in 1118

Add support for the new PyTorch XLA TPU runtime

* Accelerate now supports the latest TPU runtimes 1393, 1385

A new optimizer method: `LocalSGD`

* This is a new wrapper around SGD which enables efficient multi-GPU training in the case when no fast interconnect is possible by searchivarius in 1378

Papers with 🤗 Accelerate

* We now have an entire section of the docs dedicated to official paper implementations and citations using the framework 1399, see it live [here](https://hf.co/docs/accelerate/usage_guides/training_zoo#in-science)

Breaking changes

`logging_dir` has been fully deprecated, please use `project_dir` or a `Project_configuration`

What's new?

* use existing mlflow experiment if exists by Rusteam in 1403
* changes required for DS integration by pacman100 in 1406
* fix deepspeed failing tests by pacman100 in 1411
* Make mlflow logging dir optional by mattplo-decath in 1413
* Fix bug on ipex for diffusers by abhilash1910 in 1426
* Improve Slack Updater by muellerzr in 1433
* Let quality yell at the user if it's a version difference by muellerzr in 1438
* Ensure that it gets installed by muellerzr in 1439
* [`core`] Introducing `CustomDtype` enum for custom dtypes by younesbelkada in 1434
* Fix XPU by muellerzr in 1440
* Make sure torch compiled model can also be unwrapped by patrickvonplaten in 1437
* fixed: ZeroDivisionError: division by zero by sreio in 1436
* fix potential OOM when resuming with multi-GPU training by exhyy in 1444
* Fixes in infer_auto_device_map by sgugger in 1441
* Raise error when logging improperly by muellerzr in 1446
* Fix ci by muellerzr in 1447
* Distributed prompting/inference utility by muellerzr in 1410
* Add to by muellerzr in 1448
* split_between_processes by stevhliu in 1449
* [docs] Replace `state.rank` -> `process_index` by pcuenca in 1450
* Auto multigpu logic by muellerzr in 1452
* Update with cli instructions by muellerzr in 1453
* Adds `in_order` argument that defaults to False, to log in order. by JulesGM in 1262
* fix error for CPU DDP using trainer api. by sywangyi in 1455
* Refactor and simplify xpu device in state by abhilash1910 in 1456
* Document how to use commands with python module instead of argparse by muellerzr in 1457
* 4-bit QLoRA via bitsandbytes (4-bit base model + LoRA) by TimDettmers in 1458
* Fix skip first batch being perminant by muellerzr in 1466
* update conversion of layers to retain original data type. by avisinghal6 in 1467
* Check for xpu specifically by muellerzr in 1472
* update `register_empty_buffer` to match torch args by NouamaneTazi in 1465
* Update gradient accumulation docs, and remove redundant example by iantbutler01 in 1461
* Imrpove sagemaker by muellerzr in 1470
* Split tensors as part of `split_between_processes` by muellerzr in 1477
* Move to device by muellerzr in 1478
* Fix gradient state bugs in multiple dataloader by Ethan-yt in 1483
* Add rdzv-backend by muellerzr in 1490
* Only use IPEX if available by muellerzr in 1495
* Update README.md by lyhue1991 in 1493
* Let gather_for_metrics always run by muellerzr in 1496
* Use empty like when we only need to create buffers by thomasw21 in 1497
* Allow key skipping in big model inference by sgugger in 1491
* fix crash when ipex is installed and torch has no xpu by sywangyi in 1502
* [`bnb`] Add fp4 support for dispatch by younesbelkada in 1505
* Fix 4bit model on multiple devices by SunMarc in 1506
* adjust overriding of model's forward function by prathikr in 1492
* Add assertion when call prepare with deepspeed config. by tensimiku in 1468
* NVME path support for deepspeed by abhilash1910 in 1484
* should set correct dtype to ipex optimize and use amp logic in native… by sywangyi in 1511
* Swap env vars for XPU and IPEX + CLI by muellerzr in 1513
* Fix a bug when parameters tied belong to the same module by sgugger in 1514
* Fixup deepspeed/cli tests by muellerzr in 1526
* Refactor mp into its own wrapper by muellerzr in 1527
* Check tied parameters by SunMarc in 1529
* Raise ValueError on iterable dataset if we've hit the end and attempting to go beyond it by muellerzr in 1531
* Officially support naive PP for quantized models + PEFT by younesbelkada in 1523
* remove ipexplugin, let ACCELERATE_USE_IPEX/ACCELERATE_USE_XPU control the ipex and xpu by sywangyi in 1503
* Prevent using extra VRAM for static device_map by LSerranoPEReN in 1536
* Update deepspeed.mdx by LiamSwayne in 1541
* Update performance.mdx by LiamSwayne in 1543
* Update deferring_execution.mdx by LiamSwayne in 1544
* Apply deprecations by muellerzr in 1537
* Add mps support to big inference modeling by SunMarc in 1545
* [documentation] grammar fixes in gradient_synchronization.mdx by LiamSwayne in 1547
* Eval mode by muellerzr in 1540
* Update migration.mdx by LiamSwayne in 1549

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* will-cromar
* Support TPU v4 with new PyTorch/XLA TPU runtime (1393)
* Support TPU v2 and v3 on new PyTorch/XLA TPU runtime (1385)
* searchivarius
* Adding support for local SGD. (1378)
* abhilash1910
* Intel GPU support initialization (1118)
* Fix bug on ipex for diffusers (1426)
* Refactor and simplify xpu device in state (1456)
* NVME path support for deepspeed (1484)
* sywangyi
* fix error for CPU DDP using trainer api. (1455)
* fix crash when ipex is installed and torch has no xpu (1502)
* should set correct dtype to ipex optimize and use amp logic in native… (1511)
* remove ipexplugin, let ACCELERATE_USE_IPEX/ACCELERATE_USE_XPU control the ipex and xpu (1503)
* Ethan-yt
* Fix gradient state bugs in multiple dataloader (1483)

What's New

- Support for Intel IPEX support has been added, check out the how-to guide [now!](https://huggingface.co/docs/accelerate/main/en/usage_guides/ipex)
- Various modifications have been added to begin work on having 🤗 Accelerate be the foundation for the `Trainer`, keep an eye on the repos to see how our progress is coming along!
- FP8 training is now supported on Ada Lovelance GPUs
- The `wandb` integration now supports logging of images and tables through `tracker.log_images` and `tracker.log_tables` respectively
- Many, many squashed bugs! (see the full detailed report for just what they were)
- **17** new contributors to the framework, congratulations to all who took their first step! 🚀

What's Changed

* Fix pypi image by muellerzr in https://github.com/huggingface/accelerate/pull/1249
* raise error when dataloader with None as batch_size when using DS by pacman100 in https://github.com/huggingface/accelerate/pull/1250
* Handle missing deepspeed config by HeyangQin in https://github.com/huggingface/accelerate/pull/1251
* [`core`] Add Quantization support for `dispatch_model` by younesbelkada in https://github.com/huggingface/accelerate/pull/1237
* Check attribute 'overflow' exists in optimizer. by tensimiku in https://github.com/huggingface/accelerate/pull/1259
* ipex intel extension for pytorch integration by sywangyi in https://github.com/huggingface/accelerate/pull/1255
* fix issue template by stas00 in https://github.com/huggingface/accelerate/pull/1264
* Change error raised to ValueError by sgugger in https://github.com/huggingface/accelerate/pull/1267
* Fix reduce operation by xyfJASON in https://github.com/huggingface/accelerate/pull/1268
* Raise import error if fp8 not available in `has_transfomer_engine_layers` by muellerzr in https://github.com/huggingface/accelerate/pull/1283
* Add missing FP8 options to CLI by muellerzr in https://github.com/huggingface/accelerate/pull/1284
* Update quicktour.mdx by standardAI in https://github.com/huggingface/accelerate/pull/1273
* Minor fix whitespace colon by guspan-tanadi in https://github.com/huggingface/accelerate/pull/1272
* fix attribute error in DataloaderShared by ZhiyuanChen in https://github.com/huggingface/accelerate/pull/1278
* Fix TypeError bug in honor_type by muellerzr in https://github.com/huggingface/accelerate/pull/1285
* Raise more explicit error when transformer_engine isn't installed by muellerzr in https://github.com/huggingface/accelerate/pull/1287
* Expound error on `recursively_apply` by muellerzr in https://github.com/huggingface/accelerate/pull/1286
* Only check for dtype if it has it in get_state_dict by muellerzr in https://github.com/huggingface/accelerate/pull/1288
* [`bnb`] fix bnb slow test by younesbelkada in https://github.com/huggingface/accelerate/pull/1292
* Raise better error on `notebook_launcher` by muellerzr in https://github.com/huggingface/accelerate/pull/1293
* Make note about grad accum and prec in performance documentation by muellerzr in https://github.com/huggingface/accelerate/pull/1296
* fix for load_checkpoint_and_dispatch(device_map=None) by anentropic in https://github.com/huggingface/accelerate/pull/1297
* Set the state device dependant to Accelerator on multigpu by muellerzr in https://github.com/huggingface/accelerate/pull/1220
* add usage guide for ipex plugin by sywangyi in https://github.com/huggingface/accelerate/pull/1270
* Simplify MPS implementation by sgugger in https://github.com/huggingface/accelerate/pull/1308
* Bug fix in setattr by aashiqmuhamed in https://github.com/huggingface/accelerate/pull/1312
* Allow xpu backend by muellerzr in https://github.com/huggingface/accelerate/pull/1313
* Default to nccl by muellerzr in https://github.com/huggingface/accelerate/pull/1314
* offload the previous module hook before the current module is moved to… by williamberman in https://github.com/huggingface/accelerate/pull/1315
* Ensure that dynamo is compatible with mixed precision by muellerzr in https://github.com/huggingface/accelerate/pull/1318
* Upgrade torch version on main tests by muellerzr in https://github.com/huggingface/accelerate/pull/1323
* Add test flag and import check for dynamo by muellerzr in https://github.com/huggingface/accelerate/pull/1322
* ensure module prefixes only match that module by xloem in https://github.com/huggingface/accelerate/pull/1319
* remove repetitive entries from device lists by xloem in https://github.com/huggingface/accelerate/pull/1321
* Fix failing test on main by muellerzr in https://github.com/huggingface/accelerate/pull/1332
* Verbosity, Progress Bar for Loading by xloem in https://github.com/huggingface/accelerate/pull/1329
* Skip failing torch 2.0+ test by muellerzr in https://github.com/huggingface/accelerate/pull/1339
* Remove unused amp import util by muellerzr in https://github.com/huggingface/accelerate/pull/1340
* Fix nested context manager for main_process_first() by flukeskywalker in https://github.com/huggingface/accelerate/pull/1304
* Small progress bar fix by xloem in https://github.com/huggingface/accelerate/pull/1341
* Pop more backend options by muellerzr in https://github.com/huggingface/accelerate/pull/1342
* Support FP8 mixed precision training for Ada Lovelace GPUs by Dango233 in https://github.com/huggingface/accelerate/pull/1348
* using deepspeed.comm for distrbiuted init by pacman100 in https://github.com/huggingface/accelerate/pull/1352
* [`bnb`] Fix bnb slow test by younesbelkada in https://github.com/huggingface/accelerate/pull/1355
* Better check for packages availability by apbard in https://github.com/huggingface/accelerate/pull/1356
* fix: typing issues, and replace deprecated python typing (Optional, Union) to `|` by kiyoon in https://github.com/huggingface/accelerate/pull/1363
* Fix default FSDP_MIN_NUM_PARAMS so it's an int by sam-hieken in https://github.com/huggingface/accelerate/pull/1367
* Special transformers case from args by muellerzr in https://github.com/huggingface/accelerate/pull/1364
* Improve `accelerate env` reporting by muellerzr in https://github.com/huggingface/accelerate/pull/1376
* Seperate out contextmanager generation by muellerzr in https://github.com/huggingface/accelerate/pull/1379
* delete textfile after tests are done by muellerzr in https://github.com/huggingface/accelerate/pull/1381
* Fix flakey thread issue by muellerzr in https://github.com/huggingface/accelerate/pull/1387
* fix config bug for 'mixed_precision' from 'yaml.safe_load()' by ys-eric-choi in https://github.com/huggingface/accelerate/pull/1386
* Log Images and other types to wandb by tcapelle in https://github.com/huggingface/accelerate/pull/962
* Bump torch version by muellerzr in https://github.com/huggingface/accelerate/pull/1392
* Fix gather_obj by muellerzr in https://github.com/huggingface/accelerate/pull/1391
* Update training_zoo.mdx by yuvalkirstain in https://github.com/huggingface/accelerate/pull/1397

New Contributors
* HeyangQin made their first contribution in https://github.com/huggingface/accelerate/pull/1251
* tensimiku made their first contribution in https://github.com/huggingface/accelerate/pull/1259
* xyfJASON made their first contribution in https://github.com/huggingface/accelerate/pull/1268
* standardAI made their first contribution in https://github.com/huggingface/accelerate/pull/1273
* guspan-tanadi made their first contribution in https://github.com/huggingface/accelerate/pull/1272
* anentropic made their first contribution in https://github.com/huggingface/accelerate/pull/1297
* aashiqmuhamed made their first contribution in https://github.com/huggingface/accelerate/pull/1312
* williamberman made their first contribution in https://github.com/huggingface/accelerate/pull/1315
* xloem made their first contribution in https://github.com/huggingface/accelerate/pull/1319
* flukeskywalker made their first contribution in https://github.com/huggingface/accelerate/pull/1304
* Dango233 made their first contribution in https://github.com/huggingface/accelerate/pull/1348
* apbard made their first contribution in https://github.com/huggingface/accelerate/pull/1356
* kiyoon made their first contribution in https://github.com/huggingface/accelerate/pull/1363
* sam-hieken made their first contribution in https://github.com/huggingface/accelerate/pull/1367
* ys-eric-choi made their first contribution in https://github.com/huggingface/accelerate/pull/1386
* tcapelle made their first contribution in https://github.com/huggingface/accelerate/pull/962
* yuvalkirstain made their first contribution in https://github.com/huggingface/accelerate/pull/1397

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.18.0...v0.19.0