Deepspeed

* [Press release] [DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression](https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/)
* New inference API [inference setup](https://deepspeed.readthedocs.io/en/latest/inference-init.html)
* [DeepSpeed Inference: Multi-GPU inference with customized inference kerenls and quantization support](https://www.deepspeed.ai/news/2021/03/15/inference-kernel-optimization.html)
* [Getting Started with DeepSpeed for Inferencing Transformer based Models](https://www.deepspeed.ai/tutorials/inference-tutorial/)
* [Mixture-of-Quantization: A novel quantization approach for reducing model size with minimal accuracy impact](https://www.deepspeed.ai/news/2020/05/27/MoQ.html)
* [MoQ tutorial](https://www.deepspeed.ai/tutorials/MoQ-tutorial/) for more details.

* Full precision (fp32) support for ZeRO Stage2 and Stage3 (1004)
* 1-bit LAMB optimizer (970)
* Refactor param_dict to config (1008)
* [ZeRO Infinity] Allow Init to take a dict for the deepspeed config (983)
* Bug fix for zero-1 w.r.t. reduce-scatter (907)
* Use amp autocast in ZeRO3 linear (990)
* Add find_unused_parameters option to DeepSpeedEngine (945)
* Relax dataset type check in deepspeed io (1012)

Special thanks to our contributors: stas00, SeanNaren, sdtblck, wbuchwalter, ghosthamlet, zhujiangang,