Deepspeed

What's Changed
* Add blob storage to CI runners by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2260
* Update replace_module.py, test-gptj.py related fix by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2269
* Fix OrderedDict import for python3.6 by Dipet in https://github.com/microsoft/DeepSpeed/pull/2267
* Ds inference/fix mp2 by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2270
* Trajepl: nebula load fix by trajepl in https://github.com/microsoft/DeepSpeed/pull/2182
* Prevent torch ext folder mkdir at tmp by jeffra in https://github.com/microsoft/DeepSpeed/pull/2274
* Ds-inference Int8 support through ZeroQuant technology by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2217
* add a new unit test for cuda ops by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2278
* Addition to code owners file by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2279
* Memory Access Utility by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2276
* Fp32 accuracy bug fix by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2285
* Refactor universal checkpointing and tensor fragments by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2253
* [ds-inference] fix progress bar by stas00 in https://github.com/microsoft/DeepSpeed/pull/2286
* Offload all gradients to nvme by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2282
* fused bias relu unittest by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2297
* Fix for pytest picking up wrong deepspeed by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2299
* Fix for Zero3 when MP>1 by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2289
* Unit test for bias add kernel by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2298
* Update relu.cu with mem_access_utils by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2306
* Add tensor parallel inference unit tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2232
* Fix the residual add mp scaling for GPTNeoX by arashb in https://github.com/microsoft/DeepSpeed/pull/2310
* Add unit tests for residual_add kernel by arashb in https://github.com/microsoft/DeepSpeed/pull/2307
* add inference eval scripts by jeffra in https://github.com/microsoft/DeepSpeed/pull/2303
* Upgrade P40 tests to torch 1.8 by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2316
* ZeRO-Inference blog by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2271
* ZeRO-Inference blog - wrap up by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2321
* ZeRO-Inference blog - Update README by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2322
* Refactor relu bias add with mem_access utils by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2317
* add quant unit test by GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2315
* only override forward if using cuda-graph by jeffra in https://github.com/microsoft/DeepSpeed/pull/2291
* Add more options to inference benchmark by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2325

New Contributors
* molly-smith made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2269

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.2...v0.7.3

What's Changed
* Enable contiguous gradients with Z1+MoE by siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2250
* Correctly detect CPU optimizer usage by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2257
* Update Half Precision Kernel Compatibility by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2261
* fix 2240: wrong time unit in flops_profiler by yzs981130 in https://github.com/microsoft/DeepSpeed/pull/2241

New Contributors
* cmikeh2 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2261
* yzs981130 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2241

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.1...v0.7.2

What's Changed
* Fix for distributed tests on pytorch>=1.12 by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2141
* delay torch import for inference compatability check by jeffra in https://github.com/microsoft/DeepSpeed/pull/2167
* Fix wrong unit of latency in flops-profiler (2090) by zionwu in https://github.com/microsoft/DeepSpeed/pull/2095
* [docs] adoption updates by jeffra in https://github.com/microsoft/DeepSpeed/pull/2173
* Update for AMD CI workflow by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2172
* [docs] update offload docs to include stage 1 by jeffra in https://github.com/microsoft/DeepSpeed/pull/2178
* Fixing model partitioning without injection by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2179
* Match compute and reduce dtype by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2145
* Enable fused_lamb_cuda_kernel on ROCm by rraminen in https://github.com/microsoft/DeepSpeed/pull/2148
* Update README to latest Composer version by hanlint in https://github.com/microsoft/DeepSpeed/pull/2177
* [deepspeed/autotuner] Missing hjson import by rahilbathwal5 in https://github.com/microsoft/DeepSpeed/pull/2175
* [docs] add more models to adoption by jeffra in https://github.com/microsoft/DeepSpeed/pull/2189
* [CI] fix lightning tests by jeffra in https://github.com/microsoft/DeepSpeed/pull/2190
* Fix typos on README.md by gasparitiago in https://github.com/microsoft/DeepSpeed/pull/2192
* Fix the layer-past for GPT based models by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2196
* Add gradient_average flag support for sparse grads by Dipet in https://github.com/microsoft/DeepSpeed/pull/2188
* Adding the compression tutorial on GPT distillation and quantization by minjiaz in https://github.com/microsoft/DeepSpeed/pull/2197
* Log user config exactly by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2201
* Fix the tensor-slicing copy for qkv parameters by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2198
* Refactor Distributed Tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2180
* fix table syntax by kamalkraj in https://github.com/microsoft/DeepSpeed/pull/2204
* Correctly detect offload configuration by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2208
* add cuda 11.7 by jeffra in https://github.com/microsoft/DeepSpeed/pull/2211
* use torch 1.9 in accelerate tests by jeffra in https://github.com/microsoft/DeepSpeed/pull/2215
* [zero-3] print warning once and support torch parameter by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2127
* Add support of OPT models by arashb in https://github.com/microsoft/DeepSpeed/pull/2205
* fix typos in readme. by zhjohnchan in https://github.com/microsoft/DeepSpeed/pull/2218
* Fix regression w. dist_init_required by jeffra in https://github.com/microsoft/DeepSpeed/pull/2225
* add doc for new bert example by conglongli in https://github.com/microsoft/DeepSpeed/pull/2224
* Remove the random-generator from context during inference by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2228
* allow saving ckpt w/o ckpt json + bloom copy fix by jeffra in https://github.com/microsoft/DeepSpeed/pull/2237
* Correctly detect zero_offload by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2213
* [docs] update community videos by jeffra in https://github.com/microsoft/DeepSpeed/pull/2249
* Refactor dist tests: Checkpointing by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2202
* Make OPT policy backward compatible with pre-OPT transformers versions by arashb in https://github.com/microsoft/DeepSpeed/pull/2254
* fix ds-inference without policy by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2247

New Contributors
* zionwu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2095
* hanlint made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2177
* rahilbathwal5 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2175
* gasparitiago made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2192
* arashb made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2205
* zhjohnchan made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2218

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.0...v0.7.1

New features
* DeepSpeed Compression: https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/

What's Changed
* Adding DeepSpeed Compression Composer by yaozhewei in https://github.com/microsoft/DeepSpeed/pull/2105
* Remove hardcoded ROCm install path by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2093
* Fix softmax dim of Residual MoE implementation in moe/layer.py by hero007feng in https://github.com/microsoft/DeepSpeed/pull/2110
* reduce ds-inference log verbosity by jeffra in https://github.com/microsoft/DeepSpeed/pull/2111
* DeepSpeed Compression announcement by conglongli in https://github.com/microsoft/DeepSpeed/pull/2114
* Checkpoint reshaping by tjruwase in https://github.com/microsoft/DeepSpeed/pull/1953
* Fix init_process_group by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2121
* DS Benchmarks QoL Improvements by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2120
* [ROCm] Wrong command broke ROCm build. by jpvillam-amd in https://github.com/microsoft/DeepSpeed/pull/2118
* DeepSpeed Communication Profiling and Logging by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2012
* Add flake8 to pre-commit checks by aphedges in https://github.com/microsoft/DeepSpeed/pull/2051
* Fix conflict between Tutel and top-2 gate in MoE layer by yetiansh in https://github.com/microsoft/DeepSpeed/pull/2053
* adding HF Accelerate+DS tests workflow by pacman100 in https://github.com/microsoft/DeepSpeed/pull/2134
* [inference tests] turn off time check for now by jeffra in https://github.com/microsoft/DeepSpeed/pull/2142
* Allow turning off loss scaling wrt GAS + update tput calculator by jeffra in https://github.com/microsoft/DeepSpeed/pull/2140
* Refactor ZeRO configs to use Pydantic by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2004
* Add purely-local sliding window sparse attention config by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/1962
* Trajepl/nebula ckpt engine by trajepl in https://github.com/microsoft/DeepSpeed/pull/2085
* Graceful exit on failures for multi-node runs by jerrymannil in https://github.com/microsoft/DeepSpeed/pull/2008
* fix: fix BF16_Optimizer compatibility issue by shjwudp in https://github.com/microsoft/DeepSpeed/pull/2152
* Fix random token-generation issue + MP-checkpoint loading/saving by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2132
* Added retain_graph as a kwarg to the main engine backward function by ncilfone in https://github.com/microsoft/DeepSpeed/pull/1149
* Elastic Training support in DeepSpeed by aj-prime in https://github.com/microsoft/DeepSpeed/pull/2156
* prevent cuda 10 builds of inference kernels on ampere by jeffra in https://github.com/microsoft/DeepSpeed/pull/2157
* [zero-3] shutdown zero.Init from within ds.init by jeffra in https://github.com/microsoft/DeepSpeed/pull/2150
* enable fp16 input autocasting by jeffra in https://github.com/microsoft/DeepSpeed/pull/2158
* Release swap buffers for persisted params by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2089
* Tensor parallelism for Mixture of Experts by siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2074

New Contributors
* hero007feng made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2110
* jpvillam-amd made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2118
* yetiansh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2053
* pacman100 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2134
* jimwu6 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2144
* trajepl made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2085
* ncilfone made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1149

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.6.7...v0.7.0

What's Changed
* Add Inference support for running the BigScience-BLOOM Architecture by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2083
* [ds-inference] checkpoint loading => tqdm by stas00 in https://github.com/microsoft/DeepSpeed/pull/2107
* Dont overwrite hook handles in flop profiler by Sanger2000 in https://github.com/microsoft/DeepSpeed/pull/2106
* Support HuggingFace NeoX injection policy by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2087


**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.6.6...v0.6.7

What's Changed
* [docs] add 530b paper by jeffra in https://github.com/microsoft/DeepSpeed/pull/1979
* small fix for the HF Bert models by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/1984
* Add unit test for various model families and inference tasks by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1981
* Fix for lightning tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1988
* fix typo when getting kernel dim in conv calculation by cli99 in https://github.com/microsoft/DeepSpeed/pull/1989
* Add torch-latest and torch-nightly CI workflows by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1990
* [bug] Add user-defined launcher args for MPI launcher by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/1933
* Propagate max errorcode to deepspeed when using PDSH launcher by jerrymannil in https://github.com/microsoft/DeepSpeed/pull/1994
* [docs] add new build badges to landing page by jeffra in https://github.com/microsoft/DeepSpeed/pull/1998
* DeepSpeed Comm. Backend v1 by awan-10 in https://github.com/microsoft/DeepSpeed/pull/1985
* Relax DeepSpeed MoE ZeRO-1 Assertion by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2007
* update CODEOWNERS by conglongli in https://github.com/microsoft/DeepSpeed/pull/2017
* [CI] force upgrade HF dependencies & output py env by jeffra in https://github.com/microsoft/DeepSpeed/pull/2015
* [inference] test suite for ds-kernels (bert, roberta, gpt2, gpt-neo, gpt-j) by jeffra in https://github.com/microsoft/DeepSpeed/pull/1992
* DeepSpeed examples refresh by jeffra in https://github.com/microsoft/DeepSpeed/pull/2021
* Fix transformer API for training-evaluation pipeline by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2018
* DataLoader Length Fix by Sanger2000 in https://github.com/microsoft/DeepSpeed/pull/1718
* DeepSpeed Monitor Module (Master) by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2013
* Use partition numel by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2011
* fix import errors by KMFODA in https://github.com/microsoft/DeepSpeed/pull/2026
* Fix inference unit test import error catching by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2024
* Retain available params until last use by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2016
* Split parameter offload from z3 by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2009
* Fix flops profiler print statements by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2038
* Add compression papers by conglongli in https://github.com/microsoft/DeepSpeed/pull/2042
* Fix the half-precision version of CPU-Adam by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2032
* Fix for AMD unit tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2047
* Wrong partition_id while copying fp32_params -> fp16 params in Z2 for MoE by siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2058
* Fix missing import in replace_module.py by aphedges in https://github.com/microsoft/DeepSpeed/pull/2050
* Comms Benchmarks by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2040
* add ds inference paper by jeffra in https://github.com/microsoft/DeepSpeed/pull/2072
* Comments for better understanding of zero stage1_2 by kisseternity in https://github.com/microsoft/DeepSpeed/pull/2027
* [docs] fix broken read-the-docs build by jeffra in https://github.com/microsoft/DeepSpeed/pull/2075
* Fix building package without a GPU by aphedges in https://github.com/microsoft/DeepSpeed/pull/2049
* Fix partition id in the fp32->fp16 param copying step for z2+cpu-offload by siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2059
* Codeowner addendum and fix to small model debugging script by samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2076
* remove require grad in params count by cli99 in https://github.com/microsoft/DeepSpeed/pull/2065
* Add missing newline for ZeroOneAdam parameter table by manuelciosici in https://github.com/microsoft/DeepSpeed/pull/2088
* fixed "None type has no len()" by xiazeyu in https://github.com/microsoft/DeepSpeed/pull/2091
* Improving memory utilization of Z2+MoE by siddharth9820 in https://github.com/microsoft/DeepSpeed/pull/2079

New Contributors
* jerrymannil made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1994
* Sanger2000 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/1718
* KMFODA made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2026
* siddharth9820 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2058
* samadejacobs made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2076
* xiazeyu made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2091

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.6.5...v0.6.6