Deepspeed

What's Changed
* CUDA optional deepspeed ops by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2507
* Remove CI trigger for push to master by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2712
* [install] only add deepspeed pkg at install by jeffra in https://github.com/microsoft/DeepSpeed/pull/2714
* Fix nightly tests for new lm-eval release by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2713
* BF16 optimizer for BF16+ZeRO Stage 1 by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2706
* Fix typo in diffusers transformer block by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2718
* Inference Refactor (replace_with_policy, model_implementations) by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2554
* Change zero_grad() argument to match pytorch by loadams in https://github.com/microsoft/DeepSpeed/pull/2741
* Automatic tensor parallelism v2 by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2670
* Fixing Optimizer Sanity Check by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2742
* [GatheredParameters] fix memory leak by stas00 in https://github.com/microsoft/DeepSpeed/pull/2665
* Abstract accelerator (step 3) by delock in https://github.com/microsoft/DeepSpeed/pull/2677
* Fix autotuning so that it records Floating Point Operations per second, not microsecond by dashstander in https://github.com/microsoft/DeepSpeed/pull/2711
* fix a misspelled attribute by stas00 in https://github.com/microsoft/DeepSpeed/pull/2750
* [zero] remove misleading dtype log by jeffra in https://github.com/microsoft/DeepSpeed/pull/2732
* Fix softmax backward by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2709
* Skip test_bias_gelu unit test if torch < 1.12 by lekurile in https://github.com/microsoft/DeepSpeed/pull/2754
* Conditionally Make Op Building More Verbose by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2759
* Bing/formatting correction by xiexbing in https://github.com/microsoft/DeepSpeed/pull/2764
* Add links to new azureML examples by cassieesvelt in https://github.com/microsoft/DeepSpeed/pull/2756
* Fix hardcoded instances to fp16 in optimizer creation log messages to the correct dtype. by loadams in https://github.com/microsoft/DeepSpeed/pull/2743
* Refactor/Pydantify monitoring config by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2640
* Pin minimum `packaging` requirement by carmocca in https://github.com/microsoft/DeepSpeed/pull/2771
* Fix for diffusers v0.12.0 by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2753
* some fix in flops_profiler by lucasleesw in https://github.com/microsoft/DeepSpeed/pull/2068
* fix upsample flops compute by skipping unused kargs by cli99 in https://github.com/microsoft/DeepSpeed/pull/2773
* Fix broken kernel inject bug by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2776
* Fix Checkpoint-loading with Meta-tensor by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2781
* Add hjson support for user configs by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2783
* Reset KV-cache at the beginning of text-generation by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2669
* Container param cleanup + remove qkv_merging by lekurile in https://github.com/microsoft/DeepSpeed/pull/2780
* Common location to install libaio-dev by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2779
* Fixing broken link to azureml-examples recipes by rtanase in https://github.com/microsoft/DeepSpeed/pull/2795
* remove outdated comment by stas00 in https://github.com/microsoft/DeepSpeed/pull/2786
* Enable page-locked tensors without CUDA by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2775
* Add container load checkpoint error reporting + refactor by lekurile in https://github.com/microsoft/DeepSpeed/pull/2792
* Add user defined launcher args for PDSH launcher by loadams in https://github.com/microsoft/DeepSpeed/pull/2804
* Fix Slurm launcher user args by loadams in https://github.com/microsoft/DeepSpeed/pull/2806
* Handle hanged tests in CI by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2808
* Fix inference CI device error by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2824
* Fix permissions issue with pip upgrade by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2823
* Fix cpu-only CI hangs by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2825
* Fix Pipeline Parallel resize unit test by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2833
* Fix auto TP for duplicate modules with different gems by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2784
* Refactor DS inference API. No longer need replace_method. by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2831
* Port Reza's INT8-quantization fix to container architecture by lekurile in https://github.com/microsoft/DeepSpeed/pull/2725
* Fix gpt-Neox rotary embedding implementation by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2782
* Fix for CI failure on system upgrade by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2849

New Contributors
* loadams made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2741
* xiexbing made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2764
* carmocca made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2771
* lucasleesw made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2068
* rtanase made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2795

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.8.0...v0.8.1

New features
* [DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality](https://www.deepspeed.ai/2022/12/11/data-efficiency.html)
* DeepSpeed Data Efficiency Library by conglongli in https://github.com/microsoft/DeepSpeed/pull/2585

What's Changed
* fix blog link by conglongli in https://github.com/microsoft/DeepSpeed/pull/2600
* Migrate ops tests to new inference_ops marker by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2599
* Move layer norm to new schedule by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2590
* [deepspeed/autotuner] Bug fix for binary search for batch size by rahilbathwal5 in https://github.com/microsoft/DeepSpeed/pull/2162
* Fix for older versions of pydantic by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2611
* Use rocm/pytorch:latest for ROCm Dockerfile by jithunnair-amd in https://github.com/microsoft/DeepSpeed/pull/2613
* skip torch.zeros and tensor.copy_ when model parallel is not used by guoyejun in https://github.com/microsoft/DeepSpeed/pull/2479
* call empty_cache to really free up GPU memory as described in comment by guoyejun in https://github.com/microsoft/DeepSpeed/pull/2620
* Remove GatheredParameters context from replace_with_policy by lekurile in https://github.com/microsoft/DeepSpeed/pull/2591
* fixes 2498 by clumsy in https://github.com/microsoft/DeepSpeed/pull/2603
* Update AVX512 Detection by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2621
* Add Megatron CI workflow by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2614
* [inference] check for unsupported model generate args by jeffra in https://github.com/microsoft/DeepSpeed/pull/2627
* [launcher] parse hostfile via regex and added error checks by jeffra in https://github.com/microsoft/DeepSpeed/pull/2626
* Unit tests setup own venv by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2628
* Fix 2409: add enable_each_rank_log to deepspeed/launcher/runner.py by inkcherry in https://github.com/microsoft/DeepSpeed/pull/2571
* Fix typo in autotuner.py by eltociear in https://github.com/microsoft/DeepSpeed/pull/2639
* [zero-3] Handle forward parameter return correctly in nested cases by samyam in https://github.com/microsoft/DeepSpeed/pull/2642
* [inference] ds-attention refactor w.r.t. ops by jeffra in https://github.com/microsoft/DeepSpeed/pull/2623
* Fix issue w. bloom int8 when changing tp size by jeffra in https://github.com/microsoft/DeepSpeed/pull/2645
* fix assertion error in zero stage 3 by GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2647
* tweaks to ds-attn, distilbert policy, and mup by jeffra in https://github.com/microsoft/DeepSpeed/pull/2649
* [doc] fix `min_loss_scale` default by stas00 in https://github.com/microsoft/DeepSpeed/pull/2660
* [launcher] fail gracefully if hostname -i doesn't work as expected by jeffra in https://github.com/microsoft/DeepSpeed/pull/2631
* Fix Opt injection by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2541
* Abstract accelerator (step 2) by delock in https://github.com/microsoft/DeepSpeed/pull/2560
* Remove unnecessary device synchronization for stage 2 by li-yi-dong in https://github.com/microsoft/DeepSpeed/pull/2500
* [Bug Fixed] torch.cuda.is_available -> torch.cuda.is_available() by wkcn in https://github.com/microsoft/DeepSpeed/pull/2661
* [fp16] lower `initial_scale_power` to `16` by stas00 in https://github.com/microsoft/DeepSpeed/pull/2663
* fix Tensor contiguous bug in model_compression by xiaoxiawu-microsoft in https://github.com/microsoft/DeepSpeed/pull/2671
* [inference] ds-mlp refactor w.r.t. ops by jeffra in https://github.com/microsoft/DeepSpeed/pull/2668
* real_accelerator validation check for both accelerator and deepspeed accelerator path by delock in https://github.com/microsoft/DeepSpeed/pull/2685
* fix typo and remove duplicated code in ZeRO stage 1 and 2 by wkcn in https://github.com/microsoft/DeepSpeed/pull/2655
* Add mlflow logging for aml by cassieesvelt in https://github.com/microsoft/DeepSpeed/pull/2495
* Fix import error of op_builder by tohtana in https://github.com/microsoft/DeepSpeed/pull/2687
* Pass training flag to forward call from module config by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2604
* Extend quantization utils features by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2683
* [GatheredParameters] add support for any iterable by stas00 in https://github.com/microsoft/DeepSpeed/pull/2664
* Fix for latest diffusers by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2699
* exclude benchmarks during install by jeffra in https://github.com/microsoft/DeepSpeed/pull/2698
* Correct loss scale in ZeRO step by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2695
* [ZeRO] non-MoE stage 1 requires CG disabled by jeffra in https://github.com/microsoft/DeepSpeed/pull/2703
* remove print side effect from importing deepspeed by jeffra in https://github.com/microsoft/DeepSpeed/pull/2704
* ZeRO3 handling frozen weights by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2653

New Contributors
* eltociear made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2639
* li-yi-dong made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2500
* wkcn made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2661
* xiaoxiawu-microsoft made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2671
* cassieesvelt made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2495
* tohtana made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2687

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.7...v0.8.0

What's Changed
* Update the locator for Megatron-LM by rapsealk in https://github.com/microsoft/DeepSpeed/pull/2564
* use get_global_rank if available by jeffra in https://github.com/microsoft/DeepSpeed/pull/2567
* Add Determined to open-source DL frameworks by sirredbeard in https://github.com/microsoft/DeepSpeed/pull/2573
* Support fp32 gradaccum for bf16 model by delock in https://github.com/microsoft/DeepSpeed/pull/2566
* Drop Maxwell Support by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2574
* Fix quantized-inference & Add generic support of checkpoint loading by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2547
* Fix MegatronLayerPolicy to have megatron_v2=True by lekurile in https://github.com/microsoft/DeepSpeed/pull/2579
* Update barrier and reduce_scatter_base to conform to PyTorch signatures by Quentin-Anthony in https://github.com/microsoft/DeepSpeed/pull/2570
* Support N-dimension input in quantization kernel by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2575
* Add checkpoint sharding unit tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2561
* Updating docs README by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2587
* Updating API docs by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2586
* Fix issues w. python 3.6 + add py-version checks to CI by jeffra in https://github.com/microsoft/DeepSpeed/pull/2589
* [benchmarks] get mask token from tokenizer by jeffra in https://github.com/microsoft/DeepSpeed/pull/2592

New Contributors
* rapsealk made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2564
* sirredbeard made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2573

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.6...v0.7.7

What's Changed
* DeepSpeed inference config. (2459) by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2472
* Update docs to autogenerate pydantic config model docs by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2509
* Add max_tokens alias to max_out_tokens arg to maintain backwards compatibility by lekurile in https://github.com/microsoft/DeepSpeed/pull/2508
* Deepspeed quantization library v0.1 by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2450
* Fix backward compatibility for InferenceConfig by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2516
* Add missing Inference sub-configs by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2518
* Add note about nvcc/hipcc requirement by jeffra in https://github.com/microsoft/DeepSpeed/pull/2519
* Update codeowners by jeffra in https://github.com/microsoft/DeepSpeed/pull/2525
* Dequantization Utils Library by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2521
* Fixes for torch 1.14 due to new torch.numel return type by jeffra in https://github.com/microsoft/DeepSpeed/pull/2522
* Ensure MOE is initialized for SD by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2534
* Make DS-Inference config readable from JSON by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2537
* Add MII tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2533
* Remove mutable default parameter in `init_inference()` by aphedges in https://github.com/microsoft/DeepSpeed/pull/2540
* Change Where DS/Triton is Used in Stable Diffusion by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2536
* Pass down the new DS inference config to replace_transformer_layer. by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2539
* Adding Gradient Accumulation Data Type Config by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2512
* Report progress at gradient accumulation boundary by ShijieZZZZ in https://github.com/microsoft/DeepSpeed/pull/2553
* encoded ds config into command line argument when launching child processes in autotuning by cli99 in https://github.com/microsoft/DeepSpeed/pull/2524
* Add missing MoE fields to inference config for backward compatibility by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2556
* Abstract accelerator (step 1) by delock in https://github.com/microsoft/DeepSpeed/pull/2504
* Fix invalid check of recorded parameter orders in zero stage3. by inkcherry in https://github.com/microsoft/DeepSpeed/pull/2550

New Contributors
* ShijieZZZZ made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2553
* delock made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2504
* inkcherry made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2550

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.5...v0.7.6

What's Changed
* Fix Bug 2319 by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2438
* update pytorch pool operator function signiture by cli99 in https://github.com/microsoft/DeepSpeed/pull/2443
* Fix build issues on Windows by eltonzheng in https://github.com/microsoft/DeepSpeed/pull/2428
* rollback ds config changes by cli99 in https://github.com/microsoft/DeepSpeed/pull/2395
* Use CUDA events for inference model profiling by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2371
* Fixing a config mismatch in unit test. by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2447
* Reduction Kernel Utility by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2436
* deepspeed/launcher/launch.py: add option enable_each_rank_log by guoyejun in https://github.com/microsoft/DeepSpeed/pull/2409
* Fixes for various CI problems by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2457
* Cache Allocation and Softmax Fixes by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2433
* Fix checkpoint loading at inference-engine by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2429
* Create a new folder structure to isolate model-specific code in DS by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2464
* don't gather partitioned activations for mp size 1 by guoyejun in https://github.com/microsoft/DeepSpeed/pull/2454
* Updating autotune json default in docs. by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2476
* Added MLFLOW environment variables for logging metrics within trainig… by savitamittal1 in https://github.com/microsoft/DeepSpeed/pull/2477
* fix accelerate link in README by kyoto7250 in https://github.com/microsoft/DeepSpeed/pull/2481
* Fix Stable-Diffusion: Add correct memory-allocation at DeepSpeed-Attention by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2474
* Fix CI issues related to cupy install by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2483
* Add `scale_attn_by_inverse_layer_idx` feature by hyunwoongko in https://github.com/microsoft/DeepSpeed/pull/2486
* Stable Diffusion Enhancements by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2491
* stage_1_and_2.py: no allreduce needed when mp size is 1 by guoyejun in https://github.com/microsoft/DeepSpeed/pull/2494
* Make bf16_optimizer work for non pipeline parallelism by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2470
* Fix nightly CI tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2493
* Make data contiguous before the inplace reshape-copy_ function. by lokoppakmsft in https://github.com/microsoft/DeepSpeed/pull/2489
* Fix typos: deepseed -> deepspeed by jinyouzhi in https://github.com/microsoft/DeepSpeed/pull/2499

New Contributors
* guoyejun made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2409
* savitamittal1 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2477
* kyoto7250 made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2481
* lokoppakmsft made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2489
* jinyouzhi made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2499

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.4...v0.7.5

What's Changed
* MOE residual matmult unit test by samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2323
* MOE matmult with memaccess by samadejacobs in https://github.com/microsoft/DeepSpeed/pull/2336
* Refactor residual add kernels by arashb in https://github.com/microsoft/DeepSpeed/pull/2333
* mem access for quantize kernel by GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2331
* increase min pre-commit versions by jeffra in https://github.com/microsoft/DeepSpeed/pull/2346
* Extend scratch buffer for long prompts by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2212
* [docs] fix zero docs by jeffra in https://github.com/microsoft/DeepSpeed/pull/2350
* Staging profile inference v1 (2348) by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2349
* Kernel Data Conversion Utility by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2327
* Add Onebit Optimizers in __init__ by l4d2boomer in https://github.com/microsoft/DeepSpeed/pull/2340
* docs(mixture-of-experts-inference): fix typo in tuto by jqueguiner in https://github.com/microsoft/DeepSpeed/pull/2345
* Use blob storage for datasets in unit tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2342
* Refactor `gptj_residual_add` kernels for better readability by arashb in https://github.com/microsoft/DeepSpeed/pull/2358
* Updated issue templates by jeffra in https://github.com/microsoft/DeepSpeed/pull/2363
* fix cuda invalid config error in dequant kernel by GuanhuaWang in https://github.com/microsoft/DeepSpeed/pull/2362
* Add missing pytest fixture scope by arashb in https://github.com/microsoft/DeepSpeed/pull/2353
* Extend residual_add kernel tests to cover pre_attn_norm by arashb in https://github.com/microsoft/DeepSpeed/pull/2354
* Refactor `fused_bias_residual` kernels for better readability by arashb in https://github.com/microsoft/DeepSpeed/pull/2356
* Capture error message during sweep tests by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2351
* Fix an exception when auto-casting dicts to fp16 by mjksmith in https://github.com/microsoft/DeepSpeed/pull/2370
* Refactor remaining distributed tests by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2216
* Fix the MLP output tensor's shape by arashb in https://github.com/microsoft/DeepSpeed/pull/2380
* add 11.8 to cuda_minor_mismatch_ok to allow building with current CUDA by Thomas-MMJ in https://github.com/microsoft/DeepSpeed/pull/2390
* Pin Transformers test version by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2402
* Change type to tuple in replace_wo_policy isinstance check by lekurile in https://github.com/microsoft/DeepSpeed/pull/2387
* Checkpoint backwards-compatbility workaround by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2384
* Add Predicated Global Load to Memory Access Utils by cmikeh2 in https://github.com/microsoft/DeepSpeed/pull/2373
* MII blog post by jeffra in https://github.com/microsoft/DeepSpeed/pull/2418
* Fix figure reference by awan-10 in https://github.com/microsoft/DeepSpeed/pull/2419
* Add SLURM Multinode Runner by dashstander in https://github.com/microsoft/DeepSpeed/pull/2404
* Fix issue with corrupted output on long generation for GPT by andrewchernyh in https://github.com/microsoft/DeepSpeed/pull/2359
* Fix GPT Neo-X multi-gpu inference by andrewchernyh in https://github.com/microsoft/DeepSpeed/pull/2401
* CI fixes related to triton by jeffra in https://github.com/microsoft/DeepSpeed/pull/2422
* [docs] update mii blog title by jeffra in https://github.com/microsoft/DeepSpeed/pull/2423
* add SD injection policy by jeffra in https://github.com/microsoft/DeepSpeed/pull/2381
* Fix checkpoint loading when it is a dictionary by RezaYazdaniAminabadi in https://github.com/microsoft/DeepSpeed/pull/2425
* Make error regex more generic in collect_results.py by molly-smith in https://github.com/microsoft/DeepSpeed/pull/2415
* fixes 2389 by clumsy in https://github.com/microsoft/DeepSpeed/pull/2411
* Fix for inference gpt-j test by mrwyattii in https://github.com/microsoft/DeepSpeed/pull/2430
* Fixing bug 2361 by jomayeri in https://github.com/microsoft/DeepSpeed/pull/2410
* Universal checkpoint for zero stage 1 by tjruwase in https://github.com/microsoft/DeepSpeed/pull/2284
* only add deps if extra is explicitly called by jeffra in https://github.com/microsoft/DeepSpeed/pull/2432
* Add TestInjectionPolicy inference unittest class for testing custom injection policies by lekurile in https://github.com/microsoft/DeepSpeed/pull/2426
* [memory estimators] new config args sync by stas00 in https://github.com/microsoft/DeepSpeed/pull/2431
* parallelize writing of layer checkpoint files across data parallel instances by adammoody in https://github.com/microsoft/DeepSpeed/pull/1419
* Fix broken link to DeepSpeed Megatron fork by lekurile in https://github.com/microsoft/DeepSpeed/pull/2440

New Contributors
* l4d2boomer made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2340
* jqueguiner made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2345
* mjksmith made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2370
* Thomas-MMJ made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2390
* lekurile made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2387
* dashstander made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2404
* andrewchernyh made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2359
* clumsy made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2411
* jomayeri made their first contribution in https://github.com/microsoft/DeepSpeed/pull/2410

**Full Changelog**: https://github.com/microsoft/DeepSpeed/compare/v0.7.3...v0.7.4