Horovod

Fixed

- Make DBFSLocalStore support "file:/dbfs/...", implement get_localized_path. ([3510](https://github.com/horovod/horovod/pull/3510))

Fixed

- Setup: Require fsspec >= 2010.07.0 ([3451](https://github.com/horovod/horovod/pull/3451))
- Fix ignored cuda arch flags ([3462]((https://github.com/horovod/horovod/pull/3462))

Fixed

- Extended CMake build script to often find CUDA even if `nvcc` is not in `$PATH`. ([3444](https://github.com/horovod/horovod/pull/3444))

Added

- Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. ([3190](https://github.com/horovod/horovod/issues/3190))
- TensorFlow: Added in-place broadcasting of variables. ([3128](https://github.com/horovod/horovod/pull/3128))
- Elastic: Added support for resurrecting blacklisted hosts. ([3319](https://github.com/horovod/horovod/pull/3319))
- MXNet: Added support for MXNet async dependency engine. ([3242](https://github.com/horovod/horovod/pull/3242), [#2963](https://github.com/horovod/horovod/pull/2963))
- Spark/Lightning: Added history to lightning estimator. ([3214](https://github.com/horovod/horovod/pull/3214))

Changed

- Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. ([3261](https://github.com/horovod/horovod/pull/3261), [#3371](https://github.com/horovod/horovod/pull/3371))
- Moved released Docker image `horovod` and `horovod-cpu` to Ubuntu 20.04 and Python 3.8. ([3393](https://github.com/horovod/horovod/pull/3393))
- Spark Estimator: Don't shuffle row groups if training data requires non-shuffle ([3369](https://github.com/horovod/horovod/pull/3369))
- Spark/Lightning: Reduced memory footprint of async dataloader. ([3239](https://github.com/horovod/horovod/pull/3239))
- Elastic: Improved handling NCCL errors under elastic scenario. ([3112](https://github.com/horovod/horovod/pull/3112))
- Spark/Lightning: Do not overwrite model with checkpoint by default. ([3201](https://github.com/horovod/horovod/pull/3201))
- Make checkpoint name optional so that user can save to h5 format. ([3411](https://github.com/horovod/horovod/pull/3411))

Deprecated

- Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. ([3190](https://github.com/horovod/horovod/issues/3190))

Removed

- Spark: Removed `h5py<3` constraint as this is not needed anymore for Tensorflow >2.5.0. ([3301](https://github.com/horovod/horovod/pull/3301))

Fixed

- Elastic Spark: Fixed indices in initial task-to-task registration. ([3410](https://github.com/horovod/horovod/pull/3410))
- PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. ([3352](https://github.com/horovod/horovod/issues/3352))
- PyTorch: Fixed finalization of ProcessSetTable. ([3351](https://github.com/horovod/horovod/pull/3351))
- Fixed remote trainers to point to the correct shared lib path. ([3258](https://github.com/horovod/horovod/pull/3258))
- Fixed imports from `tensorflow.python.keras` with tensorflow 2.6.0+. ([3403](https://github.com/horovod/horovod/pull/3403))
- Fixed Adasum communicator init logic. ([3379](https://github.com/horovod/horovod/pull/3379))
- Lightning: Fixed resume logger. ([3375](https://github.com/horovod/horovod/pull/3375))
- Fixed the checkpoint directory structure for pytorch and pytorch lightning. ([3362](https://github.com/horovod/horovod/pull/3362))
- Fixed possible integer overflow in multiplication. ([3368](https://github.com/horovod/horovod/pull/3368))
- Fixed the `pytorch_lightning_mnist.py` example. ([3245](https://github.com/horovod/horovod/pull/3245), [#3290](https://github.com/horovod/horovod/pull/3290))
- Fixed barrier segmentation fault. ([3313](https://github.com/horovod/horovod/pull/3313))
- Fixed `hvd.barrier()` tensor queue management. ([3300](https://github.com/horovod/horovod/pull/3300))
- Fixed PyArrow "list index out of range" IndexError. ([3274](https://github.com/horovod/horovod/pull/3274))
- Elastic: Fixed all workers sometimes failing on elastic Horovod failure. ([3264](https://github.com/horovod/horovod/issues/3264))
- Spark/Lightning: Fixed setting `limit_train_batches` and `limit_val_batches`. ([3237](https://github.com/horovod/horovod/pull/3237))
- Elastic: Fixed ElasticSampler and `hvd.elastic.state` losing some indices of processed samples when nodes dropped. ([3143](https://github.com/horovod/horovod/issues/3143))
- Spark/Lightning: Fixed history metrics for estimator serialization. ([3216](https://github.com/horovod/horovod/pull/3216))
- Ray: Fixed RayExecutor to fail when `num_workers=0` and `num_hosts=None`. ([3210](https://github.com/horovod/horovod/pull/3210))
- Spark/Lightning: Fixed checkpoint callback `dirpath` typo. ([3204](https://github.com/horovod/horovod/pull/3204))

Added

- Added process sets to concurrently run collective operations on subsets of Horovod processes in TensorFlow, PyTorch, and MXNet. ([2839](https://github.com/horovod/horovod/pull/2839), [#3042](https://github.com/horovod/horovod/pull/3042), [#3043](https://github.com/horovod/horovod/pull/3043), [#3054](https://github.com/horovod/horovod/pull/3054), [#3083](https://github.com/horovod/horovod/pull/3083), [#3090](https://github.com/horovod/horovod/pull/3090))
- Added XLA support for Allreduce via `tf.function(jit_compile=True)`. ([3053](https://github.com/horovod/horovod/pull/3053))
- Added fused buffer scaling and unpack/pack kernels on GPU. ([2973](https://github.com/horovod/horovod/pull/2973))
- Added support for NCCL on CUDA 11.4. ([3182](https://github.com/horovod/horovod/issues/3182))
- Added fp16 compression for MXNet. ([2987](https://github.com/horovod/horovod/issues/2987))
- Added terminate_on_nan flag to Spark Lightning estimator. ([3088](https://github.com/horovod/horovod/issues/3088))
- Added barrier() API to torch module to support simple synchronization among ranks and to achieve parity with PyTorch DDP and similar frameworks. [3139](https://github.com/horovod/horovod/pull/3139)
- Added params for customizing Tensorboard callback. ([3153](https://github.com/horovod/horovod/issues/3153))
- Added `hvd.cross_rank()` for keras. ([3008](https://github.com/horovod/horovod/issues/3008))
- Added barrier() API to torch module to support simple synchronization among ranks and to achieve parity with PyTorch DDP and similar frameworks. [3139](https://github.com/horovod/horovod/pull/3139)

Changed

- Implemented more asynchronous dependency handling on GPU. ([2963](https://github.com/horovod/horovod/pull/2963))
- Ray: RayExecutor will now use the current placement group instead of always creating a new one. ([3134](https://github.com/horovod/horovod/pull/3134))
- Lightning: turned off shuffling for validation dataset. ([2974](https://github.com/horovod/horovod/pull/2974))
- Ray: RayExecutor will use the current placement group if one exists. ([3134](https://github.com/horovod/horovod/pull/3134))
- Extended `hvd.join()` to return the last rank that joined. ([3097](https://github.com/horovod/horovod/pull/3097)

Deprecated

Removed

- Spark/Keras: remove bare Keras support. ([3191](https://github.com/horovod/horovod/pull/3191))

Fixed

- Fix Horovod develop/editable install mode and incremental builds. ([3074](https://github.com/horovod/horovod/pull/3074))
- Estimator/Lightning: use lightning datamodule. ([3084](https://github.com/horovod/horovod/pull/3084))
- Fix Horovod Spark StringType and numpy type mapping issue. ([3146](https://github.com/horovod/horovod/pull/3146))
- Fixed error in Keras LearningRateScheduler. ([3135](https://github.com/horovod/horovod/pull/3135))
- Fixed bug in Lightning Profiler on Ray. ([3122](https://github.com/horovod/horovod/pull/3122))
- Fixed torch op lazy release to prevent OOM in elastic training. ([3110](https://github.com/horovod/horovod/pull/3110))
- Lightning: Fixed usage of the checkpoint callback. ([3186](https://github.com/horovod/horovod/pull/3186))
- Fixed MPICH support to use Intel MPI's implementation. ([3148](https://github.com/horovod/horovod/pull/3148))
- Fixed race condition in PyTorch async dataloader. ([3120](https://github.com/horovod/horovod/pull/3120))
- Keras: Fixed learning rate scheduler. ([3142](https://github.com/horovod/horovod/pull/3142), [#3135](https://github.com/horovod/horovod/pull/3135))

Added

- Estimator: added support for loading data from S3, GCS, ADLS, and other remote filesystems. ([2927](https://github.com/horovod/horovod/issues/2927))
- Estimator: added custom Spark data loader interface. ([2938](https://github.com/horovod/horovod/issues/2923))
- LightningEstimator: added support to supply a logger and associated parameter to control the frequency of logging. ([2926](https://github.com/horovod/horovod/pull/2926))
- Estimator: added check to ensure all ranks have the same device type. ([2942](https://github.com/horovod/horovod/pull/2942))

Changed

- Changed behavior from using TensorBoardLogger to now using it as a fallback if a logger is not supplied. ([2926](https://github.com/horovod/horovod/pull/2926))
- Ray: disabled capturing child tasks in placement group. ([2920](https://github.com/horovod/horovod/pull/2920))

Fixed

- Fixed `hvd.tensorflow.keras.Compression`, accidentally removed in v0.22.0. ([2945](https://github.com/horovod/horovod/pull/2945))
- TorchEstimator: fixed usage of `validation_steps` in place of `validation_steps_per_epoch`. ([2918](https://github.com/horovod/horovod/pull/2918))
- TensorFlow: fixed C++ API for TF v2.6.0. ([2932](https://github.com/horovod/horovod/pull/2932))
- PyTorch: fixed `sparse_allreduce_async` for PyTorch v0.10.0. ([2965](https://github.com/horovod/horovod/pull/2965))