Nni

Major Features

New Features and improvement
* support `__version__` for SDK version
* support windows dev install
* Improve IPC limitation to 100W
* improve code storage upload logic among trials in non-local platform

HPO Updates
* Improve PBT on failure handling and support experiment resume for PBT

NAS Updates
* NAS support for TensorFlow 2.0 (preview) [TF2.0 NAS examples](https://github.com/microsoft/nni/tree/master/examples/nas/naive-tf)
* Use OrderedDict for LayerChoice
* Prettify the format of export
* Replace layer choice with selected module after applied fixed architecture

Model Compression Updates
* Model compression PyTorch 1.4 support

Training Service Updates
* update pai yaml merge logic
* support windows as remote machine in remote mode [Remote Mode](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/RemoteMachineMode.md#windows)

Web UI new supports or improvements
* Show trial error message
* finalize homepage layout
* Refactor overview's best trials module
* Remove multiphase from webui
* add tooltip for trial concurrency in the overview page
* Show top trials for hyper-parameter graph

Bug Fix
* fix dev install
* SPOS example crash when the checkpoints do not have state_dict
* Fix table sort issue when experiment had failed trial
* Support multi python env (conda, pyenv etc)

New Features and Documentation

Hyper-Parameter Optimizing

* New tuner: [Population Based Training (PBT)](https://github.com/microsoft/nni/blob/master/docs/en_US/Tuner/PBTTuner.md)
* Trials can now report infinity and NaN as result

Neural Architecture Search

* New NAS algorithm: [TextNAS](https://github.com/microsoft/nni/blob/master/docs/en_US/NAS/TextNAS.md)
* ENAS and DARTS now support [visualization](https://github.com/microsoft/nni/blob/master/docs/en_US/NAS/Visualization.md) through web UI.

Model Compression

* New Pruner: [GradientRankFilterPruner](https://github.com/microsoft/nni/blob/master/docs/en_US/Compressor/Pruner.md#gradientrankfilterpruner)
* Compressors will validate configuration by default
* Refactor: Adding optimizer as an input argument of pruner, for easy support of DataParallel and more efficient iterative pruning. This is a broken change for the usage of iterative pruning algorithms.
* Model compression examples are refactored and improved
* Added documentation for [implementing compressing algorithm](https://github.com/microsoft/nni/blob/master/docs/en_US/Compressor/Framework.md)

Training Service

* Kubeflow now supports pytorchjob crd v1 (thanks external contributor jiapinai)
* Experimental [DLTS](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/DLTSMode.md) support

Overall Documentation Improvement

* Documentation is significantly improved on grammar, spelling, and wording (thanks external contributor AHartNtkn)

Fixed Bugs

* ENAS cannot have more than one LSTM layers (thanks external contributor marsggbo)
* NNI manager's timers will never unsubscribe (thanks external contributor guilhermehn)
* NNI manager may exhaust head memory (thanks external contributor Sundrops)
* Batch tuner does not support customized trials (2075)
* Experiment cannot be killed if it failed on start (2080)
* Non-number type metrics break web UI (2278)
* A bug in lottery ticket pruner
* Other minor glitches

Major Features

Neural Architecture Search
* Support [C-DARTS](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/CDARTS.md) algorithm and add [the example](https://github.com/microsoft/nni/tree/v1.4/examples/nas/cdarts) using it
* Support a preliminary version of [ProxylessNAS](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/Proxylessnas.md) and the corresponding [example](https://github.com/microsoft/nni/tree/v1.4/examples/nas/proxylessnas)
* Add unit tests for the NAS framework

Model Compression
* Support DataParallel for compressing models, and provide [an example](https://github.com/microsoft/nni/blob/v1.4/examples/model_compress/multi_gpu.py) of using DataParallel
* Support [model speedup](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Compressor/ModelSpeedup.md) for compressed models, in Alpha version

Training Service
* Support complete PAI configurations by allowing users to specify PAI config file path
* Add example config yaml files for the new PAI mode (i.e., paiK8S)
* Support deleting experiments using sshkey in remote mode (thanks external contributor tyusr)

WebUI
* WebUI refactor: adopt fabric framework

Others
* Support running [NNI experiment at foreground](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Tutorial/Nnictl.md#manage-an-experiment), i.e., `--foreground` argument in `nnictl create/resume/view`
* Support canceling the trials in UNKNOWN state
* Support large search space whose size could be up to 50mb (thanks external contributor Sundrops)

Documentation
* Improve [the index structure](https://nni.readthedocs.io/en/latest/) of NNI readthedocs
* Improve [documentation for NAS](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/NasGuide.md)
* Improve documentation for [the new PAI mode](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/TrainingService/PaiMode.md)
* Add QuickStart guidance for [NAS](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/QuickStart.md) and [model compression](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Compressor/QuickStart.md)
* Improve documentation for [the supported EfficientNet](https://github.com/microsoft/nni/blob/v1.4/docs/en_US/TrialExample/EfficientNet.md)

Bug Fixes
* Correctly support NaN in metric data, JSON compliant
* Fix the out-of-range bug of `randint` type in search space
* Fix the bug of wrong tensor device when exporting onnx model in model compression
* Fix incorrect handling of nnimanagerIP in the new PAI mode (i.e., paiK8S)

Major Features

Neural Architecture Search Algorithms Support
* [Single Path One Shot](https://github.com/microsoft/nni/tree/v1.3/examples/nas/spos/) algorithm and the example using it

Model Compression Algorithms Support
* Example: [Knowledge Distillation](https://github.com/microsoft/nni/blob/v1.3/docs/en_US/TrialExample/KDExample.md) algorithm and the example using it
* Pruners
* [L2Filter Pruner](https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.md#3-l2filter-pruner)
* [ActivationAPoZRankFilterPruner](https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.md#1-activationapozrankfilterpruner)
* [ActivationMeanRankFilterPruner](https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.md#2-activationmeanrankfilterpruner)
* [BNN Quantizer](https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Quantizer.md#bnn-quantizer)
Training Service
* NFS Support for PAI

Instead of using HDFS as default storage, since OpenPAI v0.11, OpenPAI can have NFS or AzureBlob or other storage as default storage. In this release, NNI extended the support for this recent change made by OpenPAI, and could integrate with OpenPAI v0.11 or later version with various default storage.

* Kubeflow update adoption
Add support for zero gpuNum in kubernetes (1830 | thanks to external contributor skyser2003)
Adopted the Kubeflow 0.7's new supports for tf-operator. (thanks to external contributor skyser2003)

Engineering (code and build automation)
* Enforced [ESLint](https://eslint.org/) on static code analysis.

Small changes & Bug Fixes
* correctly recognize builtin tuner and customized tuner
* logging in dispatcher base
* fix the bug where tuner/assessor's failure sometimes kills the experiment.
* Fix local system as remote machine [issue](https://github.com/microsoft/nni/issues/1852)
* de-duplicate trial configuration in smac tuner [ticket](https://github.com/microsoft/nni/issues/1364)

Major Features
* [Feature Engineering](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/Overview.md)
- New feature engineering interface
- Feature selection algorithms: [Gradient feature selector](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/GradientFeatureSelector.md) & [GBDT selector](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/GBDTSelector.md)
- [Examples for feature engineering](https://github.com/microsoft/nni/tree/v1.2/examples/feature_engineering)
* Neural Architecture Search (NAS) on NNI
- [New NAS interface](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/NasInterface.md)
- NAS algorithms: [ENAS](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.md#enas), [DARTS](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.md#darts), [P-DARTS](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.md#p-darts) (in PyTorch)
- NAS in classic mode (each trial runs independently)
* Model compression
- [New model pruning algorithms](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/Compressor/Overview.md): lottery ticket pruning approach, L1Filter pruner, Slim pruner, FPGM pruner
- [New model quantization algorithms](https://github.com/microsoft/nni/blob/v1.2/docs/en_US/Compressor/Overview.md): QAT quantizer, DoReFa quantizer
- Support the API for exporting compressed model.
* Training Service
- Support OpenPAI token authentication
* Examples:
- [An example to automatically tune rocksdb configuration with NNI](https://github.com/microsoft/nni/tree/v1.2/examples/trials/systems/rocksdb-fillrandom).
- [A new MNIST trial example supports tensorflow 2.0](https://github.com/microsoft/nni/tree/v1.2/examples/trials/mnist-tfv2).
* Engineering Improvements
- For remote training service, trial jobs require no GPU are now scheduled with round-robin policy instead of random.
- Pylint rules added to check pull requests, new pull requests need to comply with these [pylint rules](https://github.com/microsoft/nni/blob/v1.2/pylintrc).
* Web Portal & User Experience
- Support user to add customized trial.
- User can zoom out/in in detail graphs, except Hyper-parameter.
* Documentation
- Improved NNI API documentation with more API docstring.

Bug fix
- Fix the table sort issue when failed trials haven't metrics. -Issue 1764
- Maintain selected status(Maximal/Minimal) when the page switched. -PR 1710
- Make hyper-parameters graph's default metric yAxis more accurate. -PR 1736
- Fix GPU script permission issue. -Issue 1665

Major Features

* New tuner: [PPO Tuner](https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tuner/PPOTuner.md)
* [View stopped experiments](https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tutorial/Nnictl.md#view)
* Tuners can now use dedicated GPU resource (see `gpuIndices` in [tutorial](https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tutorial/ExperimentConfig.md) for details)
* Web UI improvements
- Trials detail page can now list hyperparameters of each trial, as well as their start and end time (via "add column")
- Viewing huge experiment is now less laggy
* More examples
- [EfficientNet PyTorch example](https://github.com/ultmaster/EfficientNet-PyTorch)
- [Cifar10 NAS example](https://github.com/microsoft/nni/blob/v1.1/examples/trials/nas_cifar10/README.md)
* [Model compression toolkit - Alpha release](https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Compressor/Overview.md): We are glad to announce the alpha release for model compression toolkit on top of NNI, it's still in the experiment phase which might evolve based on usage feedback. We'd like to invite you to use, feedback and even contribute

Fixed Bugs

* Multiphase job hangs when search space exhuasted (issue 1204)
* `nnictl` fails when log not available (issue 1548)