Nni

Major Features

* Tuners and Assessors
- Support Auto-Feature generator & selection -Issue877 -PR 1387
+ Provide auto feature interface
+ Tuner based on beam search
+ [Add Pakdd example](./examples/trials/auto-feature-engineering/README.md)
- Add a parallel algorithm to improve the performance of TPE with large concurrency. -PR 1052
- Support multiphase for hyperband -PR 1257

* Training Service
- Support private docker registry -PR 755

* Engineering Improvements
- Python wrapper for rest api, support retrieve the values of the metrics in a programmatic way PR 1318
- New python API : get_experiment_id(), get_trial_id() -PR 1353 -Issue 1331 & -Issue1368
- Optimized NAS Searchspace -PR 1393
+ Unify NAS search space with _type -- "mutable_type"e
+ Update random search tuner
- Set gpuNum as optional -Issue 1365
- Remove outputDir and dataDir configuration in PAI mode -Issue 1342
- When creating a trial in Kubeflow mode, codeDir will no longer be copied to logDir -Issue 1224

* Web Portal & User Experience
- Show the best metric curve during search progress in WebUI -Issue 1218
- Show the current number of parameters list in multiphase experiment -Issue1210 -PR 1348
- Add "Intermediate count" option in AddColumn. -Issue 1210
- Support search parameters value in WebUI -Issue 1208
- Enable automatic scaling of axes for metric value in default metric graph -Issue 1360
- Add a detailed documentation link to the nnictl command in the command prompt -Issue 1260
- UX improvement for showing Error log -Issue 1173

* Documentation
- Update the docs structure -Issue 1231
- [Multi phase document improvement](./docs/en_US/AdvancedFeature/MultiPhase.md) -Issue 1233 -PR 1242
+ Add configuration example
- [WebUI description improvement](./docs/en_US/Tutorial/WebUI.md) -PR 1419


Bug fix
* (Bug fix)Fix the broken links in 0.9 release -Issue 1236
* (Bug fix)Script for auto-complete
* (Bug fix)Fix pipeline issue that it only check exit code of last command in a script. -PR 1417
* (Bug fix)quniform fors tuners -Issue 1377
* (Bug fix)'quniform' has different meaning beween GridSearch and other tuner. -Issue 1335
* (Bug fix)"nnictl experiment list" give the status of a "RUNNING" experiment as "INITIALIZED" -PR 1388
* (Bug fix)SMAC cannot be installed if nni is installed in dev mode -Issue 1376
* (Bug fix)The filter button of the intermediate result cannot be clicked -Issue 1263
* (Bug fix)API "/api/v1/nni/trial-jobs/xxx" doesn't show a trial's all parameters in multiphase experiment -Issue 1258
* (Bug fix)Succeeded trial doesn't have final result but webui show ×××(FINAL) -Issue 1207
* (Bug fix)IT for nnictl stop -Issue 1298
* (Bug fix)fix security warning
* (Bug fix)Hyper-parameter page broken -Issue 1332
* (Bug fix)Run flake8 tests to find Python syntax errors and undefined names -PR 1217

Major Features
* General NAS programming interface
* Add `enas-mode` and `oneshot-mode` for NAS interface: [PR 1201](https://github.com/microsoft/nni/pull/1201#issue-291094510)
* [Gaussian Process Tuner with Matern kernel](https://github.com/microsoft/nni/blob/master/docs/en_US/Tuner/GPTuner.md)

* Multiphase experiment supports
* Added new training service support for multiphase experiment: PAI mode supports multiphase experiment since v0.9.
* Added multiphase capability for the following builtin tuners:
* TPE, Random Search, Anneal, Naïve Evolution, SMAC, Network Morphism, Metis Tuner.

For details, please refer to [Write a tuner that leverages multi-phase](https://github.com/microsoft/nni/blob/master/docs/en_US/AdvancedFeature/MultiPhase.md#write-a-tuner-that-leverages-multi-phase)

* Web Portal
* Enable trial comparation in Web Portal. For details, refer to [View trials status](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.md#view-trials-status)
* Allow users to adjust rendering interval of Web Portal. For details, refer to [View Summary Page](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.md#view-summary-page)
* show intermediate results more friendly. For details, refer to [View trials status](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.md#view-trials-status)
* [Commandline Interface](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/Nnictl.md)
* `nnictl experiment delete`: delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
* `nnictl platform clean`: It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
Bug fix and other changes
* Tuner Installation Improvements: add [sklearn](https://scikit-learn.org/stable/) to nni dependencies.
* (Bug Fix) Failed to connect to PAI http code - [Issue 1076](https://github.com/microsoft/nni/issues/1076)
* (Bug Fix) Validate file name for PAI platform - [Issue 1164](https://github.com/microsoft/nni/issues/1164)
* (Bug Fix) Update GMM evaluation in Metis Tuner
* (Bug Fix) Negative time number rendering in Web Portal - [Issue 1182](https://github.com/microsoft/nni/issues/1182), [Issue #1185](https://github.com/microsoft/nni/issues/1185)
* (Bug Fix) Hyper-parameter not shown correctly in WebUI when there is only one hyper parameter - [Issue 1192](https://github.com/microsoft/nni/issues/1192)

Major Features

* Support NNI on Windows for PAI/Remote mode

* NNI running on windows for remote mode

* NNI running on windows for PAI mode

* [Advanced features for using GPU](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/ExperimentConfig.md#configuration-spec)

* Run multiple trial jobs on the same GPU for local and remote mode

* Run trial jobs on the GPU running non-NNI jobs

* Kubeflow v1beta2 operator

* Support Kubeflow TFJob/PyTorchJob v1beta2

* [General NAS programming interface](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/GeneralNasInterfaces.md)

* Provide NAS programming interface for users to easily express their neural architecture search space through NNI annotation

* Provide a new command `nnictl trial codegen` for debugging the NAS code

* Tutorial of NAS programming interface, example of NAS on mnist, customized random tuner for NAS

* Support resume tuner/advisor's state for experiment resume

* For experiment resume, tuner/advisor will be resumed by replaying finished trial data

* Web Portal

* Improve the design of copying trial's parameters

* Support 'randint' type in hyper-parameter graph

* Use should ComponentUpdate to avoid unnecessary render

Bug fix and other changes

* Bug fix that `nnictl update` has inconsistent command styles

* Support import data for SMAC tuner

* Bug fix that experiment state transition from ERROR back to RUNNING

* Fix bug of table entries

* Nested search space refinement

* Refine 'randint' type and support lower bound

* [Comparison of different hyper-parameter tuning algorithm](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/HpoComparision.md)

* [Comparison of NAS algorithm](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/NasComparision.md)

* [NNI practice on Recommenders](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/NniPracticeSharing/RecommendersSvd.md)

Major Features
* [Support NNI on Windows](./docs/en_US/WindowsLocalMode.md)
* NNI running on windows for local mode
* [New advisor: BOHB](./docs/en_US/bohbAdvisor.md)
* Support a new advisor BOHB, which is a robust and efficient hyperparameter tuning algorithm, combines the advantages of Bayesian optimization and Hyperband
* [Support import and export experiment data through nnictl](./docs/en_US/NNICTLDOC.mdexperiment)
* Generate analysis results report after the experiment execution
* Support import data to tuner and advisor for tuning
* [Designated gpu devices for NNI trial jobs](./docs/en_US/ExperimentConfig.mdlocalConfig)
* Specify GPU devices for NNI trial jobs by gpuIndices configuration, if gpuIndices is set in experiment configuration file, only the specified GPU devices are used for NNI trial jobs.
* Web Portal enhancement
* Decimal format of metrics other than default on the Web UI
* Hints in WebUI about Multi-phase
* Enable copy/paste for hyperparameters as python dict
* Enable early stopped trials data for tuners.
* NNICTL provide better error message
* nnictl provide more meaningful error message for yaml file format error

Bug fix
* Unable to kill all python threads after nnictl stop in async dispatcher mode
* nnictl --version does not work with make dev-instal
* All trail jobs status stays on 'waiting' for long time on PAI platform

Major Features
* [Version checking](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/PaiMode.md#version-check)
* check whether the version is consistent between nniManager and trialKeeper
* [Report final metrics for early stop job](https://github.com/Microsoft/nni/issues/776)
* If includeIntermediateResults is true, the last intermediate result of the trial that is early stopped by assessor is sent to tuner as final result. The default value of includeIntermediateResults is false.
* [Separate Tuner/Assessor](https://github.com/Microsoft/nni/issues/841)
* Adds two pipes to separate message receiving channels for tuner and assessor.
* Make log collection feature configurable
* Add intermediate result graph for all trials

Bug fix
* [Add shmMB config key for PAI](https://github.com/Microsoft/nni/issues/842)
* Fix the bug that doesn't show any result if metrics is dict
* Fix the number calculation issue for float types in hyperband
* Fix a bug in the search space conversion in SMAC tuner
* Fix the WebUI issue when parsing experiment.json with illegal format
* Fix cold start issue in Metis Tuner

* Add release note.
* Fix Metis tuner cold start issue.