Optuna

This is the release note of [v2.0.0](https://github.com/optuna/optuna/milestone/26?closed=1).

Highlights

The second major version of Optuna 2.0 is released. It accommodates a multitude of new features, including Hyperband pruning, hyperparameter importance, built-in CMA-ES support, grid sampler, and LightGBM integration. Storage access is also improved, significantly speeding up optimization. Documentation has been revised and navigation is made easier. See the [blog](https://medium.com/vincent_44453/optuna-v2-3165e3f1fc2) for details.

Hyperband Pruner

The stable version of `HyperbandPruner` is available with a simpler interface and improved performance.

<img src="https://user-images.githubusercontent.com/5983694/88739399-6be88200-d175-11ea-9985-ce8c71d9538f.png" width="540px">

Hyperparameter Importance

The stable version of the hyperparameter importance module is available.
- Our implementation of fANOVA, `FanovaImportanceEvaluator` is now the default importance evaluator. This replaces the previous requirement for [`fanova`](https://github.com/automl/fanova) with scikit-learn.
- A new importance visualization function `visualization.plot_param_importances`.


Built-in CMA-ES Sampler

The stable version of `CmaEsSampler` is available. This new `CmaEsSampler` can be used with pruning for major performance improvements.

Grid Sampler

The stable version of `GridSampler` is available through an intuitive interface for users familiar with Optuna. When the entire grid is exhausted, the optimization stops automatically, so you can specify `n_trials=None`.

LightGBM Tuner

The stable version of `LightGBMTuner` is available. The behavior regarding `verbosity` option has been improved. The random seed was fixed unexpectedly if the verbosity level is not zero, but now the user given seed is used correctly.

Experimental Features

- New integration modules: TensorBoard integration, Catalyst integration, and AllenNLP pruning integration are available as experimental.
- A new visualization function for multi-objective optimization: `multi_objective.visualization.plot_pareto_front` is available as an experimental feature.
- New methods to manually create/add trials: `trial.create_trial` and `study.Study.add_trial` are available as experimental features.

Breaking Changes

Several deprecated features (e.g., `Study.study_id` and `Trial.trial_id`) are removed. See 1346 for details.

- Remove deprecated features in `optuna.trial` (1371)
- Remove deprecated arguments from `LightGBMTuner` (1374)
- Remove deprecated features in `integration/chainermn.py` (1375)
- Remove deprecated features in `optuna/structs.py` (1377)
- Remove deprecated features in `optuna/study.py` (1379)

Several features are deprecated.

- Deprecate `optuna study optimize` command (1384)
- Deprecate `step` argument in `IntLogUniformDistribution` (1387, thanks nzw0301!)

Other.

- `BaseStorage.set_trial_param` to return `None` instead of `bool` (1327)
- Match `suggest_float` and `suggest_int` specifications on `step` and `log` arguments (1329)
- `BaseStorage.set_trial_intermediate_valute` to return `None` instead of `bool` (1337)
- Make `optuna.integration.lightgbm_tuner` private (1378)
- Fix pruner index handling to 0-indexing (1430, thanks bigbird555!)
- Continue to allow using `IntLogUnioformDistribution.step` during deprecation (1438)
- Align `LightGBMTuner` verbosity level to the original LightGBM (1504)

New Features

- Add snippet of API for integration with Catalyst (1056, thanks VladSkripniuk!)
- Add pruned trials to trials being considered in `CmaEsSampler` (1229)
- Add pruned trials to trials being considered in `SkoptSampler` (1431)
- Add TensorBoard integration (1244, thanks VladSkripniuk!)
- Add deprecation decorator (1382)
- Add `plot_pareto_front` function (1303)
- Remove experimental decorator from `HyperbandPruner` (1435)
- Remove experimental decorators from hyperparameter importance (HPI) features (1440)
- Remove experimental decorator from `Study.stop` (1450)
- Remove experimental decorator from `GridSampler` (1451)
- Remove experimental decorators from `LightGBMTuner` (1452)
- Introducing `optuna.visualization.plot_param_importances` (1299)
- Rename `integration/CmaEsSampler` to `integration/PyCmaSampler` (1325)
- Match `suggest_float` and `suggest_int` specifications on `step` and `log` arguments (1329)
- `optuna.create_trial` and `Study.add_trial` to create custom studies (1335)
- Allow omitting the removal version in `deprecated` (1418)
- Mark `CatalystPruningCallback` integration as experimental (1465)
- Followup TensorBoard integration (1475)
- Implement a pruning callback for AllenNLP (1399)
- Remove experimental decorator from HPI visualization (1477)
- Add `optuna.visualization.plot_edf` function (1482)
- `FanovaImportanceEvaluator` as default importance evaluator (1491)
- Reduce HPI variance with default args (1492)

Enhancements

- Support automatic stop of `GridSampler` (1026)
- Implement fANOVA using `sklearn` instead of `fanova` (1106)
- Add a caching mechanism to make `NSGAIIMultiObjectiveSampler` faster (1257)
- Add `log` argument support for `suggest_int` of skopt integration (1277, thanks nzw0301!)
- Add `read_trials_from_remote_storage` method to Storage implementations (1298)
- Implement `log` argument for `suggest_int` of pycma integration (1302)
- Raise `ImportError` if `bokeh` version is 2.0.0 or newer (1326)
- Fix the x-axis title of the hyperparameter importances plot (1336, thanks harupy!)
- `BaseStorage.set_trial_intermediate_valute` to return `None` instead of `bool` (1337)
- Simplify log messages (1345)
- Improve layout of `plot_param_importances` figure (1355)
- Do not run the GC after every trial by default (1380)
- Skip storage access if logging is disabled (1403)
- Specify `stacklevel` for `warnings.warn` for more helpful warning message (1419, thanks harupy!)
- Replace `DeprecationWarning` with `FutureWarning` in `deprecated` (1428)
- Fix pruner index handling to 0-indexing (1430, thanks bigbird555!)
- Load environment variables in `AllenNLPExecutor` (1449)
- Stop overwriting seed in `LightGBMTuner` (1461)
- Suppress progress bar of `LightGBMTuner` if `verbosity` == 1 (1460)
- RDB storage to do eager backref "join"s when fetching all trials (1501)
- Overwrite intermediate values correctly (1517)
- Overwrite parameters correctly (1518)
- Always cast choices into tuple in `CategoricalDistribution` (1520)

Bug Fixes

RDB Storage Bugs on Distributed Optimization are Fixed

Several critical bugs are addressed in this release with the RDB storage, most related to distributed optimization.

- Fix CMA-ES boundary constraints and initial mean vector of LogUniformDistribution (1243)
- Temporary hotfix for `sphinx` update breaking existing type annotations (1342)
- Fix for PyTorch Lightning v0.8.0 (1392)
- Fix exception handling in `ChainerMNStudy.optimize` (1406)
- Use `step` to calculate range of `IntUniformDistribution` in `PyCmaSampler` (1456)
- Avoid exploding queries with large exclusion sets (1467)
- Temporary fix for problem with length limit of 5000 in MLflow (1481, thanks PhilipMay!)
- Fix race condition for trial number computation (1490)
- Fix `CachedStorage` skipping trial param row insertion on cache miss (1498)
- Fix `_CachedStorage` and `RDBStorage` distribution compatibility check race condition (1506)
- Fix frequent deadlock caused by conditional locks (1514)

Installation

- [Backport] Add `packaging` in install_requires (1561)
- Set `python_requires` in `setup.py` to clarify supported Python version (1350, thanks harupy!)
- Specify `classifiers` in setup.py (1358)
- Hotfix to avoid latest `keras` 2.4.0 (1386)
- Hotfix to avoid PyTorch Lightning 0.8.0 (1391)
- Relax `sphinx` version (1393)
- Update version constraints of `cmaes` (1404)
- Align `sphinx-rtd-theme` and Python versions used on Read the Docs to CircleCI (1434, thanks harupy!)
- Remove checking and alerting installation `pfnopt` (1474)
- Avoid latest `sphinx` (1485)
- Add `packaging` in install_requires (1561)

Documentation

- Fix experimental decorator (1248, thanks harupy!)
- Create a documentation for the root namespace `optuna` (1278)
- Add missing documentation for `BaseStorage.set_trial_param` (1316)
- Fix documented exception type in `BaseStorage.get_best_trial` and add unit tests (1317)
- Add hyperlinks to key features (1331)
- Add `.readthedocs.yml` to use the same document dependencies on the CI and Read the Docs (1354, thanks harupy!)
- Use `Colab` to demonstrate a notebook instead of `nbviewer` (1360)
- Hotfix to allow building the docs by avoiding latest `sphinx` (1369)
- Update layout and color of docs (1370)
- Add FAQ section about OOM (1385)
- Rename a title of reference to a module name (1390)
- Add a list of functions and classes for each module in reference doc (1400)
- Use `.. warning::` instead of `.. note::` for the deprecation decorator (1407)
- Always use Sphinx RTD theme (1414)
- Fix color of version/build in documentation sidebar (1415)
- Use a different font color for argument names (1436, thanks harupy!)
- Move css from `_templates/footer.html` to `_static/css/custom.css` (1439)
- Add missing commas in FAQ (1458)
- Apply auto-formatting to `custom.css` to make it pretty and consistent (1463, thanks harupy!)
- Update `CONTRIBUTING.md` (1466)
- Add missing `CatalystPruningCallback` in the documentation (1468, thanks harupy!)
- Fix incorrect type annotations for `catch` (1473, thanks harupy!)
- Fix double `FrozenTrial` (1478)
- Wider main content container in the documentation (1483)
- Add `TensorBoardCallback` to docs (1486)
- Add description about zero-based numbering of `step` (1489)
- Add links to examples from the integration references (1507)
- Fix broken link in `plot_edf` (1510)
- Update docs of default importance evaluator (1524)

Examples

- Set `timeout` for relatively long-running examples (1349)
- Fix broken link to example and add README for AllenNLP examples (1397)
- Add whitespace before opening parenthesis (1398)
- Fix GPU run for PyTorch Ignite and Lightning examples (1444, thanks arisliang!)
- Add Stable-Baselines3 RL Example (1420, thanks araffin!)
- Replace `suggest_*uniform` in examples with `suggest_(int|float)` (1470)

Tests

- Fix `plot_param_importances` test (1328)
- Fix incorrect test names in `test_experimental.py` (1332, thanks harupy!)
- Simplify decorator tests (1423)
- Add a test for `CmaEsSampler._get_trials()` (1433)
- Use argument of `pytorch_lightning.Trainer` to disable `checkpoint_callback` (1453)
- Install RDB servers and their bindings for storage tests (1497)
- Upgrade versions of `pytorch` and `torchvision` (1502)
- Make HPI tests deterministic (1505)

Code Fixes

- Introduces `optuna._imports.try_import` to DRY optional imports (1315)
- Friendlier error message for unsupported `plotly` versions (1338)
- Rename private modules in `optuna.visualization` (1359)
- Rename private modules in `optuna.pruners` (1361)
- Rename private modules in `optuna.samplers` (1362)
- Change `logger` to `_trial`'s module variable (1363)
- Remove deprecated features in `HyperbandPruner` (1366)
- Add missing `__init__.py` files (1367, thanks harupy!)
- Fix double quotes from Black formatting (1372)
- Rename private modules in `optuna.storages` (1373)
- Add a list of functions and classes for each module in reference doc (1400)
- Apply deprecation decorator (1413)
- Remove unnecessary exception handling for `GridSampler` (1416)
- Remove either `warnings.warn()` or `optuna.logging.Logger.warning()` from codes which have both of them (1421)
- Simplify usage of `deprecated` by omitting removed version (1422)
- Apply experimental decorator (1424)
- Fix the experimental warning message for `CmaEsSampler` (1432)
- Remove `optuna.structs` from MLflow integration (1437)
- Add type hints to `slice.py` (1267, thanks bigbird555!)
- Add type hints to `intermediate_values.py` (1268, thanks bigbird555!)
- Add type hints to `optimization_history.py` (1269, thanks bigbird555!)
- Add type hints to `utils.py` (1270, thanks bigbird555!)
- Add type hints to `test_logging.py` (1284, thanks bigbird555!)
- Add type hints to `test_chainer.py` (1286, thanks bigbird555!)
- Add type hints to `test_keras.py` (1287, thanks bigbird555!)
- Add type hints to `test_cma.py` (1288, thanks bigbird555!)
- Add type hints to `test_fastai.py` (1289, thanks bigbird555!)
- Add type hints to `test_integration.py` (1293, thanks bigbird555!)
- Add type hints to `test_mlflow.py` (1322, thanks bigbird555!)
- Add type hints to `test_mxnet.py` (1323, thanks bigbird555!)
- Add type hints to `optimize.py` (1364, thanks bigbird555!)
- Replace `suggest_*uniform` in examples with `suggest_(int|float)` (1470)
- Add type hints to `distributions.py` (1513)
- Remove unnecessary `FloatingPointDistributionType` (1516)

Continuous Integration

- Add a step to push images to Docker Hub (1295)
- Check code coverage in `tests-python37` on CircleCI (1348)
- Stop building Docker images in Pull Requests (1389)
- Prevent `doc-link` from running on unrelated status update events (1410, thanks harupy!)
- Avoid latest `ConfigSpace` where Python 3.5 is dropped (1471)
- Run unit tests on GitHub Actions (1352)
- Use `circleci/python` for dev image and install RDB servers (1495)
- Install RDB servers and their bindings for storage tests (1497)
- Fix `dockerimage.yml` format (1511)
- Revert 1495 and 1511 (1512)
- Run daily unit tests (1515)

Other

- Add TestPyPI release to workflow (1291)
- Add PyPI release to workflow (1306)
- Exempt issues with `no-stale` label from stale bot (1321)
- Remove stale labels from Issues or PRs when they are updated or commented on (1409)
- Exempt PRs with `no-stale` label from stale bot (1427)
- Update the documentation section in `CONTRIBUTING.md` (1469, thanks harupy!)
- Bump up version to `2.0.0` (1525)

A release candidate for the second major version of Optuna [v2.0.0-rc0](https://github.com/optuna/optuna/milestone/25?closed=1) is released! This release includes a lot of new features, cleaned up interfaces, performance improvements, internal refactorings and more. If you find any problems with this release candidate, please feel free to report them via GitHub Issues or Gitter.

Highlights

Hyperband Pruner

The stable version of `HyperbandPruner` is available. It has a more simple interface and has seen performance improvement.

Hyperparameter Importance

The stable version of the hyperparameter importance module is available.
- Our own implemented fANOVA, `FanovaImportanceEvaluator`. While the previous implementation required [`fanova`](https://github.com/automl/fanova), this new `FanovaImportanceEvaluator` can be used with only scikit-learn.
- A new importance visualization function `visualization.plot_param_importances`.

Built-in CMA-ES Sampler

The stable version of `CmaEsSampler` is available. This new `CmaEsSampler` can be used with pruning, one of the Optuna’s important features, for great performance improvements.

Grid Sampler

The stable version of `GridSampler` is available and can be through an intuitive interface for users familiar with Optuna. When the entire grid is exhausted, the optimization also automatically stops so you can specify `n_trials=None`.

LightGBM Tuner

The stable version of `LightGBMTuner` is available. The behavior regarding `verbosity` option has been improved. The random seed was fixed unexpectedly if the verbosity level is not 0, but now the user given seed is used correctly.

Experimental Features

- New integration modules: TensorBoard integration and Catalyst integration are available as experimental.
- A new visualization function for multi-objective optimization: `multi_objective.visualization.plot_pareto_front` is available as an experimental feature.
- New methods to manually create/add trials: `trial.create_trial` and `study.Study.add_trial` are available as experimental features.

Breaking Changes

Several deprecated features (e.g., `Study.study_id` and `Trial.trial_id`) are removed. See 1346 for details.

- Remove deprecated features in `optuna.trial`. (1371)
- Remove deprecated arguments from `LightGBMTuner`. (1374)
- Remove deprecated features in `integration/chainermn.py`. (1375)
- Remove deprecated features in `optuna/structs.py`. (1377)
- Remove deprecated features in `optuna/study.py`. (1379)

Several features are deprecated.

- Deprecate `optuna study optimize` command. (1384)
- Deprecate `step` argument in `IntLogUniformDistribution`. (1387, thanks nzw0301!)

Other.

- `BaseStorage.set_trial_param` to return `None` instead of `bool`. (1327)
- Match `suggest_float` and `suggest_int` specifications on `step` and `log` arguments. (1329)
- `BaseStorage.set_trial_intermediate_valute` to return `None` instead of `bool`. (1337)
- Make `optuna.integration.lightgbm_tuner` private. (1378)
- Fix pruner index handling to 0-indexing. (1430, thanks bigbird555!)
- Continue to allow using `IntLogUnioformDistribution.step` during deprecation. (1438)

New Features

- Add snippet of API for integration with Catalyst. (1056, thanks VladSkripniuk!)
- Add pruned trials to trials being considered in `CmaEsSampler`. (1229)
- Add pruned trials to trials being considered in `SkoptSampler`. (1431)
- Add TensorBoard integration. (1244, thanks VladSkripniuk!)
- Add deprecation decorator. (1382)
- Add `plot_pareto_front` function. (1303)
- Remove experimental decorator from `HyperbandPruner`. (1435)
- Remove experimental decorators from hyperparameter importance (HPI) features. (1440)
- Remove experimental decorator from `Study.stop`. (1450)
- Remove experimental decorator from `GridSampler`. (1451)
- Remove experimental decorators from `LightGBMTuner`. (1452)
- Introducing `optuna.visualization.plot_param_importances`. (1299)
- Rename `integration/CmaEsSampler` to `integration/PyCmaSampler`. (1325)
- Match `suggest_float` and `suggest_int` specifications on `step` and `log` arguments. (1329)
- `optuna.create_trial` and `Study.add_trial` to create custom studies. (1335)
- Allow omitting the removal version in `deprecated`. (1418)
- Mark `CatalystPruningCallback` integration as experimental. (1465)
- Followup TensorBoard integration. (1475)

Enhancements

- Support automatic stop of `GridSampler`. (1026)
- Implement fANOVA using `sklearn` instead of `fanova`. (1106)
- Add a caching mechanism to make `NSGAIIMultiObjectiveSampler` faster. (1257)
- Add `log` argument support for `suggest_int` of skopt integration. (1277, thanks nzw0301!)
- Add `read_trials_from_remote_storage` method to Storage implementations. (1298)
- Implement `log` argument for `suggest_int` of pycma integration. (1302)
- Raise `ImportError` if `bokeh` version is 2.0.0 or newer. (1326)
- Fix the x-axis title of the hyperparameter importances plot. (1336, thanks harupy!)
- `BaseStorage.set_trial_intermediate_valute` to return `None` instead of `bool`. (1337)
- Simplify log messages. (1345)
- Improve layout of `plot_param_importances` figure. (1355)
- Do not run the GC after every trial by default. (1380)
- Skip storage access if logging is disabled. (1403)
- Specify `stacklevel` for `warnings.warn` for more helpful warning message. (1419, thanks harupy!)
- Replace `DeprecationWarning` with `FutureWarning` in `deprecated`. (1428)
- Fix pruner index handling to 0-indexing. (1430, thanks bigbird555!)
- Load environment variables in `AllenNLPExecutor`. (1449)
- Stop overwriting seed in `LightGBMTuner`. (1461)

Bug Fixes

- Fix CMA-ES boundary constraints and initial mean vector of LogUniformDistribution. (1243)
- Temporary hotfix for `sphinx` update breaking existing type annotations. (1342)
- Fix for PyTorch Lightning v0.8.0. (1392)
- Fix exception handling in `ChainerMNStudy.optimize`. (1406)
- Use `step` to calculate range of `IntUniformDistribution` in `PyCmaSampler`. (1456)

Installation

- Set `python_requires` in `setup.py` to clarify supported Python version. (1350, thanks harupy!)
- Specify `classifiers` in setup.py. (1358)
- Hotfix to avoid latest `keras` 2.4.0. (1386)
- Hotfix to avoid PyTorch Lightning 0.8.0. (1391)
- Relax `sphinx` version. (1393)
- Update version constraints of `cmaes`. (1404)
- Align `sphinx-rtd-theme` and Python versions used on Read the Docs to CircleCI. (1434, thanks harupy!)
- Remove checking and alerting installation `pfnopt`. (1474)

Documentation

- Fix experimental decorator. (1248, thanks harupy!)
- Create a documentation for the root namespace `optuna`. (1278)
- Add missing documentation for `BaseStorage.set_trial_param`. (1316)
- Fix documented exception type in `BaseStorage.get_best_trial` and add unit tests. (1317)
- Add hyperlinks to key features. (1331)
- Add `.readthedocs.yml` to use the same document dependencies on the CI and Read the Docs. (1354, thanks harupy!)
- Use `Colab` to demonstrate a notebook instead of `nbviewer`. (1360)
- Hotfix to allow building the docs by avoiding latest `sphinx`. (1369)
- Update layout and color of docs. (1370)
- Add FAQ section about OOM. (1385)
- Rename a title of reference to a module name. (1390)
- Add a list of functions and classes for each module in reference doc. (1400)
- Use `.. warning::` instead of `.. note::` for the deprecation decorator. (1407)
- Always use Sphinx RTD theme. (1414)
- Fix color of version/build in documentation sidebar. (1415)
- Use a different font color for argument names. (1436, thanks harupy!)
- Move css from `_templates/footer.html` to `_static/css/custom.css`. (1439)
- Add missing commas in FAQ. (1458)
- Apply auto-formatting to `custom.css` to make it pretty and consistent. (1463, thanks harupy!)
- Update `CONTRIBUTING.md`. (1466)
- Add missing `CatalystPruningCallback` in the documentation. (1468, thanks harupy!)
- Fix incorrect type annotations for `catch`. (1473, thanks harupy!)

Examples

- Set `timeout` for relatively long-running examples. (1349)
- Fix broken link to example and add README for AllenNLP examples. (1397)
- Add whitespace before opening parenthesis. (1398)
- Fix GPU run for PyTorch Ignite and Lightning examples. (1444, thanks arisliang!)

Tests

- Fix `plot_param_importances` test. (1328)
- Fix incorrect test names in `test_experimental.py`. (1332, thanks harupy!)
- Simplify decorator tests. (1423)
- Add a test for `CmaEsSampler._get_trials()`. (1433)
- Use argument of `pytorch_lightning.Trainer` to disable `checkpoint_callback`. (1453)

Code Fixes

- Introduces `optuna._imports.try_import` to DRY optional imports. (1315)
- Friendlier error message for unsupported `plotly` versions. (1338)
- Rename private modules in `optuna.visualization`. (1359)
- Rename private modules in `optuna.pruners`. (1361)
- Rename private modules in `optuna.samplers`. (1362)
- Change `logger` to `_trial`'s module variable. (1363)
- Remove deprecated features in `HyperbandPruner`. (1366)
- Add missing `__init__.py` files. (1367, thanks harupy!)
- Fix double quotes from Black formatting. (1372)
- Rename private modules in `optuna.storages`. (1373)
- Add a list of functions and classes for each module in reference doc. (1400)
- Apply deprecation decorator. (1413)
- Remove unnecessary exception handling for `GridSampler`. (1416)
- Remove either `warnings.warn()` or `optuna.logging.Logger.warning()` from codes which have both of them. (1421)
- Simplify usage of `deprecated` by omitting removed version. (1422)
- Apply experimental decorator. (1424)
- Fix the experimental warning message for `CmaEsSampler`. (1432)
- Remove `optuna.structs` from MLflow integration. (1437)
- Add type hints to `slice.py`. (1267, thanks bigbird555!)
- Add type hints to `intermediate_values.py`. (1268, thanks bigbird555!)
- Add type hints to `optimization_history.py`. (1269, thanks bigbird555!)
- Add type hints to `utils.py`. (1270, thanks bigbird555!)
- Add type hints to `test_logging.py`. (1284, thanks bigbird555!)
- Add type hints to `test_chainer.py`. (1286, thanks bigbird555!)
- Add type hints to `test_keras.py`. (1287, thanks bigbird555!)
- Add type hints to `test_cma.py`. (1288, thanks bigbird555!)
- Add type hints to `test_fastai.py`. (1289, thanks bigbird555!)
- Add type hints to `test_integration.py`. (1293, thanks bigbird555!)
- Add type hints to `test_mlflow.py`. (1322, thanks bigbird555!)
- Add type hints to `test_mxnet.py`. (1323, thanks bigbird555!)
- Add type hints to `optimize.py`. (1364, thanks bigbird555!)

Continuous Integration

- Add a step to push images to Docker Hub. (1295)
- Check code coverage in `tests-python37` on CircleCI. (1348)
- Stop building Docker images in Pull Requests. (1389)
- Prevent `doc-link` from running on unrelated status update events. (1410, thanks harupy!)
- Avoid latest `ConfigSpace` where Python 3.5 is dropped. (1471)

Other

- Add TestPyPI release to workflow. (1291)
- Add PyPI release to workflow. (1306)
- Exempt issues with `no-stale` label from stale bot. (1321)
- Remove stale labels from Issues or PRs when they are updated or commented on. (1409)
- Exempt PRs with `no-stale` label from stale bot. (1427)

This is the release note of [v1.5.0](https://github.com/optuna/optuna/milestone/24?closed=1).

Highlights

LightGBM Tuner with Cross-validation

LightGBM tuner, which provides efficient [stepwise parameter tuning](https://medium.com/optuna/lightgbm-tuner-new-optuna-integration-for-hyperparameter-optimization-8b7095e99258) for LightGBM, supports cross-validation as an experimental feature with `LightGBMTunerCV`. See #1156 for details.


NSGA-II

A sampler based on NSGA-II, a well-known multi-objective optimization algorithm, is now available as the default multi-objective sampler.
The following benchmark result, on the ZDT1 function, shows that NSGA-II outperforms random sampling.
Please refer to 1163 for further details.


Mean Decrease Impurity (MDI) Hyperparameter Importance Evaluator

The default hyperparameter importance evaluator is replaced with a naive mean decrease impurity algorithm. It uses the random forest feature importances in Scikit-learn and therefore requires this package. See 1253 for more details.



`optuna.TrialPruned` Alias

`optuna.TrialPruned` is a new alias for `optuna.exceptions.TrialPruned`. It is now possible to write shorter and more readable code when pruning trials. See 1204 for details.

New Features

- Add a method to stop `study.optimize`. (1025)
- Use `--study-name` instead of `--study` in CLI commands. (1079, thanks seungjaeryanlee!)
- Add cross-validation support for `LightGBMTuner`. (1156)
- Add NSGA-II based multi-objective sampler. (1163)
- Implement log argument for `suggest_int`. (1201, thanks nzw0301!)
- Import `optuna.exceptions.TrialPruned` in `__init__.py`. (1204)
- Mean Decrease Impurity (MDI) hyperparameter importance evaluator. (1253)

Enhancements

- Add storage cache. (1140)
- Fix `_get_observation_pairs` for conditional parameters. (1166, thanks y0z!)
- Alternative implementation to hide the interface so that all samplers can use `HyperbandPruner`. (1196)
- Fix for O(N) queries being produced if even a single trial is cached. (1259, thanks zzorba!)
- Move caching mechanism from `RDBStorage` to `_CachedStorage`. (1263)
- Cache study-related info in `_CachedStorage`. (1264)
- Move deep-copies for optimization speed improvement. (1274)
- Implement `log` argument for `suggest_int` of ChainerMN integration. (1275, thanks nzw0301!)
- Add warning when `Trial.suggest_int` modifies `high`. (1276)
- Input validation for `IntLogUniformDistribution`. (1279, thanks himkt!)

Bug Fixes


- Support multiple studies in `InMemoryStorage`. (1228)
- Fix out of bounds error of CMA-ES. (1231)
- Fix `sklearn` - `skopt` version incompatibility. (1236)
- Fix a type casting error when using `CmaEsSampler`. (1240)
- Upgrade the version of `cmaes`. (1242)

Documentation

- Rename `test_` to `valid_` in docs and docstring. (1167, thanks himkt!)
- Add storage specification to `BaseStorage` class doc. (1174)
- Add docstring to `BaseStorage` method interfaces. (1175)
- Add an explanation of failed trials from samplers' perspective. (1214)
- Add `LightGBMTuner` reference. (1217)
- Modifying code examples to include training data. (1221)
- Ask optuna tag in Stack Overflow question. (1249)
- Add notes for `auto` argument values in `HyperbandPruner` and `SuccessiveHalvingPruner`. (1252)
- Add description of `observation_key` in `XGBoostPruningCallback`. (1260)
- Cosmetic fixes to documentation in `BaseStorage`. (1261)
- Modify documentation and fix file extension in the test for AllenNLP integration. (1265, thanks himkt!)
- Fix `experimental` decorator to decorate a class properly. (1285, thanks harupy!)

Examples

- Add pruning to PyTorch example. (1119)
- Use `dump_best_config` in example. (1225, thanks himkt!)
- Stop suggesting using deprecated option in AllenNLP example. (1282)
- Add link to regression example in the header of `keras_integration.py`. (1301, thanks zishiwu123!)

Tests

- Increase test coverage of storage tests for single worker cases. (1191)
- Fix `sklearn` - `skopt` version incompatibility. (1236)

Code Fixes

- Dissect `trial.py`. (1210, thanks himkt!)
- Rename `trial/*.py` to `trial/_*.py`. (1239)
- Add type hints to contour.py. (1254, thanks bigbird555!)
- Consistent Hyperband bracket ID variable names. (1262)
- Apply minor code fix to 1201. (1273)
- Avoid mutable default argument in `AllenNLPExecutor.__init__`. (1280)
- Reorder arguments of `Trial.suggest_float`. (1292)
- Fix unintended change on calculating `n_brackets` in `HyperbandPruner`. (1294)
- Add experimental decorator to `LightGBMTuner` and `LightGBMTunerCV`. (1305)

Continuous Integration

- Add GitHub action that posts link to documentation. (1247, thanks harupy!)
- Add a workflow to create distribution packages. (1283)
- Stop setting environment variables for GitHub Package. (1296)

This is the release note of [v1.4.0](https://github.com/optuna/optuna/milestone/23?closed=1).

Highlights

Experimental Multi-objective Optimization

Multi-objective optimization is available as an experimental feature. Currently, it only provides random sampling, but it will be continuously developed in the following releases. Feedback is highly welcomed. See 1054 for details.

Enhancement of Storages

A new Redis-based storage is available. It is a fast and flexible in-memory storage. It can also persist studies on-disk without having to configure relational databases. It is still an experimental feature, and your feedback is highly welcomed. See 974 for details.


Performance tuning has been applied to `RDBStorage`. For instance, it speeds up creating study lists by over 3000 times (i.e., 7 minutes to 0.13 seconds). See 1109 for details.

Experimental Integration Modules for MLFlow and AllenNLP

A new callback function is provided for MLFlow users. It reports Optuna’s optimization results (i.e., parameter values and metric values) to MLFlow.
See 1028 for details.

A new integration module for AllenNLP is available. It enables you to reuse your jsonnet configuration files for hyperparameter tuning. See 1086 for details.



Breaking Changes

- Delete the argument `is_higher_better` from `TensorFlowPruningHook`. (1083, thanks nuka137!)
- Applied `abc.abstractmethod` decorator to the abstract methods of `BaseTrial` and fixed `ChainerMNTrial`. (1087, thanks gorogoroumaru!)
- Input validation for `LogUniformDistribution` for negative domains. (1099)

New Features

- Added `RedisStorage` class to support storing activity on Redis. (974, thanks pablete!)
- Add MLFlow integration callback. (1028, thanks PhilipMay!)
- Add `study` argument to `optuna.integration.lightgbm.LightGBMTuner`. (1032)
- Support multi-objective optimization. (1054)
- Add duration into `FrozenTrial` and `DataFrame`. (1071)
- Support parallel execution of `LightGBMTuner`. (1076)
- Add `number` property to `FixedTrial` and `BaseTrial`. (1077)
- Support `DiscreteUniformDistribution` in `suggest_float`. (1081, thanks himkt!)
- Add AllenNLP integration. (1086, thanks himkt!)
- Add an argument of `max_resource` to `HyperbandPruner` and deprecate `n_brackets`. (1138)
- Improve the trial allocation algorithm of `HyperbandPruner`. (1141)
- Add `IntersectionSearchSpace` to speed up the search space calculation. (1142)
- Implement AllenNLP config exporter to save training config with `best_params` in study. (1150, thanks himkt!)
- Remove redundancy from `HyperbandPruner` by deprecating `min_early_stopping_rate_low`. (1159)
- Add pruning interval for `KerasPruningCallback`. (1161, thanks VladSkripniuk!)
- `suggest_float` with step in `multi_objective`. (1205, thanks nzw0301!)

Enhancements

- Reseed sampler's random seed generator in Study. (968)
- Apply lazy import for `optuna.dashboard`. (1074)
- Applied `abc.abstractmethod` decorator to the abstract methods of `BaseTrial` and fixed `ChainerMNTrial`. (1087, thanks gorogoroumaru!)
- Refactoring of `StudyDirection`. (1090)
- Refactoring of `StudySummary`. (1095)
- Refactoring of `TrialState` and `FrozenTrial`. (1101)
- Apply lazy import for `optuna.structs` to raise `DeprecationWarning` when using. (1104)
- Optimize `get_all_strudy_summaries` function for RDB storages. (1109)
- `single()` returns True when `step` or `q` is greater than `high-low`. (1111)
- Return `trial_id` at `study._append_trial()`. (1114)
- Use `scipy` for sampling from truncated normal in TPE sampler. (1122)
- Remove unnecessary deep-copies. (1135)
- Remove unnecessary shape-conversion and a loop from TPE sampler. (1145)
- Support Optuna callback functions at LightGBM Tuner. (1158)
- Fix the default value of `max_resource` to `HyperbandPruner`. (1171)
- Fix the method to calculate ``n_brackets`` in `HyperbandPruner`. (1188)

Bug Fixes

- Support Copy-on-Write for thread safety in in-memory storage. (1139)
- Fix the range of sampling in TPE sampler. (1143)
- Add figure title to contour plot. (1181, thanks harupy!)
- Raise `ValueError` that is not raised. (1208, thanks harupy!)
- Fix a bug that occurs when multiple callbacks are passed to `MultiObjectiveStudy.optimize`. (1209)

Installation

- Set version constraint on the `cmaes` library. (1082)
- Stop installing PyTorch Lightning if Python version is 3.5. (1193)
- Install PyTorch without CPU option on macOS. (1215, thanks himkt!)

Documentation

- Add an Example and Variable Explanations to `HyperBandPruner`. (972)
- Add a reference of cli in the sphinx docs. (1065)
- Fix docstring on `optuna/integration/*.py`. (1070, thanks nuka137!)
- Fix docstring on `optuna/distributions.py`. (1089)
- Remove duplicate description of `FrozenTrial.distributions`. (1093)
- Optuna Read the Docs top page addition. (1098)
- Update the outputs of some examples in `first.rst`. (1100, thanks A03ki!)
- Fix `plot_intermediate_values` example. (1103)
- Thanks barneyhill for creating the original pull request 1050!
- Use latest `sphinx` version on RTD. (1108)
- Add class doc to `TPESampler`. (1144)
- Fix a markup in pruner page. (1172, thanks nzw0301!)
- Add examples for doctest to `optuna/storages/rdb/storage.py`. (1212, thanks nuka137!)

Examples

- Update PyTorch Lightning example for 0.7.1 version. (1013, thanks festeh!)
- Add visualization example script. (1085)
- Update `pytorch_simple.py` to suggest `lr` from `suggest_loguniform`. (1112)
- Rename test datasets in examples. (1164, thanks VladSkripniuk!)
- Fix the metric name in `KerasPruningCallback` example. (1218)

Tests

- Add TPE tests. (1126)
- Bundle allennlp test data in the repository. (1149, thanks himkt!)
- Add test for deprecation error of `HyperbandPruner`. (1189)
- Add examples for doctest to optuna/storages/rdb/storage.py. (1212, thanks nuka137!)

Code Fixes

- Update type hinting of `GridSampler.__init__`. (1102)
- Replace `mock` with `unittest.mock`. (1121)
- Remove unnecessary `is_log` logic in TPE sampler. (1123)
- Remove redundancy from `HyperbandPruner` by deprecating `min_early_stopping_rate_low`. (1159)
- Use `Trial.system_attrs` to store `LightGBMTuner`'s results. (1177)
- Remove `_TimeKeeper` and use `timeout` of `Study.optimize`. (1179)
- Define key names of `system_attrs` as variables in `LightGBMTuner`. (1192)
- Minor fixes. (1203, thanks nzw0301!)
- Move `colorlog` after `threading`. (1211, thanks himkt!)
- Pass `IntUniformDistribution`'s step to `UniformIntegerHyperparameter`'s `q`. (1222, thanks nzw0301!)

Continuous Integration

- Create dockerimage.yml. (901)
- Add notebook verification for visualization examples. (1088)
- Avoid installing `torch` with CUDA in CI. (1118)
- Avoid installing `torch` with CUDA in CI by locking version. (1124)
- Constraint `llvmlite` version for Python 3.5. (1152)
- Skip GitHub Actions builds on forked repositories. (1157, thanks himkt!)
- Fix `--cov` option for pytest. (1187, thanks harupy!)
- Unique GitHub Actions step name. (1190)

Other

- GitHub Actions to automatically label PRs. (1068)
- Revert "GitHub Actions to automatically label PRs.". (1094)
- Remove settings for yapf. (1110, thanks himkt!)
- Update pull request template. (1113)
- GitHub Actions to automatically label stale issues and PRs. (1116)
- Upgrade `actions/stale` to never close ticket. (1131)
- Run `actions/stale` on weekday mornings Tokyo time. (1132)
- Simplify pull request template. (1147)
- Use major version instead of semver for stale. (1173, thanks hross!)
- Do not label `contribution-welcome` and `bug` issues as stale. (1216)

This is the release note of [v1.3.0](https://github.com/optuna/optuna/milestone/22?closed=1).

Highlights

Experimental CMA-ES

A new built-in CMA−ES sampler is available. It is still an experimental feature, but we recommend trying it because it is much faster than the existing CMA-ES sampler from the `integration` submodule. See 920 for details.

Experimental Hyperparameter Importance

Hyperparameter importances can be evaluated using `optuna.importance.get_param_importances`. This is an experimental feature that currently requires [fanova](https://github.com/automl/fanova). See #946 for details.

Breaking Changes

Changes to the Per-Trial Log Format

The per-trial log now shows the parameter configuration for the last trial instead of the so far best trial. See 965 for details.

New Features

- Add `step` parameter on `IntUniformDistribution`. (910, thanks hayata-yamamoto!)
- Add CMA-ES sampler. (920)
- Add experimental hyperparameter importance feature. (946)
- Implement `ThresholdPruner`. (963, thanks himkt!)
- Add initial implementation of `suggest_float`. (1021, thanks himkt!)

Enhancements

- Log parameters from last trial instead of best trial. (965)
- Fix overlap of labels in parallel coordinate plots. (979, thanks VladSkripniuk!)

Bug Fixes

- Support metric aliases for LightGBM Tuner 960. (977, thanks VladSkripniuk!)
- Use `SELECT FOR UPDATE` while updating trial state. (1014)

Documentation

- Add FAQ entry on how to save/resume studies using in-memory storage. (869, thanks victorhcm!)
- Fix pruning `n_warmup_steps` documentation. (980, thanks PhilipMay!)
- Apply gray background for `code-block:: console`. (983)
- Add syntax highlighting and fixed spelling variants. (990, thanks daikikatsuragawa!)
- Add examples for doctest to `optuna/samplers/*.py` and `optuna/integration/*.py`. (999, thanks nuka137!)
- Embed plotly figures in documentation. (1003, thanks harupy!)
- Improve callback docs for optimize function. (1016, thanks PhilipMay!)
- Fix docstring on `optuna/integration/tensorflow.py`. (1019, thanks nuka137!)
- Fix docstring in `RDBStorage`. (1022)
- Fix direction in doctest. (1036, thanks himkt!)
- Add a link to the AllenNLP example in README.md. (1040)
- Apply document code formatting with Black. (1044, thanks PhilipMay!)
- Remove obsolete description from contribution guidelines. (1047)
- Improve contribution guidelines. (1052)
- Document `intersection_search_space` parameters. (1053)
- Add descriptions to cli commands. (1064)

Examples

- Add `allennlp` example. (949, thanks himkt!)

Code Fixes

- Add number field in trials table. (939)
- Implement some methods almost compatible with Scikit-learn private methods. (952, thanks himkt!)
- Use function annotation syntax for type hints. (989, 993, 996, thanks bigbird555!)
- Add RDB storage `number` column comment. (1006)
- Sort dependencies in setup.py (fix 1005). (1007, thanks VladSkripniuk!)
- Fix `mypy==0.770` errors. (1009)
- Fix a validation error message. (1010)
- Remove python version check. (1023)
- Fix a typo on `optuna/integration/pytorch_lightning.py`. (1024, thanks nai62!)
- Add a todo comment in `GridSampler`. (1027)
- Change formatter from `autopep8` to `black` (string normalization separate commit). (1030)
- Update module import of `sklearn.utils.safe_indexing` for `scikit-learn==0.24`. (1031, thanks kuroko1t!)
- Fix `black` error. (1034)
- Remove duplicate import of `FATAL`. (1035)
- Fix import order and plot label truncation. (1046)

Continuous Integration

- Add version restriction to `pytorch_lightning` and `bokeh`. (998)
- Relax PyTorch Lightning version constraint to fix daily CI build. (1002)
- Store documentation as an artifact on CircleCI. (1008, thanks harupy!)
- Introduce GitHub Action to execute CI for examples. (1011)
- Ignore allennlp in Python3.5 and Python3.8. (1042, thanks himkt!)
- Remove daily CircleCI builds. (1048)

Other

- Refactor Mypy configuration into `setup.cfg`. (985, thanks pablete!)
- Ignore `.pytest_cache`. (991, thanks harupy!)

This is the release note of [v1.2.0](https://github.com/optuna/optuna/milestone/21?closed=1).

Highlights

Trial Queue

`Study.enqueue_trial` allows the user to specify fixed parameter values to be tried next instead of values from the suggestion algorithms. An example is available in the [reference](https://optuna.readthedocs.io/en/latest/reference/study.html#optuna.study.Study.enqueue_trial). See 520.

Note that this feature introduced an RDB schema change. If you have stored studies created by Optuna v1.1.0 or less in RDBs, please execute `$ optuna storage upgrade --storage $URL` after updating Optuna.

Experimental Grid Search

Grid search has been introduced as an experimental feature through `GridSampler`. See 665.

Breaking Changes

- `Trial.report` to not update the trial value. (854)

New Features

- Add trial queue. (520)
- Add a sampler class based on grid search. (665)
- Display progress bar in optimization process. (844)
- `Trial.report` to not update the trial value. (854)
- Add `experimental` warning. (884)
- `experimental` decorator to accept optional `name` and apply to progress bar. (918)
- Add a `n_startup_trials` option to `SkoptSampler`. (951)

Enhancements

- Fix `OptunaSearchCV` by adding the method that was implemented in `scikit-learn>=0.22.1`. (881, thanks himkt!)
- User-friendly error message for fixed categorical parameter not found in choices. (900)
- Trial level suggest for same variable with different parameters give warning. (908, thanks PhilipMay!)
- More verbose warning message in `Trial._check_distribution`. (934, thanks PhilipMay!)

Bug Fixes

- Fix `LightGBMTuner` to handle metrics with evaluation positions. (912)
- Fix a TPE implementation on `DiscreteUniformDistribution`. (917)
- If you use `Trial.suggest_discrete_uniform` with the default sampler (i.e., `TPESampler`), optimization performance may degrade due to the issue (916). Please update Optuna to this version or later.
- Use `weakref.finalize` instead of `__del__` for `RDBStorage`. (941)

Documentation

- Add examples for `doctest` to `optuna/trial.py`. (882, thanks nuka137!)
- Add an example of `enqueue_trial`. (927)
- Fix typo for `cma.EvotionStrategy`. (929)
- Explain installing local Optuna explicitly. (932, thanks keisuke-umezawa!)
- Documentation of algorithm behind Optuna. (940, thanks arpitkh101!)
- Run a doctest of pycma sampler. (944)
- Fix doctest in logging. (953, thanks keisuke-umezawa!)
- Fix doctest in `parallel_coordinate`. (955, thanks keisuke-umezawa!)
- Add examples for `doctest` to `optuna/exceptions.py`. (958, thanks nuka137!)
- Remove description and example of `optuna.structs.TrialPruned` (followup of 958). (959)
- Add examples for `doctest` to `optuna/prunes/*.py`. (964, thanks nuka137!)
- Fix doctest in visualization. (973, thanks keisuke-umezawa!)

Examples

- Add an example of `xgboost.cv` using `XGBoostPruningCallback`. (907, thanks yutayamazaki!)
- TensorFlow estimator example with only v2 APIs. (924, thanks nuka137!)
- TensorFlow estimator integration example with only v2 APIs. (935, thanks nuka137!)
- Simplify PyTorch Ignite example by removing an evaluator. (971)

Tests

- Add `__init__.py` to `tests/samplers_tests/tpe_tests`. (945, thanks keisuke-umezawa!)
- Cast to `numpy.ndarray` as Python lists are no longer accepted in `xgboost==1.0`. (947)

Code Fixes

- Simplified `_check_distribution`. (937, thanks PhilipMay!)

Continuous Integration

- Remove version constraints of PyTorch and its related libraries. (893)
- Bump up pip version. (911)
- Fix sphinx version to avoid build error. (942)
- Apply Python3-style of type hinting to `optuna.integration.lightgbm_tuner.train`. (943)