Optuna

This is the release note of [v1.1.0](https://github.com/optuna/optuna/milestone/20?closed=1).

New Features

Experimental Hyperband

Hyperband, an extension of the successive halving pruning algorithm, has been introduced as an experimental feature through `HyperbandPruner`. It is compatible with `RandomSampler` and `TPESampler`. See 809.

Breaking Changes

`SuccessiveHalvingPruner` minimum resource heuristic

When `min_resource` is omitted, instead of defaulting to 1, the `SuccessiveHalvingPruner` now uses a heuristic to guess a suitable value. See 812.

Enhancements

- Set `pool_pre_ping=True` for MySQL to avoid connection errors. (806)
- Initial implementation of 'auto' for `min_resource` in `SuccessiveHalvingPruner`. (812)
- Update `TPESampler` to support `HyperbandPruner`. (828)
- Fix `XGBoostPruningCallback` to be compatible with `xgboost.cv`. (865, thanks yutayamazaki!)

Bug Fixes

- Make `PyTorchIgnitePruningHandler` report correct epoch. (847)
- Fix LightGBM tuner raises exception when `max_depth=-1`. (872)
- Sort intermediate values in `plot_intermediate_values`. (889)

Installation

- Update SciPy version. (838, thanks yasuyuky!)

Documentation

- Fix `KerasPruningCallback` pruning example code. (832, thanks harupy!)
- Fix docstrings of visualization functions. (833, thanks harupy!)
- Update a badge image for Python 3.8. (836)
- Add a note for JupyterLab users. (843)
- FAQ documentation bullet list. (856)
- Add to FAQ dynamically altering search spaces. (857)
- Elaborate on details of `HyperbandPruner`. (875)
- Update FAQ about trials raising exceptions and returning NaN. (880)
- Add Neptune to the list of external libraries which use Optuna. (890)
- Add OptGBM to the `examples/README.md`. (896)
- Add link to examples. (897)

Examples

- Add `--pruning` option to `pytorch_lightning_simple.py`. (794, thanks yutayamazaki!)
- Add FastAI example with `FastAIPruningCallback`. (848)
- Fix TensorFlow eager example with TensorFlow 2.X. (855, thanks nuka137!)
- Update the PyTorch Lightning example to support `pytorch_lightning==0.6.0`. (858)
- Add an example for LightGBM Tuner. (860, thanks jinnaiyuu!)
- Convert None to NaN before reporting to MLFlow. (863)
- Remove description about CLI execution of MLFlow example. (864)
- Fix TensorFlow estimator example with TensorFlow 2.X. (868, thanks nuka137!)
- Fix TensorFlow estimator pruning example with TensorFlow 2.X. (871)

Code Fixes

- Remove `typing` external dependency. (840, thanks hbredin!)
- RDB storage module level logger. (866)

Other

- Git ignore `.DS_Store`. (829)
- Require `torchvision>=0.5.0` with Pillow version fix. (839, thanks hugovk!)

This is the release note of v1.0.0. See [here](https://github.com/optuna/optuna/milestone/19?closed=1) for the complete list of solved issues and merged PRs.

Highlights

The first major version of Optuna v1.0 is released. It includes a stable API, significant performance improvements by reduced memory allocations and database storage accesses. Documentation has been revised and Jupyter Notebook examples are available on Google Colab. The visualization API has become customizable such that layouts can be modified.

Python 2 Support Drop
Due to the end-of-life (EOL) of Python 2 in January 1, 2020, Optuna has dropped the support for Python 2 and has now shifted to support Python 3.5.1 and above.

Blog
A developer blog has started. The [first post is about this release and future roadmaps](https://medium.com/optuna/optuna-v1-86192cd09be5).

New Features
- Use Joblib to parallelize optimization. (692, thanks AnesBenmerzoug!)
- `trials_dataframe` to allow returning `DataFrame` with flattened columns. (736)
- Make plot functions return Plotly graph object. (704)

Enhancements
- Add options to disable trials deep copy. (696)
- Slice plot to dynamically resize figure width depending on number of parameters. (720)
- Fix implementations of `__eq__`. (726)
- Speed up `study.best_trial` for RDB. (729, thanks chris-chris!)
- Use the preferred module level logger in `structs.py`. (768)
- Use `SourceFileLoader` instead of `imp` module. (778)
- Add warning about use of `InMemoryStorage` with multi-processing. (780)
- Avoid redundant sort (reverse) in successive halving. (807)

Bug Fixes
- Use default parameter when tuned parameter doesn't improve. (721)

Documents
- Introduce `doctest`. (702)
- Comment out notes for contributors in GitHub templates. (738, thanks crcrpar!)
- Add installation via `conda`. (739, thanks crcrpar!)
- Add badges to README. (740, thanks crcrpar!)
- First Contribution clarity edits. (744)
- Clarity edits to `configurations.rst`. (747)
- Add Gitter badge to README.md. (749)
- Replace Slack with Gitter in the issue template. (750)
- Setup `intersphinx`. (754)
- Follow Google Analytics Privacy Disclosure Policy. (776)
- Update example dataframe in tutorial. (784)
- Document `BasePruner` in docs. (788, thanks crcrpar!)
- Update docstring of `BasePruner.prune`. (793, thanks crcrpar!)
- Add favicon to doc. (796, thanks harupy!)
- Fix indentation in `tutorial/index.rst`. (816, thanks harupy!)
- Fix warnings in the documentation build. (819, thanks harupy!)
- Add integration modules list to README. (830)

Examples
- Add an example for `OptunaSearchCV`. (722, thanks yutayamazaki!)
- Add `scikit-image` example. (730, thanks tohmae!)
- Merge visualization examples. (735)
- Add "Open In Colab" badge to visualization example. (737, thanks crcrpar!)
- Add badge on README that launches starter notebook. (741, thanks harupy!)
- Consistent import of `optuna` in examples. (746)
- Remove Google Colab preinstalled library from `quickstart.ipynb`. (753)
- Refactor model definition of `examples/pytorch_simple.py`. (779, thanks crcrpar!)
- Fix Colab badge on visualization example notebook. (789, thanks harupy!)
- Add MLflow example. (795, thanks harupy!)
- Use callback to report results to MLflow. (799)
- Add README with gif to MLflow example. (802, thanks harupy!)
- Revise quickstart Jupyter notebook. (804)

Tests
- Stop passing step to `should_prune` in `chainermn`. (764, thanks crcrpar!)
- Remove a temporary workaround in the `test_callbacks` function. (769)
- Add lock acquisition to make `test_callbacks` test thread-safe. (773)
- Simplify `Study` unit test parameterizations. (774)
- Remove `cache_mode` argument to fix a broken test. (790)

Code Fixes
- Dissect `optuna/visualization.py`. (681, thanks crcrpar!)
- Refactor `setup.py`. (742)
- Fix deprecated property access. (751)
- Remove `StudySummary.__ne__`. (756)
- Remove redundant trailing commas. (757)
- Fix a typo in `RDBStorage`. (765)
- Change `logger` to study's module variable. (770, thanks crcrpar!)
- Separate standard library imports from third party imports. (777, thanks yutayamazaki!)
- Early commit after RDB session creation. (792)
- Refactoring of successive halving. (808)

Installation
- Remove `cython` from requirements. (781)
- Add version restriction of `scipy`. (801)
- Add version constraint of `scikit-learn` (`<=0.22.0`). (826)
- Add version constraint of `pillow` to avoid `torchvision`'s issue. (827)

Breaking Changes
- Prefix '_' to private functions in `optuna.dashboard`. (698)
- Prefix '_' to private classes and attributes in `optuna.samplers`. (699)
- Prefix '_' to private attributes of pruners. (703)
- Make plot functions return Plotly graph object. (704)
- Prefix '_' to private attributes of classes in `optuna.integration`. (705)
- Remove `enable_cache` option from `RDBStorage`. (706)
- Remove Python 2 version check from `integration.__init__.py` condition. (712)
- Remove `six`. (714)
- Remove `__future__`. (715)
- Make `bokeh-allow-websocket-origins` required. (716)
- Make `study_id` private. (718)
- Add `FrozenTrial.__lt__` to sort trials without key. (719)
- Remove version checks from `setup.py`. (724)
- `trials_dataframe` to allow returning `DataFrame` with flattened columns. (736)
- `BaseDistribution.__hash__` to take `__class__` into account. (743)
- Only allow `float`, `str` and castable to `float` in categorical distribution. (758)
- Improving usability of `TrialState` in trials dataframes. (771)
- Remove deprecated `product_search_space` function. (772)
- Add 'attrs' option to `Study.trials_dataframe()`. (775)

Continuous Integration
- Drop CI jobs for Python 2.7. (710)
- Add CI jobs for Python 3.8. (759)
- Merge with master branch in CI jobs. (791)

Show the best trials of multi-objective optimization and train a neural network with one of the best parameters.

console
$ STORAGE=sqlite:///example.db
$ STUDY_NAME=example-mo
$ optuna best-trials --storage $STORAGE --study-name $STUDY_NAME
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+
| number | values | datetime_start | datetime_complete | duration | params | state |
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+
| 0 | [0.23884292794146034, 0.6905832476748404] | 2021-10-01 15:02:32 | 2021-10-01 15:02:32 | 0:00:00.035815 | {'lr': 0.05318673615579818, 'optimizer': 'adam'} | COMPLETE |
| 2 | [0.3157886300888031, 0.05110976427394465] | 2021-10-01 15:02:32 | 2021-10-01 15:02:32 | 0:00:00.030019 | {'lr': 0.08044012012204389, 'optimizer': 'sgd'} | COMPLETE |
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+

$ optuna best-trials --storage $STORAGE --study-name $STUDY_NAME --format json > result.json
$ OPTIMIZER=`jq '.[0].params.optimizer' result.json`
$ LR=`jq '.[0].params.lr' result.json`
$ python train.py $OPTIMIZER $LR


See 2847 for more details.

Multi-objective Optimization Support of Weights & Biases and MLflow Integrations

Weights & Biases and MLflow integration modules support tracking multi-objective optimization. Now, they accept arbitrary numbers of objective values with metric names.

Weights & Biases

python
from optuna.integration import WeightsAndBiasesCallback

wandbc = WeightsAndBiasesCallback(metric_name=["mse", "mae"])

...

study = optuna.create_study(directions=["minimize", "minimize"])
study.optimize(objective, n_trials=100, callbacks=[wandbc])



MLflow

python
from optuna.integration import MLflowCallback

mlflc = MLflowCallback(metric_name=["accuracy", "latency"])

...

study = optuna.create_study(directions=["minimize", "minimize"])
study.optimize(objective, n_trials=100, callbacks=[mlflc])


See 2835 and 2863 for more details.

Breaking Changes

- Align CLI output format (2882)
- In particular, the return format of `optuna ask` has been simplified. The first layer of nesting with the key “trial” is removed. Parsing can be simplified from `jq ‘.trial.params’` to `jq ‘.params’`.

New Features

- Support multi-objective optimization in `WeightsAndBiasesCallback` (2835, thanks xadrianzetx!)
- Introduce trials CLI (2847)
- Support multi-objective optimization in `MLflowCallback` (2863, thanks xadrianzetx!)

Enhancements

- Add Plotly-like interpolation algorithm to `optuna.visualization.matplotlib.plot_contour` (2810, thanks xadrianzetx!)
- Sort values when the categorical values is numerical in `plot_parallel_coordinate` (2821, thanks TakuyaInoue-github!)
- Refactor `MLflowCallback` (2855, thanks xadrianzetx!)
- Minor refactoring of `plot_parallel_coordinate` (2856)
- Update `sklearn.py` (2966, thanks Garve!)

Bug Fixes

- Fix `datetime_complete` in `_CachedStorage` (2846)
- Hyperband no longer assumes it is the only pruner (2879, thanks cowwoc!)
- Fix method `untransform` of `_SearchSpaceTransform` with `distribution.single() == True` (2947, thanks yoshinobc!)

Installation

- Avoid `keras` 2.6.0 (2851)
- Drop `tensorflow` and `keras` version constraints (2852)
- Avoid latest `allennlp==2.7.0` (2894)
- Introduce the version constraint of scikit-learn (2953)

Documentation

- Fix `bounds`' shape in the document (2830)
- Simplify documentation of `FrozenTrial` (2833)
- Fix typo: replace CirclCI with CircleCI (2840)
- Added alternative callback function 2844 (2845, thanks DeviousLab!)
- Update URL of cmaes repository (2857)
- Improve the docstring of `MLflowCallback` (2883)
- Fix `create_trial` document (2888)
- Fix an argument in docstring of `_CachedStorage` (2917)
- Use `:obj:` for `True`, `False`, and `None` instead of inline code (2922)
- Use inline code syntax for `constraints_func` (2930)
- Add link to Weights & Biases example (2962, thanks xadrianzetx!)

Examples

- Do not use latest `keras==2.6.0` (https://github.com/optuna/optuna-examples/pull/44)
- Fix typo in Dask-ML GitHub Action workflow (https://github.com/optuna/optuna-examples/pull/45, thanks jrbourbeau!)
- Support Python 3.9 for TensorFlow and MLFlow (https://github.com/optuna/optuna-examples/pull/47)
- Replace deprecated argument `lr` with `learning_rate` in tf.keras (https://github.com/optuna/optuna-examples/pull/51)
- Avoid latest `allennlp==2.7.0` (https://github.com/optuna/optuna-examples/pull/52)
- Save checkpoint to tmpfile and rename it (https://github.com/optuna/optuna-examples/pull/53)
- PyTorch checkpoint cosmetics (https://github.com/optuna/optuna-examples/pull/54)
- Add Weights & Biases example (https://github.com/optuna/optuna-examples/pull/55, thanks xadrianzetx!)
- Use `MLflowCallback` in MLflow example (https://github.com/optuna/optuna-examples/pull/58, thanks xadrianzetx!)

Tests

- Fixed relational operator not including `1` (2865, thanks Yu212!)
- Add scenario tests for samplers (2869)
- Add test cases for storage upgrade (2890)
- Add test cases for `show_progress_bar` of `optimize` (2900, thanks xadrianzetx!)
- Speed-up sampler tests by using random sampling of skopt (2910)
- Fixes `namedtuple` type name (2961, thanks sobolevn!)

Code Fixes

- Changed y-axis and x-axis access according to matplotlib docs (2834, thanks 01-vyom!)
- Fix a BoTorch deprecation warning (2861)
- Relax metric name type hinting in `WeightsAndBiasesCallback` (2884, thanks xadrianzetx!)
- Fix recent `alembic` 1.7.0 type hint error (2887)
- Remove old unused `Trial._after_func` method (2899)
- Fixes `namedtuple` type name (2961, thanks sobolevn!)

Continuous Integration

- Enable act to run for other workflows (2656)
- Drop `tensorflow` and `keras` version constraints (2852)
- Avoid segmentation fault of `test_lightgbm.py` on macOS (2896)

Other

- Preinstall RDB binding Python libraries in Docker image (2818)
- Bump to v2.10.0.dev (2829)
- Bump to v2.10.0 (2975)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

01-vyom, Crissman, DeviousLab, Garve, HideakiImamura, TakuyaInoue-github, Yu212, c-bata, cowwoc, himkt, hvy, jrbourbeau, keisuke-umezawa, not522, nzw0301, sobolevn, toshihikoyanase, xadrianzetx, yoshinobc

number: 3
params:
x: A
y: 0
state: COMPLETE

This is the release note of v0.19.0. See [here](https://github.com/optuna/optuna/milestone/17?closed=1) for the complete list of solved issues and merged PRs.

Highlights
----------

The GitHub organization of this repository has been changed from `pfnet`, the organization for Preferred Networks, Inc. to `optuna` in order to widen the community, growing the project as an open source software.

[`optuna.exceptions`](https://optuna.readthedocs.io/en/latest/reference/exceptions.html#) has been introduced. Now, all exceptions defined in Optuna, including [`TrialPruned`](https://optuna.readthedocs.io/en/latest/reference/exceptions.html#optuna.exceptions.TrialPruned), have been moved out from `optuna.structs` to this new submodule. This is a part of a larger refactoring that we are currently working on to clean up the interfaces and make Optuna even easier to use. _To avoid breaking existing code however, `optuna.structs.TrialPruned` is still available but marked as deprecated._

Upcoming Python 2 Support Drop
----------------------------
Due to the end-of-life (EOL) of Python 2 in January 2020, Optuna will drop Python 2 support in December 2019.
This decision was made considering the following facts:
- Python 2 goes end-of-life (EOL) in [January 2020](https://www.python.org/dev/peps/pep-0373/#maintenance-releases).
- Many scientific computation packages, including NumPy, which is one of the core dependency of Optuna, are [planning or already started to drop support for Python 2](https://python3statement.org/).
We plan to drop Python 2 support in the first release in December 2019.

Compatibility
-------------
- Disallow negative step numbers in `Trial.report`. (701)
- Disallow erroneous arguments to `PercentilePruner`. (693)
- Extract exception classes to `optuna.exceptions` module. (691)
- Make `{FrozenTrial,Trial}.trial_id` private. (663)

New Features
------------
- Allow specifying step interval for `{Median,Percentile}Pruner`. (660)

Enhancements
------------

- Make distribution classes single inheritance. (573)

Bug Fixes
---------
- Fix the handling of `time_budget` option on `lightgbm_tuner.train()`. (684, thanks momijiame!)

Documents
---------
- Improve documentation of `Trial.should_prune`. (690)
- Expose `RDBStorage` constructor. (689)
- Add a note about type casting of reported values to `trial.report()` doc. (687)
- Add punctuation to copyright in docs. (682)
- Replace the company name with `Optuna Contributors` in the docs. (676)
- Update the documentation of `Trial` class. (668)
- Add Code of Conduct. (675, thanks Crissman!)

Examples
--------
- Improve visualization examples. (667, thanks Crissman!)
- Update PyTorch Lightning example for `pytorch-lightning==0.5.3`. (700)

Tests
-----
- Add more `optuna.visualization` tests. (630, thanks crcrpar!)
- Remove the version constraint for `pytorch_lightning`. (694)

This is the release note of v0.18.1. See [here](https://github.com/pfnet/optuna/milestone/18?closed=1) for the complete list of solved issues and merged PRs.

Bug Fixes
---------
- Fix `import optuna` failure on Python2.7 environment without LightGBM installed. (674)

Examples
--------
- Change subsample's low value from 0 to 0.1 in the catboost example. (678)