Optuna

This is the release note of [v3.0.0-rc0](https://github.com/optuna/optuna/milestone/44?closed=1).
**_This is a release candidate of Optuna V3. We plan to release the major version within a few weeks. Please try this version and report bugs!_**

Highlights

Constrained Optimization Support for TPE
`TPESampler`, the default sampler of Optuna, now supports constrained optimization. It takes a function `constraints_func` as an argument, and examines whether trials are feasible or not. Feasible trials are prioritized over infeasible ones similarly to `NSGAIISampler`. See [3506](https://github.com/optuna/optuna/pull/3506) for more details.

python
def objective(trial):
Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)

Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))

v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2

return v0, v1

def constraints(trial):
return trial.user_attrs["constraints"]

if __name__ == "__main__":
sampler = optuna.samplers.TPESampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)

optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()


| MOTPE without constraints | MOTPE with constraints |
| - | - |
|  |  |


A Major Refactoring of Visualization Module

We have undertaken major refactoring of the visualization features as one of the major tasks of Optuna V3. The current situation is as follows.

Unification of implementations of different backends: `plotly` and `matplotlib`

Historically, the implementations of Optuna's visualization features were split between two different backends, `plotly` and `matplotlib`. Many of these implementations were duplicated and unmaintainable, and many were implemented as a single large function, resulting in poor testability and, as a result, becoming the cause of many bugs. We clarified the specifications that each visualization function in Optuna must meet and defined the backend-independent information needed to perform the visualization. By using this information commonly across different backends, we achieved a highly maintainable and testable implementation, and improved the stability of the visualization functions dramatically. We are currently rewriting the unit tests, and the resulting tests will be simple yet powerful.

Visual Regression Test

It is very important to detect hidden bugs in the implementation through PR reviews. However, visualizations are likely to contain bugs that are difficult to find just by reading the code, and many of these bugs are only revealed when the visualization is actually performed. Therefore, we introduced the Visual Regression Test to improve the review process. In the PR for visualization features, you can jump to the Visual Regression Test link by clicking on the link generated from within the PR. Reviewers can verify that the PR implementation is performing the visualization properly.

<img width="1715" alt="173838319-24433136-bd59-47d5-afdb-2694aafe354d (1)" src="https://user-images.githubusercontent.com/38826298/183350545-bbe74c09-4721-461b-b3b5-cdba5092403f.png">

Improve Code Quality Including Many Bugfix

In the latter development cycle of Optuna v3, we put emphasis on improving the overall code quality of the library. We fixed several bugs and possible corruption of internal data structures on e.g. handling Inf/NaN values (3567, 3592, 3738, 3739, 3740) and invalid inputs (3668, 3808, 3814, 3819). For example, there had been bugs before v3 when NaN values were used in a `CategoricalDistribution` or `GridSampler`. In several other functions, NaN values were unacceptable but the library failed silently without any warning or error. Such bugs are fixed in this release.


New Features

- Add constraints option to `TPESampler` (3506)
- Add `skip_if_exists` argument to `enqueue_trial` (3629)
- Remove experimental from `plot_pareto_front` (3643)
- Add `popsize` argument to `CmaEsSampler` (3649)
- Add `seed` argument for `BoTorchSampler` (3756)
- Add `seed` argument for `SkoptSampler` (3791)
- Revert AllenNLP integration back to experimental (3822)

Enhancements

- Move `is_heartbeat_enabled` from storage to heartbeat (3596)
- Refactor `ImportanceEvaluators` (3597)
- Avoid maximum limit when MLflow saves information (3651)
- Control metric decimal digits precision in `bayesmark` benchmark report (3693)
- Support `inf` values for crowding distance (3743)
- Normalize importance values (3828)

Bug Fixes

- Fix `CategoricalDistribution` with NaN (3567)
- Fix NaN comparison in grid sampler (3592)
- Fix bug in `IntersectionSearchSpace` (3666)
- Remove `trial_values` records whose values are `None` (3668)
- Fix PostgreSQL primary key unsorted problem (3702, thanks wattlebirdaz!)
- Raise error on NaN in `_constrained_dominates` (3738)
- Fix `inf`-related issue on implementation of `_calculate_nondomination_rank` (3739)
- Raise errors for NaN in constraint values (3740)
- Fix `_calculate_weights` such that it throws `ValueError` on invalid weights (3742)
- Change warning for `axis_order` of `plot_pareto_front` (3802)
- Fix check for number of objective values (3808)
- Raise `ValueError` when waiting trial is told (3814)
- Fix `Study.tell` with invalid values (3819)
- Fix infeasible case in NSGAII test (3839)

Installation

- Add a version constraint of cached-path (3665)

Documentation

- Add documentation of SHAP integration (3623)
- Remove news entry on Optuna user survey (3645)
- Introduce `optuna-fast-fanova` (3647)
- Add github discussions link (3660)
- Fix a variable name of ask-and-tell tutorial (3663)
- Clarify which trials are used for importance evaluators (3707)
- Fix typo in `Study.optimize` (3720, thanks 29Takuya!)
- Update link to plotly's jupyterlab-support page (3722, thanks 29Takuya!)
- Update `CONTRIBUTING.md` (3726)
- Remove "Edit on Github" button (3777, thanks cfkazu!)
- Remove duplicated period at the end of copyright (3778)
- Add note for deprecation of `plot_pareto_front`'s `axis_order` (3803)
- Describe the purpose of `prepare_study_with_trials` (3809)
- Fix a typo in docstring of `ShapleyImportanceEvaluator` (3810)
- Add a reference for MOTPE (3838, thanks y0z!)

Examples

- Introduce stale bot (https://github.com/optuna/optuna-examples/pull/119)
- Use Hydra 1.2 syntax (https://github.com/optuna/optuna-examples/pull/122)
- Fix CI due to `thop` (https://github.com/optuna/optuna-examples/pull/123)
- Hotfix `allennlp` dependency (https://github.com/optuna/optuna-examples/pull/124)
- Remove unreferenced variable in `pytorch_simple.py` (https://github.com/optuna/optuna-examples/pull/125)
- set `OMPI_MCA_rmaps_base_oversubscribe=yes` before `mpirun` (https://github.com/optuna/optuna-examples/pull/126)

Tests

- Simplify multi-objective TPE tests (3653)
- Add edge cases to multi-objective TPE tests (3662)
- Remove tests on `TypeError` (3667)
- Add edge cases to the tests of the parzen estimator (3673)
- Add tests for `_constrained_dominates` (3683)
- Refactor tests of constrained TPE (3689)
- Add `inf` and NaN tests for `test_constraints_func` (3690)
- Fix calling storage API in study tests (3695, thanks wattlebirdaz!)
- DRY `test_frozen.py` (3696)
- Unify the tests of `plot_contour`s (3701)
- Add test cases for crossovers of NSGAII (3705)
- Enhance the tests of `NSGAIISampler._crowding_distance_sort` (3706)
- Unify edf test files (3730)
- Fix `test_calculate_weights_below` (3741)
- Refactor `test_intermediate_plot.py` (3745)
- Test samplers are reproducible (3757)
- Add tests for `_dominates` function (3764)
- DRY importance tests (3785)
- Move tests for `create_trial` (3794)
- Remove `with_c_d` option from `prepare_study_with_trials` (3799)
- Use `DeterministicRelativeSampler` in `test_trial.py` (3807)

Code Fixes

- Add typehint for deprecated and experimental (3575)
- Move heartbeat-related thread operation in `_optimize.py` to `_heartbeat.py` (3609)
- Remove useless object inheritance (3628, thanks harupy!)
- Remove useless `except` clauses (3632, thanks harupy!)
- Rename `optuna.testing.integration` with `optuna.testing.pruner` (3638)
- Cosmetic fix in Optuna CLI (3641)
- Enable `strict_equality` for `mypy` 3579 (3648, thanks wattlebirdaz!)
- Make file names in testing consistent with `optuna` module (3657)
- Remove the implementation of `read_trials_from_remote_storage` in the all storages apart from `CachedStorage` (3659)
- Remove unnecessary deep copy in Redis storage (3672, thanks wattlebirdaz!)
- Workaround `mypy` bug (3679)
- Unify `plot_contour`s (3682)
- Remove `storage.get_all_study_summaries(include_best_trial: bool)` (3697, thanks wattlebirdaz!)
- Unify the logic of edf functions (3698)
- Unify the logic of `plot_param_importances` functions (3700)
- Enable `disallow_untyped_calls` for `mypy` (3704, thanks 29Takuya!)
- Use `get_trials` with `states` argument to filter trials depending on trial state (3708)
- Return Python's native float values (3714)
- Simplify `bayesmark` benchmark report rendering (3725)
- Unify the logic of intermediate plot (3731)
- Unify the logic of slice plot (3732)
- Unify the logic of `plot_parallel_coordinate`s (3734)
- Unify implementation of `plot_optimization_history` between `plotly` and `matplotlib` (3736)
- Extract `fail_objective` and `pruned_objective` for tests (3737)
- Remove deprecated storage functions (3744, thanks 29Takuya!)
- Remove unnecessary optionals from `visualization/_pareto_front.py` (3752)
- Change types inside `_ParetoInfoType` (3753)
- Refactor pareto front (3754)
- Use `_ContourInfo` to plot in `plot_contour` (3755)
- Follow up 3465 (3763)
- Refactor importances plot (3765)
- Remove `no_trials` option of `prepare_study_with_trials` (3766)
- Follow the coding style of comments in `plot_contour` files (3767)
- Raise `ValueError` for invalid returned type of `target` in `_filter_nonfinite` (3768)
- Fix value error condition in `plot_contour` (3769)
- DRY constraints in `Sampler.after_trial` (3775)
- DRY `stop_objective` (3786)
- Refactor non-exist param test in `plot_contour` test (3787)
- Remove `less_than_two` and `more_than_three` options from `prepare_study_with_trials` (3789)
- Fix return value's type of `_get_node_value` (3818)
- Remove unused `type: ignore` (3832)
- Fix typos and remove unused argument in `QMCSampler` (3837)

Continuous Integration

- Use `coverage` directly (3347, thanks higucheese!)
- Install 3rd party libraries in CI for lint (3580)
- Make `bayesmark` benchmark results comparable to `kurobako` (3584)
- Enable `warn_unused_ignores` for `mypy` (3627, thanks harupy!)
- Add `onnx` and version constrained `protobuf` to document dependencies (3658)
- Add `mo-kurobako` benchmark to CI (3691)
- Enable mypy's strict configs (3710)
- Run visual regression tests to find regression bugs of visualization module (3721)
- Remove downloading old `libomp` for mac tests (3728)
- Match Python versions between `bayesmark` CI jobs (3750)
- Set `OMPI_MCA_rmaps_base_oversubscribe=yes` before `mpirun` (3758)
- Add `budget` option to benchmarks (3774)
- Add `n_concurrency` option to benchmarks (3776)
- Use `n-runs` instead of `repeat` to represent the number of studies in the bayesmark benchmark (3780)
- Fix type hints for `mypy 0.971` (3797)
- Pin scipy to avoid the CI failure (3834)
- Extract float value from tensor for `trial.report` in `PyTorchLightningPruningCallback` (3842)

Other

- Clarify the criteria to assign reviewers in the PR template (3619)
- Bump up version number to v3.0.0rc0.dev (3621)
- Make `tox.ini` consistent with `checking` (3654)
- Avoid to stale description-checked issues (3816)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

29Takuya, HideakiImamura, c-bata, cfkazu, contramundum53, g-votte, harupy, higucheese, himkt, hvy, keisuke-umezawa, knshnb, not522, nzw0301, sile, toshihikoyanase, wattlebirdaz, xadrianzetx, y0z

This is the release note of [v3.0.0-b1](https://github.com/optuna/optuna/milestone/43?closed=1).

Highlights

A Samplers Comparison Table
We added a sampler comparison table on [the samplers' documentation page](https://optuna.readthedocs.io/en/latest/reference/samplers/index.html). It includes supported options (parameter types, pruning, multi-objective optimization, constrained optimization, etc.), time complexity, and recommended budgets for each sampler. Please use this to select appropriate samplers for your tasks! See #3571 and 3593 for more details.

<img width="1224" alt="sampler_comparison_table" src="https://user-images.githubusercontent.com/38826298/172111648-eb56206f-8539-48ac-a0ab-b1cc199e2d22.png">

A New Importance Evaluator: `ShapleyImportanceEvaluator`

Optuna now supports mean absolute SHAP value for evaluating parameter importances through integration with the [SHAP library](https://github.com/slundberg/shap). SHAP value is a game-theoretic measure of parameter importance featuring nice theoretical properties (See [paper](https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf) for more information).

<img width="469" alt="168213146-465a8116-94f2-49c9-b4ce-ec12970d82f8" src="https://user-images.githubusercontent.com/38826298/172112327-cda4a831-e1aa-486d-a481-68a44b14637c.png">

To use mean absolute SHAP importances, an object of `optuna.integration.shap.ShapleyImportanceEvaluator` can be passed to `evaluator` argument in `optuna.visualization.plot_param_importances` or `optuna.importance.get_param_importances`.


import optuna
from optuna.integration.shap import ShapleyImportanceEvaluator

study = optuna.create_study()
study.optimize(objective, n_trials=100)

optuna.visualization.plot_param_importances(study, evaluator=ShapleyImportanceEvaluator())


See the 3507 for more details.

A New Benchmarking Task

The benchmarking environment for black-box optimization algorithms on the GitHub Actions was introduced in the previous release. We have further enhanced its capabilities. The benchmarking functionality introduced can be run on all users' forks using GitHub Actions. You can also freely customize and run benchmarks on more computationally powerful clusters, for example, AWS, using the code in the `optuna/benchmarks` directory.

Neural Architecture Search Benchmark Support

Optuna's algorithms can now be benchmarked using NASLib, the Neural Architecture Search benchmark library. For now we only support one dataset, NASBench 201, which deals with image recognition. Larger datasets and datasets from other areas such as natural language processing will be supported in the future.

| cifar10 | cifar100 | imagenet16-120|
| ---- | ---- | ---- |
|  |  |  |

See [README](https://github.com/optuna/optuna/blob/master/benchmarks/README.md#performance-benchmarks-with-naslib) and 3465 for more information.

Multi-objective Optimization Benchmark Support

We are now able to benchmark our multi-objective optimization algorithms. They are not yet available on GitHub Actions, but you can use `optuna/benchmarks/run_mo_kurobakmo.py` directly. They will be available on GitHub Actions in the next release, so stay tuned! See 3271 and 3349 for more details.

Python 3.10 Support

This is the first version to officially support Python 3.10. All tests are passed including integration modules, with a few exceptions.

Storage Database Migration

To use Optuna v3.0.0-b1 with `RDBStorage` that was created in the previous versions of Optuna, please run `optuna storage upgrade` to migrate your database.

sh
`YOUR_RDB_URL` is the URL of your database.
optuna storage upgrade –storage YOUR_RDB_URL


If you use `RedisStorage`, copy your study with `RDBStorage` using [`copy_study`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.copy_study.html?highlight=copy_study) with the Optuna you used to create the study, thenrun `optuna storage upgrade` with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new `RedisStorage`.

sh
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’




Breaking Changes

- Fix distribution compatibility for linear and logarithmic distribution (3444)
- Remove `get_study_id_from_trial_id` (3538)

New Features

- Add `targets` argument to `plot_pareto_plont` of `plotly` backend (3495, thanks TakuyaInoue-github!)
- Support `constraints_func` in `plot_pareto_front` in matplotlib visualization (3497, thanks fukatani!)
- Calculate the feature importance with mean absolute SHAP values (3507, thanks liaison!)
- Make `GridSampler` reproducible (3527, thanks gasin!)
- Replace `ValueError` with `warning` in `GridSearchSampler` (3545)
- Implement `callbacks` argument of `OptunaSearchCV` (3577)
- Add option to skip table creation to `RDBStorage` (3581)

Enhancements

- Set `precision` of `sqlalchemy.Float` in `RDBStorage` table definition (3327)
- Accept `nan` in `trial.report` (3348, thanks belldandyxtq!)
- Lazy import of alembic, sqlalchemy, and scipy (3381)
- Unify pareto front (3389, thanks semiexp!)
- Make `set_trial_param()` of `RedisStorage` faster (3391, thanks masap!)
- Make `_set_best_trial()` of `RedisStorage` faster (3392, thanks masap!)
- Make `set_study_directions()` of `RedisStorage` faster (3393, thanks masap!)
- Make optuna compatible with wandb sweep panels (3403, thanks captain-pool!)
- Change "Trials" to "Trial" in `plot_slice`, `plot_pareto_front`, and `plot_optimization_history` (3449, thanks dubey-anshuman!)
- Make contour plots handle trials with nonfinite values (3451)
- Query studies for trials only once in EDF plots (3460)
- Make Parallel-Coordinate plots handle trials with nonfinite values (3471, thanks divyanshugit!)
- Separate heartbeat functionality from `BaseStorage` (3475)
- Remove `torch.distributed` calls from `TorchDistributedTrial` properties (3490, thanks nlgranger!)
- Remove the internal logic that calculates the interaction of two or more variables in fANOVA (3543)
- Handle inf/-inf for `trial_values` table in RDB (3559)
- Add `intermediate_value_type` column to represent inf/-inf on `RDBStorage` (3564)

Bug Fixes

- Import `COLOR_SCALE` inside import util context (3492)
- Remove `-v` option of `optuna study set-user-attr` command (3499, thanks nyanhi!)
- Filter trials with nonfinite value in `optuna.visualization.plot_param_importances` and `optuna.visualization.matplotlib.plot_param_importance` (3500, thanks takoika!)
- Fix `--verbose` and `--quiet` options in CLI (3532, thanks nyanhi!)
- Replace `ValueError` with `RuntimeError` in `get_best_trial` (3541)
- Take the same search space as in `CategoricalDistribution` by `GridSampler` (3544)

Installation

- Partially support Python 3.10 (3353)
- Clean up `setup.py` (3517)
- Remove duplicate requirements from `document` section (3613)

Documentation

- Clean up exception docstrings (3429)
- Revise docstring in MLFlow and WandB callbacks (3477)
- Change the parameter name from `classifier` to `regressor` in the code snippet of `README.md` (3481)
- Add link to Minituna in `CONTRIBUTING.md` (3482)
- Fix `benchmarks/README.md` for the `bayesmark` section (3496)
- Mention `Study.stop` as a criteria to stop creating trials in document (3498, thanks takoika!)
- Fix minor English errors in the docstring of `study.optimize` (3505)
- Add Python 3.10 in supported version in `README.md` (3508)
- Remove articles at the beginning of sentences in crossovers (3509)
- Correct `FronzenTrial`'s docstring (3514)
- Mention specify hyperparameter tutorial (3515)
- Fix typo in MLFlow callback (3533)
- Improve docstring of `GridSampler`'s seed option (3568)
- Add the samplers comparison table (3571)
- Replace `youtube.com` with `youtube-nocookie.com` (3590)
- Fix time complexity of the samplers comparison table (3593)
- Remove `language` from docs configuration (3594)

Examples

- Fix version of JAX (https://github.com/optuna/optuna-examples/pull/99)
- Remove constraints by 99 (https://github.com/optuna/optuna-examples/pull/100)
- Replace some methods in the `sklearn` example (https://github.com/optuna/optuna-examples/pull/102, thanks MasahitoKumada!)
- Add Python3.10 in `allennlp.yml` (https://github.com/optuna/optuna-examples/pull/104)
- Remove numpy (https://github.com/optuna/optuna-examples/pull/105)
- Add python 3.10 to fastai CI (https://github.com/optuna/optuna-examples/pull/106)
- Add python 3.10 to non-integration examples CIs (https://github.com/optuna/optuna-examples/pull/107)
- Add python 3.10 to Hiplot CI (https://github.com/optuna/optuna-examples/pull/108)
- Add a comma to `visualization.yml` (https://github.com/optuna/optuna-examples/pull/109)
- Rename WandB example to follow naming rules (https://github.com/optuna/optuna-examples/pull/110)
- Add scikit-learn version constraint for Dask-ML (https://github.com/optuna/optuna-examples/pull/112)
- Add python 3.10 to sklearn CI (https://github.com/optuna/optuna-examples/pull/113)
- Set version constraint of `protobuf` in PyTorch Lightning example (https://github.com/optuna/optuna-examples/pull/116)

Tests

- Improve `matplotlib` parallel coordinate test (3368)
- Save figures for all `matplotlib` tests (3414, thanks divyanshugit!)
- Add `inf` test to intermediate values test (3466)
- Add test cases for `test_storages.py` (3480)
- Improve the tests of `optuna.visualization.plot_pareto_front` (3546)
- Move heartbeat-related tests in `test_storages.py` to another file (3553)
- Use `seed` method of `np.random.RandomState` for reseeding and fix `test_reseed_rng` (3569)
- Refactor `test_get_observation_pairs` (3574)
- Add tests for `inf/nan` objectives for `ShapleyImportanceEvaluator` (3576)
- Add deprecated warning test to the multi-objective sampler test file (3601)

Code Fixes

- Ignore incomplete trials in `matplotlib.plot_parallel_coordinate` (3415)
- Update warning message and add a test when a trial fails with exception (3454)
- Remove old distributions from NSGA-II sampler (3459)
- Remove duplicated DB access in `_log_completed_trial` (3551)
- Reduce the number of `copy.deepcopy()` calls in `importance` module (3554)
- Remove duplicated `check_trial_is_updatable` (3557)
- Replace `optuna.testing.integration.create_running_trial` with `study.ask` (3562)
- Refactor `test_get_observation_pairs` (3574)
- Update label of feasible trials if `constraints_func` is specified (3587)
- Replace unused variable name with underscore (3588)
- Enable `no-implicit-optional` for `mypy` (3599, thanks harupy!)
- Enable `warn_redundant_casts` for `mypy` (3602, thanks harupy!)
- Refactor the type of value of `TrialIntermediateValueModel` (3603)
- Fix broken `mypy` checks of Alembic's `get_current_head()` method (3608)
- Sort dependencies by name (3614)

Continuous Integration

- Introduce the benchmark for multi-objectives samplers (3271, thanks drumehiron!)
- Add WFG benchmark test (3349, thanks kei-mo!)
- Add workflow to use `reviewdog` (3357)
- Add NASBench201 from NASLib (3465)
- Fix speed benchmarks CI (3470)
- Support PyTorch 1.11.0 (3510)
- Restore `virtualenv` for benchmark extras (3585)
- Use `protobuf<4.0.0` to resolve Sphinx CI error (3591)
- Unpin `protobuf` (3598, thanks harupy!)
- Extract MPI tests from integration CI as independent CI (3606)

Other

- Bump up version to v3.0.0b1.dev (3457)
- Fix `kurobako` benchmark code to run it locally (3468)
- Fix label of issue template (3493)
- Improve issue templates (3536)
- Hotfix for `fakeredis` 1.7.4 release (3549)
- Remove the version constraint of `fakeredis` (3561)
- Relax version constraint of `fakeredis` (3607)
- Shorten the durations of the stale bot for PRs (3611)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, MasahitoKumada, TakuyaInoue-github, belldandyxtq, c-bata, captain-pool, contramundum53, divyanshugit, drumehiron, dubey-anshuman, fukatani, g-votte, gasin, harupy, himkt, hvy, kei-mo, keisuke-umezawa, knshnb, liaison, masap, nlgranger, not522, nyanhi, nzw0301, semiexp, sile, takoika, toshihikoyanase, xadrianzetx

This is the release note of [v3.0.0-b0](https://github.com/optuna/optuna/milestone/41?closed=1).

Highlights

Simplified Distribution Classes: Float, Int and Categorical

Search space definitions, which consist of `BaseDistribution` and its child classes in Optuna, are greatly simplified. We have introduced `FloatDistribution`, `IntDistribution`, and `CategoricalDistribution`. If you use the suggest API and `Study.optimize`, the search space information is stored as these three distributions. Previous `UniformDistribution`, `LogUniformDistribution`, `DiscreteUniformDistribution`, `IntUniformDistribution`, and `IntLogUniformDistribution` are deprecated. If you pass deprecated distributions to APIs such as `Study.ask` or `create_trial`, they are internally converted to corresponding `FloatDistribution` or `IntDistribution`.

Storage Database Migration

To use Optuna v3.0.0-b0 with `RDBStorage` that was created in the previous versions of Optuna, please run `optuna storage upgrade` to migrate your database.

If you use `RedisStorage`, copy your study with `RDBStorage` using [`copy_study`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.copy_study.html?highlight=copy_study) with the Optuna you used to create the study, thenrun `optuna storage upgrade` with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new `RedisStorage`.

sh
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’


Consistent Ask-and-Tell Interface with `Study.optimize`

`Study.tell` fails a trial when it is called with certain invalid combinations of `state` and `values`, instead of raising an error. This change aims to make `Study.tell` consistent with `Study.optimize`, which continues an optimization even if an objective returns an invalid value.

`Study.tell` now also returns the resulting trial (`FrozenTrial`) in order to allow inspecting how the arguments were interpreted.

Before

`Study.tell` raises an exception when it is called with an invalid combination of `state` and `values`.

python
study.tell(study.ask(), values=None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/…/optuna/optuna/study/study.py", line 579, in tell
raise ValueError(
ValueError: No values were told. Values are required when state is TrialState.COMPLETE.


After

`Study.tell` automatically fails the trial.

python
trial: FrozenTrial = study.tell(study.ask(), value=None)
assert trial.state == TrialState.FAIL


See 3144 for more details.

Stable `Study` APIs

We are converting all positional arguments of `create_study`, `delete_study`, `load_study`, and `copy_study` to keyword-only arguments since the order of arguments were inconsistent. This is not yet a breaking-change, but if you use these features with positional arguments, then you will get a warning message to use them with keyword-only arguments.

In addition, we have fixed all of problems described in 2955, so we have stabled the `Study` APIs. Specifically, `Study.add_trial`, `Study.add_trials`, `Study.enqueue_trial`, and `copy_study` have been stabled.

See 3270 and 2955 for more details.

Improved Visualization

Several bugs in the visualization module have been resolved. For instance,
the parallel coordinates plot ignores trials with missing parameters ([3373](https://github.com/optuna/optuna/pull/3373)) and the scale of the objective value is fixed ([#3369](https://github.com/optuna/optuna/pull/3369)). The edf plot filters trials with `inf` values ([#3395](https://github.com/optuna/optuna/pull/3395) and [#3435](https://github.com/optuna/optuna/pull/3435)).

Before: Trials with missing parameters are wrongly connected to each other.
<img width="1203" alt="158495812-acb399e5-d817-4cae-8c8b-c69bcc91efea" src="https://user-images.githubusercontent.com/38826298/162878578-fc162708-5c3b-45a0-b098-47445a1242a7.png">

After: Trials with missing parameters are removed from the plot.
<img width="1191" alt="158495810-98a5ec29-581c-426a-a255-11d16ae1c144" src="https://user-images.githubusercontent.com/38826298/162878606-aa937868-b8f5-4b51-b09b-6aced6fd28ff.png">

Breaking Changes

- Add option to exclude best trials from study summaries (3109)
- Migrate to `{Float,Int}Distribution` using `alembic` (3113)
- Move validation logic from `_run_trial` to `study.tell` (3144)
- Enable `FloatDistribution` and `IntDistribution` (3246)
- Use an enqueued parameter that is out of range from suggest API (3298)
- Convert deprecated distribution to new distribution internally (3420)

New Features

- Add `CatBoostPruningCallback` (2734, thanks tohmae!)
- Create common API for all NSGA-II crossover operations (3221)
- Add a history of retried trial numbers in `Trial.system_attrs` (3223, thanks belltailjp!)
- Convert all positional arguments to keyword-only (3270, thanks higucheese!)
- Stabilize `study.py` (3309)
- Add `targets` and deprecate `axis_order` in `optuna.visualization.matplotlib.plot_pareto_front` (3341, thanks shu65!)

Enhancements

- Enable cache for `study.tell()` (3265, thanks masap!)
- Warn if heartbeat is used with ask-and-tell (3273)
- Make `optuna.study.get_all_study_summaries()` of `RedisStorage` fast (3278, thanks masap!)
- Improve Ctrl-C interruption handling (3374, thanks CorentinNeovision!)
- Use same colormap among `plotly` visualization methods (3376)
- Make EDF plots handle trials with nonfinite values (3435)
- Make logger message optional in `filter_nonfinite` (3438)

Bug Fixes

- Make TPE work with a categorical variable with different choice types (3190, thanks keisukefukuda!)
- Fix axis range issue in `matplotlib` contour plot (3249, thanks harupy!)
- Allow `fail_state_trials` show warning when heartbeat is enabled (3301)
- Clip untransformed values sampled from int uniform distributions (3319)
- Fix missing `user_attrs` and `system_attrs` in study summaries (3352)
- Fix objective scale in parallel coordinate of Matplotlib (3369)
- Fix `matplotlib.plot_parallel_coordinate` with log distributions (3371)
- Fix parallel coordinate with missing value (3373)
- Add utility to filter trials with `inf` values from visualizations (3395)
- Return the best trial number, not worst trial number by `best_index_` (3410)
- Avoid using `px.colors.sequential.Blues` that introduces `pandas` dependency (3422)
- Fix `_is_reverse_scale` (3424)

Installation

- Drop TensorFlow support for Python 3.6 (3296)
- Pin AllenNLP version (3367)
- Skip run `fastai` job on Python 3.6 (3412)
- Avoid latest `click==8.1.0` that removed a deprecated feature (3413)
- Avoid latest PyTorch lightning until integration is updated (3417)
- Revert "Avoid latest `click==8.1.0` that removed a deprecated feature" (3430)

Documentation

- Add reference to tutorial page in CLI (3267, thanks tsukudamayo!)
- Carry over notes on `step` behavior to new distributions (3276)
- Correct the disable condition of `show_progress_bar` (3287)
- Add a document to lead FAQ and example of heartbeat (3294)
- Add a note for `copy_study`: it creates a copy regardless of its state (3295)
- Add note to recommend Python 3.8 or later in documentation build with artifacts (3312)
- Fix crossover references in `Raises` doc section (3315)
- Add reference to `QMCSampler` in tutorial (3320)
- Fix layout in tutorial (with workaround) (3322)
- Scikit-learn required for `plot_param_importances` (3332, thanks ll7!)
- Add a link to multi-objective tutorial from a pareto front page (3339, thanks kei-mo!)
- Add reference to tutorial page in visualization (3340, thanks Hiroyuki-01!)
- Mention tutorials of User-Defined Sampler/Pruner from the API reference pages (3342, thanks hppRC!)
- Add reference to saving/resuming study with RDB backend (3345, thanks Hiroyuki-01!)
- Fix a typo (3360)
- Remove deprecated command `optuna study optimize` in FAQ (3364)
- Fix nit typo (3380)
- Add see also section for `best_trial` (3396, thanks divyanshugit!)
- Updates the tutorial page for re-use the best trial (3398, thanks divyanshugit!)
- Add explanation about `Study.best_trials` in multi-objective optimization tutorial (3443)

Examples

- Remove Python 3.6 from `haiku`'s CI (https://github.com/optuna/optuna-examples/pull/83)
- Apply `black` 22.1.0 & run `checks` daily (https://github.com/optuna/optuna-examples/pull/84)
- Add `hiplot` example (https://github.com/optuna/optuna-examples/pull/86)
- Stop running jobs using TF with Python3.6 (https://github.com/optuna/optuna-examples/pull/87)
- Pin AllenNLP version (https://github.com/optuna/optuna-examples/pull/89)
- Add Medium link (https://github.com/optuna/optuna-examples/pull/91)
- Use official `CatBoostPruningCallback` (https://github.com/optuna/optuna-examples/pull/92)
- Stop running `fastai` job on Python 3.6 (https://github.com/optuna/optuna-examples/pull/93)
- Specify Python version using `str` in workflow files (https://github.com/optuna/optuna-examples/pull/95)
- Introduce upper version constraint of PyTorchLightning (https://github.com/optuna/optuna-examples/pull/96)
- Update `SimulatedAnnealingSampler` to support `FloatDistribution` (https://github.com/optuna/optuna-examples/pull/97)

Tests

- Add plot value tests to `matplotlib_tests/test_param_importances` (3180, thanks belldandyxtq!)
- Make tests of `plot_optimization_history` methods consistent (3234)
- Add integration test for `RedisStorage` (3258, thanks masap!)
- Change the order of arguments in the `catalyst` integration test (3308)
- Cleanup `MLflowCallback` tests (3378)
- Test serialize/deserialize storage on parametrized conditions (3407)
- Add tests for parameter of 'None' for TPE (3447)

Code Fixes

- Switch to `IntDistribution` (3181, thanks nyanhi!)
- Fix type hints for Python 3.8 (3240)
- Remove `UniformDistribution`, `LogUniformDistribution` and `DiscreteUniformDistribution` code paths (3275)
- Merge `set_trial_state()` and `set_trial_values()` into one function (3323, thanks masap!)
- Follow up for `{Float, Int}Distributions` (3337, thanks nyanhi!)
- Move the `get_trial_xxx` abstract functions to base (3338, thanks belldandyxtq!)
- Update type hints of `states` (3359, thanks BasLaa!)
- Remove unused function from `RedisStorage` (3394, thanks masap!)
- Remove unnecessary string concatenation (3406)
- Follow coding style and fix typos in `tests/integration_tests` (3408)
- Fix log message formatting in `filter_nonfinite` (3436)
- Add `RetryFailedTrialCallback` to `optuna.storages.*` (3441)
- Unify `fail_stale_trials` in each storage implementation (3442, thanks knshnb!)

Continuous Integration

- Add performance benchmarks using `bayesmark` (3354)
- Fix speed benchmarks (3362)
- Pin `setuptools` (3427)

Other

- Bump up version to `v3.0.0b0.dev` (3289)
- Add description field for `question-and-help-support` (3305)
- Update README to inform `v3.0.0a2` (3314)
- Add Optuna-related URLs for PyPi (3355, thanks andriyor!)
- Bump Optuna to `v3.0.0-b0` (3458)


Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

BasLaa, CorentinNeovision, HideakiImamura, Hiroyuki-01, andriyor, belldandyxtq, belltailjp, contramundum53, divyanshugit, harupy, higucheese, himkt, hppRC, hvy, kei-mo, keisuke-umezawa, keisukefukuda, knshnb, ll7, masap, not522, nyanhi, nzw0301, shu65, sile, tohmae, toshihikoyanase, tsukudamayo, xadrianzetx

This is the release note of [v3.0.0-a2](https://github.com/optuna/optuna/milestone/42?closed=1).

Highlights

`Study.optimize` Warning Configuration Fix

This is a small release that fixes a bug that the same warning message was emitted more than once when calling `Study.optimize`.

Bug Fixes

- [Backport] Allow `fail_state_trials` show warning when heartbeat is enabled (3303)

Other

- Bump Optuna (3302)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, himkt

This is the release note of [v3.0.0-a1](https://github.com/optuna/optuna/milestone/40?closed=1).

Highlights

Second alpha pre-release in preparation for the upcoming major version update v3.

Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.

_To read about the entire v3 roadmap, please refer to the [Wiki](https://github.com/optuna/optuna/wiki/Optuna-V3-Roadmap)._

_While this is a pre-release, we encourage users to keep using the latest releases of Optuna, including this one, for a smoother transition to the coming major release. Early feedback is welcome!_

A New Algorithm: Quasi-Monte Carlo Sampler

Now, you can utilize a new sampling algorithm based on the Quasi-Monte Carlo method, `optuna.samplers.QMCSampler`. This is oftentimes a good alternative to the existing `optuna.samplers.RandomSampler`. The generated (sampled) sequences have lower discrepancies compared to the standard random sequences, which are sampled uniformly. The following figures show the performance comparison to other existing samplers. Note that this algorithm is only supported for python >= 3.7.

See [2423](https://github.com/optuna/optuna/pull/2423) for more details.

Parkinson in HPOBench | Slice in HPOBench
-- | --
 | 


Constraints Support for Pareto-front plot

The Pareto front plot now supports visualization of constrained optimization. In Optuna, `NSGAIISampler` and `BoTorchSampler` allow constrained optimization by taking a function `constraints_func` as argument, then examine whether trials are feasible or not. The `optuna.visualization.plot_pareto_front` receives a similar function and uses this function to plot the trials in different colors depending on whether they violate the constraints or not.

See 3128 for more details.

python
def objective(trial):
Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)

Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))

v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2

return v0, v1

def constraints(trial):
return trial.user_attrs["constraints"]

if __name__ == "__main__":
sampler = optuna.samplers.NSGAIISampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)

optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()




Distribution Cleanup

We are actively working on cleaning up distributions for integer and floating-point. In Optuna v3, these distribution are unified to `optuna.distributions.IntDistribution` and `optuna.distributions.FloatDistribution`. v3.0.0-a1 contains several changes for this project and you will temporarily see `UserWarning` when you call `Trial.suggest_int` and `Trial.suggest_float`. We apologize for the inconvenience and the warning will be removed from the next release.

See [2941](https://github.com/optuna/optuna/issues/2941) for more information.

Stabilization of Experimental Modules

We make AllenNLP integration and `FrozenTrial.create_trial` stable.

See [3196](https://github.com/optuna/optuna/pull/3196) and [#3228](https://github.com/optuna/optuna/pull/3228) for more information

Breaking Changes

- Remove `type_checking.py` (3235)

New Features

- Add QMC sampler (2423, thanks kstoneriv3!)
- Refactor pareto front and support `constraints_func` in `plot_pareto_front` (3128, thanks semiexp!)
- Add `skip_if_finished` flag to `Study.tell` (3150, thanks xadrianzetx!)
- Add `user_attrs` argument to `Study.enqueue_trial` (3185, thanks knshnb!)
- Option to inherit intermediate values in `RetryFailedTrialCallback` (3269, thanks knshnb!)
- Add setter method for `DiscreteUniformDistribution.q` (3283)
- Stabilize allennlp integrations (3228)
- Stabilize `create_trial` (3196)

Enhancements

- Reduce number of queries to fetch `directions`, `user_attrs` and `system_attrs` of study summaries (3108)
- Support `FloatDistribution` across codebase (3111, thanks xadrianzetx!)
- Use `json.loads` to decode pruner configuration loaded from environment variables (3114)
- Show progress bar based on `timeout` (3115, thanks xadrianzetx!)
- Support `IntDistribution` across codebase (3126, thanks nyanhi!)
- Make progress bar available with n_jobs!=1 (3138, thanks masap!)
- Wrap `RedisStorage` in `CachedStorage` (3204, thanks masap!)
- Use `functools.wraps` in `track_in_mlflow` decorator (3216)
- Make `RedisStorage` fast when running multiple trials (3262, thanks masap!)
- Reduce database query result for `Study.ask()` (3274, thanks masap!)

Bug Fixes

- Fix bug of nondeterministic behavior of `TPESampler` when `group=True` (3187, thanks xuzijian629!)
- Handle non-numerical params in `matplotlib.contour_plot` (3213, thanks xadrianzetx!)
- Fix log scale axes padding in `matplotlib.contour_plot` (3218, thanks xadrianzetx!)
- Handle `-inf` and `inf` values in `RDBStorage` (3238, thanks xadrianzetx!)
- Skip limiting the value if it is `nan` (3286)

Installation

- Bump to `torch` related packages (3156)
- Use `pytorch-lightning>=1.5.0` (3157)
- Remove testoutput from doctest of `mlflow` integration (3170)
- Restrict `nltk` version (3201)
- Add version constraints of `setuptools` (3207)
- Remove version constraint of `setuptools` (3231)
- Remove Sphinx version constraint (3237)

Documentation

- Add a note `logging_callback` only works in single process situation (3143)
- Correct `FrozenTrial`'s docstring (3161)
- Promote to use of v3.0.0a0 in `README.md` (3167)
- Mention tutorial of callback for `Study.optimize` from API page (3171, thanks xuzijian629!)
- Add reference to tutorial page in `study.enqueue_trial` (3172, thanks knshnb!)
- Fix typo in specify_params (3174, thanks knshnb!)
- Guide to tutorial of Multi-objective Optimization in visualization tutorial (3182, thanks xuzijian629!)
- Add explanation about Parallelize Optimization at FAQ (3186, thanks MasahitoKumada!)
- Add order in tutorial (3193, thanks makinzm!)
- Fix inconsistency in `distributions` documentation (3222, thanks xadrianzetx!)
- Add FAQ entry for heartbeat (3229)
- Replace AUC with accuracy in docs (3242)
- Fix `Raises` section of `FloatDistribution` docstring (3248, thanks xadrianzetx!)
- Add `{Float,Int}Distribution` to docs (3252)
- Update explanation for metrics of `AllenNLPExecutor` (3253)
- Add missing cli methods to the list (3268)
- Add docstring for property `DiscreteUniformDistribution.q` (3279)

Examples

- Add pytorch-lightning DDP example (https://github.com/optuna/optuna-examples/pull/43, thanks tohmae!)
- Install latest AllenNLP (https://github.com/optuna/optuna-examples/pull/73)
- Restrict `nltk` version (https://github.com/optuna/optuna-examples/pull/75)
- Add version constraints of `setuptools` (https://github.com/optuna/optuna-examples/pull/76)
- Remove constraint of `setuptools` (https://github.com/optuna/optuna-examples/pull/79)

Tests

- Add tests for transformer with upper bound parameter (3163)
- Add tests in `visualization_tests/matplotlib_tests/test_slice.py` (3175, thanks keisukefukuda!)
- Add test case of the value in optimization history with matplotlib (3176, thanks TakuyaInoue-github!)
- Add tests for generated plots of `matplotlib.plot_edf` (3178, thanks makinzm!)
- Improve pareto front figure tests for matplotlib (3183, thanks akawashiro!)
- Add tests for generated plots of `plot_edf` (3188, thanks makinzm!)
- Match contour tests between Plotly and Matplotlib (3192, thanks belldandyxtq!)
- Implement missing `matplotlib.contour_plot` test (3232, thanks xadrianzetx!)
- Unify the validation function of edf value between visualization backends (3233)
- Add test for default grace period (3263, thanks masap!)
- Add the missing tests of Plotly's `plot_parallel_coordinate` (3266, thanks MasahitoKumada!)
- Switch function order progbar tests (3280, thanks BasLaa!)

Code Fixes

- Black fix (3147)
- Switch to `FloatDistribution` (3166, thanks xadrianzetx!)
- Remove `deprecated` decorator of the feature of `n_jobs` (3173, thanks MasahitoKumada!)
- Fix black and blackdoc errors (3260, thanks masap!)
- Remove experimental label from `MaxTrialsCallback` (3261, thanks knshnb!)
- Remove redundant `_check_trial_id` (3264, thanks masap!)
- Make existing int/float distributions wrapper of `{Int,Float}Distribution` (3244)

Continuous Integration

- Use python 3.8 for CI and docker (3026)
- Add performance benchmarks using `kurobako` (3155)
- Use Python 3.7 in checks CI job (3239)

Other

- Bump up version to v3.0.0a1.dev (3142)
- Introduce a form to make TODOs explicit when creating issues (3169)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

BasLaa, HideakiImamura, MasahitoKumada, TakuyaInoue-github, akawashiro, belldandyxtq, g-votte, himkt, hvy, keisuke-umezawa, keisukefukuda, knshnb, kstoneriv3, makinzm, masap, not522, nyanhi, nzw0301, semiexp, tohmae, toshihikoyanase, tupui, xadrianzetx, xuzijian629

This is the release note of [v3.0.0-a0](https://github.com/optuna/optuna/milestone/39?closed=1).

Highlights

First alpha pre-release in preparation for the upcoming major version update v3.

Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.

*To read about the entire v3 roadmap, please refer to the [Wiki](https://github.com/optuna/optuna/wiki/Optuna-V3-Roadmap).*

*While this is a pre-release, we encourage users to keep using the latest releases of Optuna, including this one, for a smoother transition to the coming major release. Early feedback is welcome!*

CLI Improvements

Optuna CLI speed and usability improvements. Previously, it took several seconds to launch a CLI command, 3000 significantly speeds up the commands by halving the module load time.

The usability of the ask-and-tell interface is also improved. The `ask` command allows users to define search space with short and simple JSON strings after 2905. The `tell` command supports `--skip-if-finished` which ignores duplicated reports of values and statuses instead of raising errors. It for instance improves robustness against pod retries on cluster environments. See 2905.

Before:
console
$ optuna ask --storage sqlite:///mystorage.db --study-name mystudy \
--search-space '{"x": {"name": "UniformDistribution", "attributes": {"low": 0.0, "high": 1.0}}}'


After:
console
$ optuna ask --storage sqlite:///mystorage.db --study-name mystudy \
--search-space '{"x": {"type": "float", "low": 0.0, "high": 1.0}}'


New NSGA-II Crossover Options

The optimization performance of NSGA-II has been greatly improved for real-valued problems. We introduce the `crossover` argument in `NSGAIISampler`. You can select several variants of the `crossover` option from `uniform` (default), `blxalpha`, `sbx`, `vsbx`, `undx`, and `spx`.

The following figure shows that the newly introduced crossover algorithms perform better than existing algorithms, that is, the uniform crossover algorithm and the Gaussian process based algorithm, in terms of biasness, convergence, and diversity. Note that the previous method, other implementations (in kurobako), and the default of the new method are based on uniform crossover.

See 2903 for more information.


New History Visualization with Multiple Studies

The optimization history plot now supports visualization of multiple studies. It receives a list of studies. If the `error_bar` option is `False`, it outputs those histories in one figure. If the `error_bar` option is `True`, it calculates and shows the means and the standard deviations of those histories.

See 2807 for more details.

python
import optuna


def objective(trial):
return trial.suggest_float("x", 0, 1) ** 2


n_studies = 5
studies = [optuna.create_study(study_name=f"{i}th-study") for i in range(n_studies)]
for study in studies:
study.optimize(objective, n_trials=20)

This generates the first figure.
fig = optuna.visualization.plot_optimization_history(studies)
fig.write_image("./multiple.png")

This generates the second figure.
fig = optuna.visualization.plot_optimization_history(studies, error_bar=True)
fig.write_image("./error_bar.png")



AllenNLP Distributed Pruning

The AllenNLP integration supports pruning in distributed environments. This change enables users to use the `optuna_pruner` callback option along with the `distributed` option as can be seen in the following training configuration. See 2977.

yaml
...
trainer: {
optimizer: 'adam',
cuda_device: -1,
callbacks: [
{
type: 'optuna_pruner',
}
],
},
distributed: {
cuda_devices: [-1, -1],
},


Preparations for Unification of Distributions Classes

There are several implementations of `BaseDistribution` in Optuna, such as `UniformDistribution`, `DiscreteUniformDistribution`, `IntUniformDistribution`, `CategoricalDistribution`, This release includes part of ongoing work in reducing the number of these distribution classes to just `FloatDistribution`, `IntDistribution`, and `CategoricalDistribution`, aligning the classes to the trial suggest interface (`suggest_float`, `suggest_int`, and `suggest_categorical`). Please note that users are not recommended to use these distributions yet, because samplers haven’t been updated to support those. See 3063 for more details.

Breaking Changes

Some deprecated features including the `optuna.structs` module, `LightGBMTuner.best_booster`, and the `optuna dashboard` command are removed in 3057 and 3058. If you use such features please migrate to the new ones.

| Removed APIs | Corresponding active APIs |
| --- | --- |
| `optuna.structs.StudyDirection` | `optuna.study.StudyDirection` |
| `optuna.structs.StudySummary` | `optuna.study.StudySummary` |
| `optuna.structs.FrozenTrial` | `optuna.trial.FrozenTrial` |
| `optuna.structs.TrialState` | `optuna.trial.TrialState` |
| `optuna.structs.TrialPruned` | `optuna.exceptions.TrialPruned` |
| `optuna.integration.lightgbm.LightGBMTuner.best_booster` | `optuna.integration.lightgbm.LightGBMTuner.get_best_booster` |
| `optuna dashboard` | [`optuna-dashboard`](https://github.com/optuna/optuna-dashboard) |

- Unify `suggest` APIs for floating-point parameters (2990, thanks xadrianzetx!)
- Clean up deprecated features (3057, thanks nuka137!)
- Remove `optuna dashboard` (3058)

New Features

- Add interval for LightGBM callback (2490)
- Allow multiple studies and add error bar option to `plot_optimization_history` (2807)
- Support PyTorch-lightning DDP training (2849, thanks tohmae!)
- Add crossover operators for NSGA-II (2903, thanks yoshinobc!)
- Add abbreviated JSON formats of distributions (2905)
- Extend `MLflowCallback` interface (2912, thanks xadrianzetx!)
- Support AllenNLP distributed pruning (2977)
- Make `trial.user_attrs` logging optional in `MLflowCallback` (3043, thanks xadrianzetx!)
- Support multiple input of studies when plot with Matplotlib (3062, thanks TakuyaInoue-github!)
- Add `IntDistribution` & `FloatDistribution` (3063, thanks nyanhi!)
- Add `trial.user_attrs` to `pareto_front` hover text (3082, thanks kasparthommen!)
- Support error bar for Matplotlib (3122, thanks TakuyaInoue-github!)
- Add `optuna tell` with `--skip-if-finished` (3131)

Enhancements

- Add single distribution support to `BoTorchSampler` (2928)
- Speed up `import optuna` (3000)
- Fix `_contains` of `IntLogUniformDistribution` (3005)
- Render importance scores next to bars in `matplotlib.plot_param_importances` (3012, thanks xadrianzetx!)
- Make default value of `verbose_eval` `NoneN for `LightGBMTuner`/`LightGBMTunerCV` to avoid conflict (3014, thanks chezou!)
- Unify colormap of `plot_contour` (3017)
- Relax `FixedTrial` and `FrozenTrial` allowing not-contained parameters during `suggest_*` (3018)
- Raise errors if `optuna ask` CLI receives `--sampler-kwargs` without `--sampler` (3029)
- Remove `_get_removed_version_from_deprecated_version` function (3065, thanks nuka137!)
- Reformat labels for small importance scores in `plotly.plot_param_importances` (3073, thanks xadrianzetx!)
- Speed up Matplotlib backend `plot_contour` using SciPy's `spsolve` (3092)
- Remove updates in cached storage (3120, thanks shu65!)

Bug Fixes

- Add tests of `sample_relative` and fix type of return values of `SkoptSampler` and `PyCmaSampler` (2897)
- Fix `GridSampler` with `RetryFailedTrialCallback` or `enqueue_trial` (2946)
- Fix the type of `trial.values` in MLflow integration (2991)
- Fix to raise `ValueError` for invalid `q` in `DiscreteUniformDistribution` (3001)
- Do not call `trial.report` during sanity check (3002)
- Fix `matplotlib.plot_contour` bug (3046, thanks IEP!)
- Handle `single` distributions in `fANOVA` evaluator (3085, thanks xadrianzetx!)

Installation

- Support scikit-learn v1.0.0 (3003)
- Pin `tensorflow` and `tensorflow-estimator` versions to `<2.7.0` (3059)
- Add upper version constraint of PyTorchLightning (3077)
- Pin `keras` version to `<2.7.0` (3078)
- Remove version constraints of `tensorflow` (3084)

Documentation

- Add note of the behavior when calling multiple `trial.report` (2980)
- Add note for DDP training of `pytorch-lightning` (2984)
- Add note to `OptunaSearchCV` about direction (3007)
- Clarify `n_trials` in the docs (3016, thanks Rohan138!)
- Add a note to use pickle with different optuna versions (3034)
- Unify the visualization docs (3041, thanks sidshrivastav!)
- Fix a grammatical error in FAQ doc (3051, thanks belldandyxtq!)
- Less ambiguous documentation for `optuna tell` (3052)
- Add example for `logging.set_verbosity` (3061, thanks drumehiron!)
- Mention the tutorial of `002_configurations.py` in the `Trial` API page (3067, thanks makkimaki!)
- Mention the tutorial of `003_efficient_optimization_algorithms.py` in the `Trial` API page (3068, thanks makkimaki!)
- Add link from `set_user_attrs` in `Study` to the `user_attrs` entry in Tutorial (3069, thanks MasahitoKumada!)
- Update description for missing samplers and pruners (3087, thanks masaaldosey!)
- Simplify the unit testing explanation (3089)
- Fix range description in `suggest_float` docstring (3091, thanks xadrianzetx!)
- Fix documentation for the package installation procedure on different OS (3118, thanks masap!)
- Add description of `ValueError` and `TypeErorr` to `Raises` section of `Trial.report` (3124, thanks MasahitoKumada!)

Examples

- Use `RetryFailedTrialCallback` in `pytorch_checkpoint` example (https://github.com/optuna/optuna-examples/pull/59, thanks xadrianzetx!)
- Add Python 3.9 to CI yaml files (https://github.com/optuna/optuna-examples/pull/61)
- Replace `suggest_uniform` with `suggest_float` (https://github.com/optuna/optuna-examples/pull/63)
- Remove deprecated warning message in `lightgbm` (https://github.com/optuna/optuna-examples/pull/64)
- Pin `tensorflow` and `tensorflow-estimator` versions to `<2.7.0` (https://github.com/optuna/optuna-examples/pull/66)
- Restrict upper version of `pytorch-lightning` (https://github.com/optuna/optuna-examples/pull/67)
- Add an external resource to `README.md` (https://github.com/optuna/optuna-examples/pull/68, thanks solegalli!)

Tests

- Add test case of samplers for conditional objective function (2904)
- Test int distributions with default step (2924)
- Be aware of trial preparation when checking heartbeat interval (2982)
- Simplify the DDP model definition in the test of `pytorch-lightning` (2983)
- Wrap data with `np.asarray` in `lightgbm` test (2997)
- Patch calls to deprecated `suggest` APIs across codebase (3027, thanks xadrianzetx!)
- Make `return_cvbooster` of `LightGBMTuner` consistent to the original value (3070, thanks abatomunkuev!)
- Fix `parametrize_sampler` (3080)
- Fix verbosity for `tests/integration_tests/lightgbm_tuner_tests/test_optimize.py` (3086, thanks nyanhi!)
- Generalize empty search space test case to all hyperparameter importance evaluators (3096, thanks xadrianzetx!)
- Check if texts in legend by order agnostic way (3103)
- Add tests for axis scales to `matplotlib.plot_slice` (3121)

Code Fixes

- Add test case of samplers for conditional objective function (2904)
- Fix 2949, remove `BaseStudy` (2986, thanks twsl!)
- Use `optuna.load_study` in `optuna ask` CLI to omit `direction`/`directions` option (2989)
- Fix typo in `Trial` warning message (3008, thanks xadrianzetx!)
- Replaces boston dataset with california housing dataset (3011, thanks avats-dev!)
- Fix deprecation version of `suggest` APIs (3054, thanks xadrianzetx!)
- Add `remove_version` to the missing `deprecated` argument (3064, thanks nuka137!)
- Add example of `optuna.logging.get_verbosity` (3066, thanks MasahitoKumada!)
- Support `{Float|Int}Distribution` in NSGA-II crossover operators (3139, thanks xadrianzetx!)

Continuous Integration

- Install `botorch` to CI jobs on mac (2988)
- Use libomp 11.1.0 for Mac (3024)
- Run `mac-tests` CI at a scheduled time (3028)
- Set concurrency to github workflows (3095)
- Skip CLI tests when calculating the coverage (3097)
- Migrate `mypy` version to 0.910 (3123)
- Avoid installing the latest MLfow to prevent doctests from failing (3135)

Other

- Bump up version to 2.11.0dev (2976)
- Add roadmap news to `README.md` (2999)
- Bump up version number to 3.0.0a1.dev (3006)
- Add Python 3.9 to `tox.ini` (3025)
- Fix version number to 3.0.0a0 (3140)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

Crissman, HideakiImamura, IEP, MasahitoKumada, Rohan138, TakuyaInoue-github, abatomunkuev, avats-dev, belldandyxtq, chezou, drumehiron, g-votte, himkt, hvy, kasparthommen, keisuke-umezawa, makkimaki, masaaldosey, masap, not522, nuka137, nyanhi, nzw0301, shu65, sidshrivastav, sile, solegalli, tohmae, toshihikoyanase, twsl, xadrianzetx, yoshinobc, ytsmiling