Optuna

This is the release note of [v3.0.0](https://github.com/optuna/optuna/milestone/46?closed=1).

This is not something you have to read from top to bottom to learn about the summary of Optuna v3. The recommended way is reading the [release blog](https://medium.com/optuna/optuna-3-part-1-4c6a9022a648).

If you want to update your existing projects from Optuna v2.x to Optuna v3, please see the [migration guide](https://github.com/optuna/optuna/discussions/3930) and try out Optuna v3.

Highlights

New Features

New NSGA-II Crossover Options
New crossover options are added to NSGA-II sampler, the default multi-objective algorithm of Optuna. The performance for floating point parameters are improved. Please visit [2903](https://github.com/optuna/optuna/pull/2903), [#3221](https://github.com/optuna/optuna/pull/3221), and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/samplers/generated/optuna.samplers.NSGAIISampler.html) for more information.




A New Algorithm: Quasi-Monte Carlo Sampler

Quasi-Monte Carlo sampler is now supported. It can be used in place of `RandomSampler`, and can improve performance especially for high dimensional problems. See [2423](https://github.com/optuna/optuna/pull/2423), [#2964](https://github.com/optuna/optuna/issues/2964), and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/samplers/generated/optuna.samplers.QMCSampler.html) for more information.


Constrained Optimization Support for TPE


TPESampler now supports constraint-aware optimization. For more information on this feature, please visit [3506](https://github.com/optuna/optuna/pull/3506) and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/samplers/generated/optuna.samplers.TPESampler.html).

| Without constraints | With constraints |
| - | - |
|  |  |



Constraints Support for Pareto-front Plot
Pareto-front plot now shows which trials satisfy the constraints and which do not. For more information, please see the following PRs ([3128](https://github.com/optuna/optuna/pull/3128), [#3497](https://github.com/optuna/optuna/pull/3497), and [#3389](https://github.com/optuna/optuna/pull/3389)) and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/visualization/generated/optuna.visualization.plot_pareto_front.html).



A New Importance Evaluator: `ShapleyImportanceEvaluator`
We introduced a new importance evaluator, `optuna.integration.ShapleyImportanceEvaluator`, which uses [SHAP](https://shap.readthedocs.io/en/latest/index.html). See [#3507](https://github.com/optuna/optuna/pull/3507) and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.integration.ShapleyImportanceEvaluator.html) for more information.





New History Visualization with Multiple Studies
Optimization history plot can now compare multiple studies or display the mean and variance of multiple studies optimized with the same settings. For more information, please see the following multiple PRs ([2807](https://github.com/optuna/optuna/pull/2807), [#3062](https://github.com/optuna/optuna/pull/3062), [#3122](https://github.com/optuna/optuna/pull/3122), and [#3736](https://github.com/optuna/optuna/pull/3736)) and the [document](https://optuna.readthedocs.io/en/v3.0.0/reference/visualization/generated/optuna.visualization.plot_optimization_history.html).






Improved Stability

Optuna has a number of core APIs. One being the suggest API and the `optuna.Study` class. The visualization module is also frequently used to analyze results. Many of these have been simplified, stabilized, and refactored in v3.0.

Simplified Suggest API

The suggest API has been aggregated into 3 APIs: `suggest_float` for floating point parameters, `suggest_int` for integer parameters, and `suggest_catagorical` for categorical parameters. For more information, see [2939](https://github.com/optuna/optuna/issues/2939), [#2941](https://github.com/optuna/optuna/issues/2941), and PRs submitted for those issues.

Introduction of a Test Policy

We have developed and published a [test policy](https://github.com/optuna/optuna/wiki/Test-Policy) in v3.0 that defines how tests for Optuna should be written. Based on the published test policy, we have improved many unit tests. For more information, see https://github.com/optuna/optuna/issues/2974 and PRs with `test` label.

Visualization Refactoring

Optuna's visualization module had a deep history and various debts. We have worked throughout v3.0 to eliminate this debt with the help of many contributors. See 2893, 2913, 2959 and PRs submitted for those issues.

Stabilized Features

Through the development of v3.0, we have decided to provide many experimental features as stable features by going through their behavior, fixing bugs, and analyzing use cases. The following is a list of features that have been stabilized in v3.0.

- [optuna.study.MaxTrialsCallback](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.study.MaxTrialsCallback.html#optuna.study.MaxTrialsCallback)
- [optuna.study.Study.enqueue_trial](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.study.Study.html#optuna.study.Study.enqueue_trial)
- [optuna.study.Study.add_trial](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.study.Study.html#optuna.study.Study.add_trial)
- [optuna.study.Study.add_trials](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.study.Study.html#optuna.study.Study.add_trials)
- [optuna.study.copy_study](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.study.copy_study.html#optuna.study.copy_study)
- [optuna.trial.create_trial](https://optuna.readthedocs.io/en/v3.0.0/reference/generated/optuna.trial.create_trial.html#optuna.trial.create_trial)
- [optuna.visualization.plot_pareto_front](https://optuna.readthedocs.io/en/v3.0.0/reference/visualization/generated/optuna.visualization.plot_pareto_front.html#optuna.visualization.plot_pareto_front)

Performance Verification

Optuna has many algorithms implemented, but many of their behaviors and characteristics are unknown to the user. We have developed the following table to inform users of empirically known behaviors and characteristics. See [3571](https://github.com/optuna/optuna/pull/3571) and [#3593](https://github.com/optuna/optuna/pull/3593) for more details.


To quantitatively assess the performance of our algorithms, we have developed a benchmarking environment. We also evaluated the performance of the algorithms by conducting actual benchmarking experiments using this environment. See [here](https://github.com/optuna/optuna/tree/master/benchmarks), [#2964](https://github.com/optuna/optuna/issues/2964), and [#2906](https://github.com/optuna/optuna/issues/2906) for more details.




Breaking Changes

Changes to the RDB schema:

- To use Optuna v3.0.0 with `RDBStorage` that was created in the previous versions of Optuna, please execute `optuna storage upgrade` to migrate your database (3113, 3559, 3603, 3668).

This is the release note of [v3.0.0-rc0](https://github.com/optuna/optuna/milestone/44?closed=1).
**_This is a release candidate of Optuna V3. We plan to release the major version within a few weeks. Please try this version and report bugs!_**

Highlights

Constrained Optimization Support for TPE
`TPESampler`, the default sampler of Optuna, now supports constrained optimization. It takes a function `constraints_func` as an argument, and examines whether trials are feasible or not. Feasible trials are prioritized over infeasible ones similarly to `NSGAIISampler`. See [3506](https://github.com/optuna/optuna/pull/3506) for more details.

python
def objective(trial):
Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)

Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))

v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2

return v0, v1

def constraints(trial):
return trial.user_attrs["constraints"]

if __name__ == "__main__":
sampler = optuna.samplers.TPESampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)

optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()


| MOTPE without constraints | MOTPE with constraints |
| - | - |
|  |  |


A Major Refactoring of Visualization Module

We have undertaken major refactoring of the visualization features as one of the major tasks of Optuna V3. The current situation is as follows.

Unification of implementations of different backends: `plotly` and `matplotlib`

Historically, the implementations of Optuna's visualization features were split between two different backends, `plotly` and `matplotlib`. Many of these implementations were duplicated and unmaintainable, and many were implemented as a single large function, resulting in poor testability and, as a result, becoming the cause of many bugs. We clarified the specifications that each visualization function in Optuna must meet and defined the backend-independent information needed to perform the visualization. By using this information commonly across different backends, we achieved a highly maintainable and testable implementation, and improved the stability of the visualization functions dramatically. We are currently rewriting the unit tests, and the resulting tests will be simple yet powerful.

Visual Regression Test

It is very important to detect hidden bugs in the implementation through PR reviews. However, visualizations are likely to contain bugs that are difficult to find just by reading the code, and many of these bugs are only revealed when the visualization is actually performed. Therefore, we introduced the Visual Regression Test to improve the review process. In the PR for visualization features, you can jump to the Visual Regression Test link by clicking on the link generated from within the PR. Reviewers can verify that the PR implementation is performing the visualization properly.

<img width="1715" alt="173838319-24433136-bd59-47d5-afdb-2694aafe354d (1)" src="https://user-images.githubusercontent.com/38826298/183350545-bbe74c09-4721-461b-b3b5-cdba5092403f.png">

Improve Code Quality Including Many Bugfix

In the latter development cycle of Optuna v3, we put emphasis on improving the overall code quality of the library. We fixed several bugs and possible corruption of internal data structures on e.g. handling Inf/NaN values (3567, 3592, 3738, 3739, 3740) and invalid inputs (3668, 3808, 3814, 3819). For example, there had been bugs before v3 when NaN values were used in a `CategoricalDistribution` or `GridSampler`. In several other functions, NaN values were unacceptable but the library failed silently without any warning or error. Such bugs are fixed in this release.


New Features

- Add constraints option to `TPESampler` (3506)
- Add `skip_if_exists` argument to `enqueue_trial` (3629)
- Remove experimental from `plot_pareto_front` (3643)
- Add `popsize` argument to `CmaEsSampler` (3649)
- Add `seed` argument for `BoTorchSampler` (3756)
- Add `seed` argument for `SkoptSampler` (3791)
- Revert AllenNLP integration back to experimental (3822)

Enhancements

- Move `is_heartbeat_enabled` from storage to heartbeat (3596)
- Refactor `ImportanceEvaluators` (3597)
- Avoid maximum limit when MLflow saves information (3651)
- Control metric decimal digits precision in `bayesmark` benchmark report (3693)
- Support `inf` values for crowding distance (3743)
- Normalize importance values (3828)

Bug Fixes

- Fix `CategoricalDistribution` with NaN (3567)
- Fix NaN comparison in grid sampler (3592)
- Fix bug in `IntersectionSearchSpace` (3666)
- Remove `trial_values` records whose values are `None` (3668)
- Fix PostgreSQL primary key unsorted problem (3702, thanks wattlebirdaz!)
- Raise error on NaN in `_constrained_dominates` (3738)
- Fix `inf`-related issue on implementation of `_calculate_nondomination_rank` (3739)
- Raise errors for NaN in constraint values (3740)
- Fix `_calculate_weights` such that it throws `ValueError` on invalid weights (3742)
- Change warning for `axis_order` of `plot_pareto_front` (3802)
- Fix check for number of objective values (3808)
- Raise `ValueError` when waiting trial is told (3814)
- Fix `Study.tell` with invalid values (3819)
- Fix infeasible case in NSGAII test (3839)

Installation

- Add a version constraint of cached-path (3665)

Documentation

- Add documentation of SHAP integration (3623)
- Remove news entry on Optuna user survey (3645)
- Introduce `optuna-fast-fanova` (3647)
- Add github discussions link (3660)
- Fix a variable name of ask-and-tell tutorial (3663)
- Clarify which trials are used for importance evaluators (3707)
- Fix typo in `Study.optimize` (3720, thanks 29Takuya!)
- Update link to plotly's jupyterlab-support page (3722, thanks 29Takuya!)
- Update `CONTRIBUTING.md` (3726)
- Remove "Edit on Github" button (3777, thanks cfkazu!)
- Remove duplicated period at the end of copyright (3778)
- Add note for deprecation of `plot_pareto_front`'s `axis_order` (3803)
- Describe the purpose of `prepare_study_with_trials` (3809)
- Fix a typo in docstring of `ShapleyImportanceEvaluator` (3810)
- Add a reference for MOTPE (3838, thanks y0z!)

Examples

- Introduce stale bot (https://github.com/optuna/optuna-examples/pull/119)
- Use Hydra 1.2 syntax (https://github.com/optuna/optuna-examples/pull/122)
- Fix CI due to `thop` (https://github.com/optuna/optuna-examples/pull/123)
- Hotfix `allennlp` dependency (https://github.com/optuna/optuna-examples/pull/124)
- Remove unreferenced variable in `pytorch_simple.py` (https://github.com/optuna/optuna-examples/pull/125)
- set `OMPI_MCA_rmaps_base_oversubscribe=yes` before `mpirun` (https://github.com/optuna/optuna-examples/pull/126)

Tests

- Simplify multi-objective TPE tests (3653)
- Add edge cases to multi-objective TPE tests (3662)
- Remove tests on `TypeError` (3667)
- Add edge cases to the tests of the parzen estimator (3673)
- Add tests for `_constrained_dominates` (3683)
- Refactor tests of constrained TPE (3689)
- Add `inf` and NaN tests for `test_constraints_func` (3690)
- Fix calling storage API in study tests (3695, thanks wattlebirdaz!)
- DRY `test_frozen.py` (3696)
- Unify the tests of `plot_contour`s (3701)
- Add test cases for crossovers of NSGAII (3705)
- Enhance the tests of `NSGAIISampler._crowding_distance_sort` (3706)
- Unify edf test files (3730)
- Fix `test_calculate_weights_below` (3741)
- Refactor `test_intermediate_plot.py` (3745)
- Test samplers are reproducible (3757)
- Add tests for `_dominates` function (3764)
- DRY importance tests (3785)
- Move tests for `create_trial` (3794)
- Remove `with_c_d` option from `prepare_study_with_trials` (3799)
- Use `DeterministicRelativeSampler` in `test_trial.py` (3807)

Code Fixes

- Add typehint for deprecated and experimental (3575)
- Move heartbeat-related thread operation in `_optimize.py` to `_heartbeat.py` (3609)
- Remove useless object inheritance (3628, thanks harupy!)
- Remove useless `except` clauses (3632, thanks harupy!)
- Rename `optuna.testing.integration` with `optuna.testing.pruner` (3638)
- Cosmetic fix in Optuna CLI (3641)
- Enable `strict_equality` for `mypy` 3579 (3648, thanks wattlebirdaz!)
- Make file names in testing consistent with `optuna` module (3657)
- Remove the implementation of `read_trials_from_remote_storage` in the all storages apart from `CachedStorage` (3659)
- Remove unnecessary deep copy in Redis storage (3672, thanks wattlebirdaz!)
- Workaround `mypy` bug (3679)
- Unify `plot_contour`s (3682)
- Remove `storage.get_all_study_summaries(include_best_trial: bool)` (3697, thanks wattlebirdaz!)
- Unify the logic of edf functions (3698)
- Unify the logic of `plot_param_importances` functions (3700)
- Enable `disallow_untyped_calls` for `mypy` (3704, thanks 29Takuya!)
- Use `get_trials` with `states` argument to filter trials depending on trial state (3708)
- Return Python's native float values (3714)
- Simplify `bayesmark` benchmark report rendering (3725)
- Unify the logic of intermediate plot (3731)
- Unify the logic of slice plot (3732)
- Unify the logic of `plot_parallel_coordinate`s (3734)
- Unify implementation of `plot_optimization_history` between `plotly` and `matplotlib` (3736)
- Extract `fail_objective` and `pruned_objective` for tests (3737)
- Remove deprecated storage functions (3744, thanks 29Takuya!)
- Remove unnecessary optionals from `visualization/_pareto_front.py` (3752)
- Change types inside `_ParetoInfoType` (3753)
- Refactor pareto front (3754)
- Use `_ContourInfo` to plot in `plot_contour` (3755)
- Follow up 3465 (3763)
- Refactor importances plot (3765)
- Remove `no_trials` option of `prepare_study_with_trials` (3766)
- Follow the coding style of comments in `plot_contour` files (3767)
- Raise `ValueError` for invalid returned type of `target` in `_filter_nonfinite` (3768)
- Fix value error condition in `plot_contour` (3769)
- DRY constraints in `Sampler.after_trial` (3775)
- DRY `stop_objective` (3786)
- Refactor non-exist param test in `plot_contour` test (3787)
- Remove `less_than_two` and `more_than_three` options from `prepare_study_with_trials` (3789)
- Fix return value's type of `_get_node_value` (3818)
- Remove unused `type: ignore` (3832)
- Fix typos and remove unused argument in `QMCSampler` (3837)

Continuous Integration

- Use `coverage` directly (3347, thanks higucheese!)
- Install 3rd party libraries in CI for lint (3580)
- Make `bayesmark` benchmark results comparable to `kurobako` (3584)
- Enable `warn_unused_ignores` for `mypy` (3627, thanks harupy!)
- Add `onnx` and version constrained `protobuf` to document dependencies (3658)
- Add `mo-kurobako` benchmark to CI (3691)
- Enable mypy's strict configs (3710)
- Run visual regression tests to find regression bugs of visualization module (3721)
- Remove downloading old `libomp` for mac tests (3728)
- Match Python versions between `bayesmark` CI jobs (3750)
- Set `OMPI_MCA_rmaps_base_oversubscribe=yes` before `mpirun` (3758)
- Add `budget` option to benchmarks (3774)
- Add `n_concurrency` option to benchmarks (3776)
- Use `n-runs` instead of `repeat` to represent the number of studies in the bayesmark benchmark (3780)
- Fix type hints for `mypy 0.971` (3797)
- Pin scipy to avoid the CI failure (3834)
- Extract float value from tensor for `trial.report` in `PyTorchLightningPruningCallback` (3842)

Other

- Clarify the criteria to assign reviewers in the PR template (3619)
- Bump up version number to v3.0.0rc0.dev (3621)
- Make `tox.ini` consistent with `checking` (3654)
- Avoid to stale description-checked issues (3816)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

29Takuya, HideakiImamura, c-bata, cfkazu, contramundum53, g-votte, harupy, higucheese, himkt, hvy, keisuke-umezawa, knshnb, not522, nzw0301, sile, toshihikoyanase, wattlebirdaz, xadrianzetx, y0z

This is the release note of [v3.0.0-b1](https://github.com/optuna/optuna/milestone/43?closed=1).

Highlights

A Samplers Comparison Table
We added a sampler comparison table on [the samplers' documentation page](https://optuna.readthedocs.io/en/latest/reference/samplers/index.html). It includes supported options (parameter types, pruning, multi-objective optimization, constrained optimization, etc.), time complexity, and recommended budgets for each sampler. Please use this to select appropriate samplers for your tasks! See #3571 and 3593 for more details.

<img width="1224" alt="sampler_comparison_table" src="https://user-images.githubusercontent.com/38826298/172111648-eb56206f-8539-48ac-a0ab-b1cc199e2d22.png">

A New Importance Evaluator: `ShapleyImportanceEvaluator`

Optuna now supports mean absolute SHAP value for evaluating parameter importances through integration with the [SHAP library](https://github.com/slundberg/shap). SHAP value is a game-theoretic measure of parameter importance featuring nice theoretical properties (See [paper](https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf) for more information).

<img width="469" alt="168213146-465a8116-94f2-49c9-b4ce-ec12970d82f8" src="https://user-images.githubusercontent.com/38826298/172112327-cda4a831-e1aa-486d-a481-68a44b14637c.png">

To use mean absolute SHAP importances, an object of `optuna.integration.shap.ShapleyImportanceEvaluator` can be passed to `evaluator` argument in `optuna.visualization.plot_param_importances` or `optuna.importance.get_param_importances`.


import optuna
from optuna.integration.shap import ShapleyImportanceEvaluator

study = optuna.create_study()
study.optimize(objective, n_trials=100)

optuna.visualization.plot_param_importances(study, evaluator=ShapleyImportanceEvaluator())


See the 3507 for more details.

A New Benchmarking Task

The benchmarking environment for black-box optimization algorithms on the GitHub Actions was introduced in the previous release. We have further enhanced its capabilities. The benchmarking functionality introduced can be run on all users' forks using GitHub Actions. You can also freely customize and run benchmarks on more computationally powerful clusters, for example, AWS, using the code in the `optuna/benchmarks` directory.

Neural Architecture Search Benchmark Support

Optuna's algorithms can now be benchmarked using NASLib, the Neural Architecture Search benchmark library. For now we only support one dataset, NASBench 201, which deals with image recognition. Larger datasets and datasets from other areas such as natural language processing will be supported in the future.

| cifar10 | cifar100 | imagenet16-120|
| ---- | ---- | ---- |
|  |  |  |

See [README](https://github.com/optuna/optuna/blob/master/benchmarks/README.md#performance-benchmarks-with-naslib) and 3465 for more information.

Multi-objective Optimization Benchmark Support

We are now able to benchmark our multi-objective optimization algorithms. They are not yet available on GitHub Actions, but you can use `optuna/benchmarks/run_mo_kurobakmo.py` directly. They will be available on GitHub Actions in the next release, so stay tuned! See 3271 and 3349 for more details.

Python 3.10 Support

This is the first version to officially support Python 3.10. All tests are passed including integration modules, with a few exceptions.

Storage Database Migration

To use Optuna v3.0.0-b1 with `RDBStorage` that was created in the previous versions of Optuna, please run `optuna storage upgrade` to migrate your database.

sh
`YOUR_RDB_URL` is the URL of your database.
optuna storage upgrade –storage YOUR_RDB_URL


If you use `RedisStorage`, copy your study with `RDBStorage` using [`copy_study`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.copy_study.html?highlight=copy_study) with the Optuna you used to create the study, thenrun `optuna storage upgrade` with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new `RedisStorage`.

sh
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’




Breaking Changes

- Fix distribution compatibility for linear and logarithmic distribution (3444)
- Remove `get_study_id_from_trial_id` (3538)

New Features

- Add `targets` argument to `plot_pareto_plont` of `plotly` backend (3495, thanks TakuyaInoue-github!)
- Support `constraints_func` in `plot_pareto_front` in matplotlib visualization (3497, thanks fukatani!)
- Calculate the feature importance with mean absolute SHAP values (3507, thanks liaison!)
- Make `GridSampler` reproducible (3527, thanks gasin!)
- Replace `ValueError` with `warning` in `GridSearchSampler` (3545)
- Implement `callbacks` argument of `OptunaSearchCV` (3577)
- Add option to skip table creation to `RDBStorage` (3581)

Enhancements

- Set `precision` of `sqlalchemy.Float` in `RDBStorage` table definition (3327)
- Accept `nan` in `trial.report` (3348, thanks belldandyxtq!)
- Lazy import of alembic, sqlalchemy, and scipy (3381)
- Unify pareto front (3389, thanks semiexp!)
- Make `set_trial_param()` of `RedisStorage` faster (3391, thanks masap!)
- Make `_set_best_trial()` of `RedisStorage` faster (3392, thanks masap!)
- Make `set_study_directions()` of `RedisStorage` faster (3393, thanks masap!)
- Make optuna compatible with wandb sweep panels (3403, thanks captain-pool!)
- Change "Trials" to "Trial" in `plot_slice`, `plot_pareto_front`, and `plot_optimization_history` (3449, thanks dubey-anshuman!)
- Make contour plots handle trials with nonfinite values (3451)
- Query studies for trials only once in EDF plots (3460)
- Make Parallel-Coordinate plots handle trials with nonfinite values (3471, thanks divyanshugit!)
- Separate heartbeat functionality from `BaseStorage` (3475)
- Remove `torch.distributed` calls from `TorchDistributedTrial` properties (3490, thanks nlgranger!)
- Remove the internal logic that calculates the interaction of two or more variables in fANOVA (3543)
- Handle inf/-inf for `trial_values` table in RDB (3559)
- Add `intermediate_value_type` column to represent inf/-inf on `RDBStorage` (3564)

Bug Fixes

- Import `COLOR_SCALE` inside import util context (3492)
- Remove `-v` option of `optuna study set-user-attr` command (3499, thanks nyanhi!)
- Filter trials with nonfinite value in `optuna.visualization.plot_param_importances` and `optuna.visualization.matplotlib.plot_param_importance` (3500, thanks takoika!)
- Fix `--verbose` and `--quiet` options in CLI (3532, thanks nyanhi!)
- Replace `ValueError` with `RuntimeError` in `get_best_trial` (3541)
- Take the same search space as in `CategoricalDistribution` by `GridSampler` (3544)

Installation

- Partially support Python 3.10 (3353)
- Clean up `setup.py` (3517)
- Remove duplicate requirements from `document` section (3613)

Documentation

- Clean up exception docstrings (3429)
- Revise docstring in MLFlow and WandB callbacks (3477)
- Change the parameter name from `classifier` to `regressor` in the code snippet of `README.md` (3481)
- Add link to Minituna in `CONTRIBUTING.md` (3482)
- Fix `benchmarks/README.md` for the `bayesmark` section (3496)
- Mention `Study.stop` as a criteria to stop creating trials in document (3498, thanks takoika!)
- Fix minor English errors in the docstring of `study.optimize` (3505)
- Add Python 3.10 in supported version in `README.md` (3508)
- Remove articles at the beginning of sentences in crossovers (3509)
- Correct `FronzenTrial`'s docstring (3514)
- Mention specify hyperparameter tutorial (3515)
- Fix typo in MLFlow callback (3533)
- Improve docstring of `GridSampler`'s seed option (3568)
- Add the samplers comparison table (3571)
- Replace `youtube.com` with `youtube-nocookie.com` (3590)
- Fix time complexity of the samplers comparison table (3593)
- Remove `language` from docs configuration (3594)

Examples

- Fix version of JAX (https://github.com/optuna/optuna-examples/pull/99)
- Remove constraints by 99 (https://github.com/optuna/optuna-examples/pull/100)
- Replace some methods in the `sklearn` example (https://github.com/optuna/optuna-examples/pull/102, thanks MasahitoKumada!)
- Add Python3.10 in `allennlp.yml` (https://github.com/optuna/optuna-examples/pull/104)
- Remove numpy (https://github.com/optuna/optuna-examples/pull/105)
- Add python 3.10 to fastai CI (https://github.com/optuna/optuna-examples/pull/106)
- Add python 3.10 to non-integration examples CIs (https://github.com/optuna/optuna-examples/pull/107)
- Add python 3.10 to Hiplot CI (https://github.com/optuna/optuna-examples/pull/108)
- Add a comma to `visualization.yml` (https://github.com/optuna/optuna-examples/pull/109)
- Rename WandB example to follow naming rules (https://github.com/optuna/optuna-examples/pull/110)
- Add scikit-learn version constraint for Dask-ML (https://github.com/optuna/optuna-examples/pull/112)
- Add python 3.10 to sklearn CI (https://github.com/optuna/optuna-examples/pull/113)
- Set version constraint of `protobuf` in PyTorch Lightning example (https://github.com/optuna/optuna-examples/pull/116)

Tests

- Improve `matplotlib` parallel coordinate test (3368)
- Save figures for all `matplotlib` tests (3414, thanks divyanshugit!)
- Add `inf` test to intermediate values test (3466)
- Add test cases for `test_storages.py` (3480)
- Improve the tests of `optuna.visualization.plot_pareto_front` (3546)
- Move heartbeat-related tests in `test_storages.py` to another file (3553)
- Use `seed` method of `np.random.RandomState` for reseeding and fix `test_reseed_rng` (3569)
- Refactor `test_get_observation_pairs` (3574)
- Add tests for `inf/nan` objectives for `ShapleyImportanceEvaluator` (3576)
- Add deprecated warning test to the multi-objective sampler test file (3601)

Code Fixes

- Ignore incomplete trials in `matplotlib.plot_parallel_coordinate` (3415)
- Update warning message and add a test when a trial fails with exception (3454)
- Remove old distributions from NSGA-II sampler (3459)
- Remove duplicated DB access in `_log_completed_trial` (3551)
- Reduce the number of `copy.deepcopy()` calls in `importance` module (3554)
- Remove duplicated `check_trial_is_updatable` (3557)
- Replace `optuna.testing.integration.create_running_trial` with `study.ask` (3562)
- Refactor `test_get_observation_pairs` (3574)
- Update label of feasible trials if `constraints_func` is specified (3587)
- Replace unused variable name with underscore (3588)
- Enable `no-implicit-optional` for `mypy` (3599, thanks harupy!)
- Enable `warn_redundant_casts` for `mypy` (3602, thanks harupy!)
- Refactor the type of value of `TrialIntermediateValueModel` (3603)
- Fix broken `mypy` checks of Alembic's `get_current_head()` method (3608)
- Sort dependencies by name (3614)

Continuous Integration

- Introduce the benchmark for multi-objectives samplers (3271, thanks drumehiron!)
- Add WFG benchmark test (3349, thanks kei-mo!)
- Add workflow to use `reviewdog` (3357)
- Add NASBench201 from NASLib (3465)
- Fix speed benchmarks CI (3470)
- Support PyTorch 1.11.0 (3510)
- Restore `virtualenv` for benchmark extras (3585)
- Use `protobuf<4.0.0` to resolve Sphinx CI error (3591)
- Unpin `protobuf` (3598, thanks harupy!)
- Extract MPI tests from integration CI as independent CI (3606)

Other

- Bump up version to v3.0.0b1.dev (3457)
- Fix `kurobako` benchmark code to run it locally (3468)
- Fix label of issue template (3493)
- Improve issue templates (3536)
- Hotfix for `fakeredis` 1.7.4 release (3549)
- Remove the version constraint of `fakeredis` (3561)
- Relax version constraint of `fakeredis` (3607)
- Shorten the durations of the stale bot for PRs (3611)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, MasahitoKumada, TakuyaInoue-github, belldandyxtq, c-bata, captain-pool, contramundum53, divyanshugit, drumehiron, dubey-anshuman, fukatani, g-votte, gasin, harupy, himkt, hvy, kei-mo, keisuke-umezawa, knshnb, liaison, masap, nlgranger, not522, nyanhi, nzw0301, semiexp, sile, takoika, toshihikoyanase, xadrianzetx

This is the release note of [v3.0.0-b0](https://github.com/optuna/optuna/milestone/41?closed=1).

Highlights

Simplified Distribution Classes: Float, Int and Categorical

Search space definitions, which consist of `BaseDistribution` and its child classes in Optuna, are greatly simplified. We have introduced `FloatDistribution`, `IntDistribution`, and `CategoricalDistribution`. If you use the suggest API and `Study.optimize`, the search space information is stored as these three distributions. Previous `UniformDistribution`, `LogUniformDistribution`, `DiscreteUniformDistribution`, `IntUniformDistribution`, and `IntLogUniformDistribution` are deprecated. If you pass deprecated distributions to APIs such as `Study.ask` or `create_trial`, they are internally converted to corresponding `FloatDistribution` or `IntDistribution`.

Storage Database Migration

To use Optuna v3.0.0-b0 with `RDBStorage` that was created in the previous versions of Optuna, please run `optuna storage upgrade` to migrate your database.

If you use `RedisStorage`, copy your study with `RDBStorage` using [`copy_study`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.copy_study.html?highlight=copy_study) with the Optuna you used to create the study, thenrun `optuna storage upgrade` with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new `RedisStorage`.

sh
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’


Consistent Ask-and-Tell Interface with `Study.optimize`

`Study.tell` fails a trial when it is called with certain invalid combinations of `state` and `values`, instead of raising an error. This change aims to make `Study.tell` consistent with `Study.optimize`, which continues an optimization even if an objective returns an invalid value.

`Study.tell` now also returns the resulting trial (`FrozenTrial`) in order to allow inspecting how the arguments were interpreted.

Before

`Study.tell` raises an exception when it is called with an invalid combination of `state` and `values`.

python
study.tell(study.ask(), values=None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/…/optuna/optuna/study/study.py", line 579, in tell
raise ValueError(
ValueError: No values were told. Values are required when state is TrialState.COMPLETE.


After

`Study.tell` automatically fails the trial.

python
trial: FrozenTrial = study.tell(study.ask(), value=None)
assert trial.state == TrialState.FAIL


See 3144 for more details.

Stable `Study` APIs

We are converting all positional arguments of `create_study`, `delete_study`, `load_study`, and `copy_study` to keyword-only arguments since the order of arguments were inconsistent. This is not yet a breaking-change, but if you use these features with positional arguments, then you will get a warning message to use them with keyword-only arguments.

In addition, we have fixed all of problems described in 2955, so we have stabled the `Study` APIs. Specifically, `Study.add_trial`, `Study.add_trials`, `Study.enqueue_trial`, and `copy_study` have been stabled.

See 3270 and 2955 for more details.

Improved Visualization

Several bugs in the visualization module have been resolved. For instance,
the parallel coordinates plot ignores trials with missing parameters ([3373](https://github.com/optuna/optuna/pull/3373)) and the scale of the objective value is fixed ([#3369](https://github.com/optuna/optuna/pull/3369)). The edf plot filters trials with `inf` values ([#3395](https://github.com/optuna/optuna/pull/3395) and [#3435](https://github.com/optuna/optuna/pull/3435)).

Before: Trials with missing parameters are wrongly connected to each other.
<img width="1203" alt="158495812-acb399e5-d817-4cae-8c8b-c69bcc91efea" src="https://user-images.githubusercontent.com/38826298/162878578-fc162708-5c3b-45a0-b098-47445a1242a7.png">

After: Trials with missing parameters are removed from the plot.
<img width="1191" alt="158495810-98a5ec29-581c-426a-a255-11d16ae1c144" src="https://user-images.githubusercontent.com/38826298/162878606-aa937868-b8f5-4b51-b09b-6aced6fd28ff.png">

Breaking Changes

- Add option to exclude best trials from study summaries (3109)
- Migrate to `{Float,Int}Distribution` using `alembic` (3113)
- Move validation logic from `_run_trial` to `study.tell` (3144)
- Enable `FloatDistribution` and `IntDistribution` (3246)
- Use an enqueued parameter that is out of range from suggest API (3298)
- Convert deprecated distribution to new distribution internally (3420)

New Features

- Add `CatBoostPruningCallback` (2734, thanks tohmae!)
- Create common API for all NSGA-II crossover operations (3221)
- Add a history of retried trial numbers in `Trial.system_attrs` (3223, thanks belltailjp!)
- Convert all positional arguments to keyword-only (3270, thanks higucheese!)
- Stabilize `study.py` (3309)
- Add `targets` and deprecate `axis_order` in `optuna.visualization.matplotlib.plot_pareto_front` (3341, thanks shu65!)

Enhancements

- Enable cache for `study.tell()` (3265, thanks masap!)
- Warn if heartbeat is used with ask-and-tell (3273)
- Make `optuna.study.get_all_study_summaries()` of `RedisStorage` fast (3278, thanks masap!)
- Improve Ctrl-C interruption handling (3374, thanks CorentinNeovision!)
- Use same colormap among `plotly` visualization methods (3376)
- Make EDF plots handle trials with nonfinite values (3435)
- Make logger message optional in `filter_nonfinite` (3438)

Bug Fixes

- Make TPE work with a categorical variable with different choice types (3190, thanks keisukefukuda!)
- Fix axis range issue in `matplotlib` contour plot (3249, thanks harupy!)
- Allow `fail_state_trials` show warning when heartbeat is enabled (3301)
- Clip untransformed values sampled from int uniform distributions (3319)
- Fix missing `user_attrs` and `system_attrs` in study summaries (3352)
- Fix objective scale in parallel coordinate of Matplotlib (3369)
- Fix `matplotlib.plot_parallel_coordinate` with log distributions (3371)
- Fix parallel coordinate with missing value (3373)
- Add utility to filter trials with `inf` values from visualizations (3395)
- Return the best trial number, not worst trial number by `best_index_` (3410)
- Avoid using `px.colors.sequential.Blues` that introduces `pandas` dependency (3422)
- Fix `_is_reverse_scale` (3424)

Installation

- Drop TensorFlow support for Python 3.6 (3296)
- Pin AllenNLP version (3367)
- Skip run `fastai` job on Python 3.6 (3412)
- Avoid latest `click==8.1.0` that removed a deprecated feature (3413)
- Avoid latest PyTorch lightning until integration is updated (3417)
- Revert "Avoid latest `click==8.1.0` that removed a deprecated feature" (3430)

Documentation

- Add reference to tutorial page in CLI (3267, thanks tsukudamayo!)
- Carry over notes on `step` behavior to new distributions (3276)
- Correct the disable condition of `show_progress_bar` (3287)
- Add a document to lead FAQ and example of heartbeat (3294)
- Add a note for `copy_study`: it creates a copy regardless of its state (3295)
- Add note to recommend Python 3.8 or later in documentation build with artifacts (3312)
- Fix crossover references in `Raises` doc section (3315)
- Add reference to `QMCSampler` in tutorial (3320)
- Fix layout in tutorial (with workaround) (3322)
- Scikit-learn required for `plot_param_importances` (3332, thanks ll7!)
- Add a link to multi-objective tutorial from a pareto front page (3339, thanks kei-mo!)
- Add reference to tutorial page in visualization (3340, thanks Hiroyuki-01!)
- Mention tutorials of User-Defined Sampler/Pruner from the API reference pages (3342, thanks hppRC!)
- Add reference to saving/resuming study with RDB backend (3345, thanks Hiroyuki-01!)
- Fix a typo (3360)
- Remove deprecated command `optuna study optimize` in FAQ (3364)
- Fix nit typo (3380)
- Add see also section for `best_trial` (3396, thanks divyanshugit!)
- Updates the tutorial page for re-use the best trial (3398, thanks divyanshugit!)
- Add explanation about `Study.best_trials` in multi-objective optimization tutorial (3443)

Examples

- Remove Python 3.6 from `haiku`'s CI (https://github.com/optuna/optuna-examples/pull/83)
- Apply `black` 22.1.0 & run `checks` daily (https://github.com/optuna/optuna-examples/pull/84)
- Add `hiplot` example (https://github.com/optuna/optuna-examples/pull/86)
- Stop running jobs using TF with Python3.6 (https://github.com/optuna/optuna-examples/pull/87)
- Pin AllenNLP version (https://github.com/optuna/optuna-examples/pull/89)
- Add Medium link (https://github.com/optuna/optuna-examples/pull/91)
- Use official `CatBoostPruningCallback` (https://github.com/optuna/optuna-examples/pull/92)
- Stop running `fastai` job on Python 3.6 (https://github.com/optuna/optuna-examples/pull/93)
- Specify Python version using `str` in workflow files (https://github.com/optuna/optuna-examples/pull/95)
- Introduce upper version constraint of PyTorchLightning (https://github.com/optuna/optuna-examples/pull/96)
- Update `SimulatedAnnealingSampler` to support `FloatDistribution` (https://github.com/optuna/optuna-examples/pull/97)

Tests

- Add plot value tests to `matplotlib_tests/test_param_importances` (3180, thanks belldandyxtq!)
- Make tests of `plot_optimization_history` methods consistent (3234)
- Add integration test for `RedisStorage` (3258, thanks masap!)
- Change the order of arguments in the `catalyst` integration test (3308)
- Cleanup `MLflowCallback` tests (3378)
- Test serialize/deserialize storage on parametrized conditions (3407)
- Add tests for parameter of 'None' for TPE (3447)

Code Fixes

- Switch to `IntDistribution` (3181, thanks nyanhi!)
- Fix type hints for Python 3.8 (3240)
- Remove `UniformDistribution`, `LogUniformDistribution` and `DiscreteUniformDistribution` code paths (3275)
- Merge `set_trial_state()` and `set_trial_values()` into one function (3323, thanks masap!)
- Follow up for `{Float, Int}Distributions` (3337, thanks nyanhi!)
- Move the `get_trial_xxx` abstract functions to base (3338, thanks belldandyxtq!)
- Update type hints of `states` (3359, thanks BasLaa!)
- Remove unused function from `RedisStorage` (3394, thanks masap!)
- Remove unnecessary string concatenation (3406)
- Follow coding style and fix typos in `tests/integration_tests` (3408)
- Fix log message formatting in `filter_nonfinite` (3436)
- Add `RetryFailedTrialCallback` to `optuna.storages.*` (3441)
- Unify `fail_stale_trials` in each storage implementation (3442, thanks knshnb!)

Continuous Integration

- Add performance benchmarks using `bayesmark` (3354)
- Fix speed benchmarks (3362)
- Pin `setuptools` (3427)

Other

- Bump up version to `v3.0.0b0.dev` (3289)
- Add description field for `question-and-help-support` (3305)
- Update README to inform `v3.0.0a2` (3314)
- Add Optuna-related URLs for PyPi (3355, thanks andriyor!)
- Bump Optuna to `v3.0.0-b0` (3458)


Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

BasLaa, CorentinNeovision, HideakiImamura, Hiroyuki-01, andriyor, belldandyxtq, belltailjp, contramundum53, divyanshugit, harupy, higucheese, himkt, hppRC, hvy, kei-mo, keisuke-umezawa, keisukefukuda, knshnb, ll7, masap, not522, nyanhi, nzw0301, shu65, sile, tohmae, toshihikoyanase, tsukudamayo, xadrianzetx

This is the release note of [v3.0.0-a2](https://github.com/optuna/optuna/milestone/42?closed=1).

Highlights

`Study.optimize` Warning Configuration Fix

This is a small release that fixes a bug that the same warning message was emitted more than once when calling `Study.optimize`.

Bug Fixes

- [Backport] Allow `fail_state_trials` show warning when heartbeat is enabled (3303)

Other

- Bump Optuna (3302)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, himkt

This is the release note of [v3.0.0-a1](https://github.com/optuna/optuna/milestone/40?closed=1).

Highlights

Second alpha pre-release in preparation for the upcoming major version update v3.

Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.

_To read about the entire v3 roadmap, please refer to the [Wiki](https://github.com/optuna/optuna/wiki/Optuna-V3-Roadmap)._

_While this is a pre-release, we encourage users to keep using the latest releases of Optuna, including this one, for a smoother transition to the coming major release. Early feedback is welcome!_

A New Algorithm: Quasi-Monte Carlo Sampler

Now, you can utilize a new sampling algorithm based on the Quasi-Monte Carlo method, `optuna.samplers.QMCSampler`. This is oftentimes a good alternative to the existing `optuna.samplers.RandomSampler`. The generated (sampled) sequences have lower discrepancies compared to the standard random sequences, which are sampled uniformly. The following figures show the performance comparison to other existing samplers. Note that this algorithm is only supported for python >= 3.7.

See [2423](https://github.com/optuna/optuna/pull/2423) for more details.

Parkinson in HPOBench | Slice in HPOBench
-- | --
 | 


Constraints Support for Pareto-front plot

The Pareto front plot now supports visualization of constrained optimization. In Optuna, `NSGAIISampler` and `BoTorchSampler` allow constrained optimization by taking a function `constraints_func` as argument, then examine whether trials are feasible or not. The `optuna.visualization.plot_pareto_front` receives a similar function and uses this function to plot the trials in different colors depending on whether they violate the constraints or not.

See 3128 for more details.

python
def objective(trial):
Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)

Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))

v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2

return v0, v1

def constraints(trial):
return trial.user_attrs["constraints"]

if __name__ == "__main__":
sampler = optuna.samplers.NSGAIISampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)

optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()




Distribution Cleanup

We are actively working on cleaning up distributions for integer and floating-point. In Optuna v3, these distribution are unified to `optuna.distributions.IntDistribution` and `optuna.distributions.FloatDistribution`. v3.0.0-a1 contains several changes for this project and you will temporarily see `UserWarning` when you call `Trial.suggest_int` and `Trial.suggest_float`. We apologize for the inconvenience and the warning will be removed from the next release.

See [2941](https://github.com/optuna/optuna/issues/2941) for more information.

Stabilization of Experimental Modules

We make AllenNLP integration and `FrozenTrial.create_trial` stable.

See [3196](https://github.com/optuna/optuna/pull/3196) and [#3228](https://github.com/optuna/optuna/pull/3228) for more information

Breaking Changes

- Remove `type_checking.py` (3235)

New Features

- Add QMC sampler (2423, thanks kstoneriv3!)
- Refactor pareto front and support `constraints_func` in `plot_pareto_front` (3128, thanks semiexp!)
- Add `skip_if_finished` flag to `Study.tell` (3150, thanks xadrianzetx!)
- Add `user_attrs` argument to `Study.enqueue_trial` (3185, thanks knshnb!)
- Option to inherit intermediate values in `RetryFailedTrialCallback` (3269, thanks knshnb!)
- Add setter method for `DiscreteUniformDistribution.q` (3283)
- Stabilize allennlp integrations (3228)
- Stabilize `create_trial` (3196)

Enhancements

- Reduce number of queries to fetch `directions`, `user_attrs` and `system_attrs` of study summaries (3108)
- Support `FloatDistribution` across codebase (3111, thanks xadrianzetx!)
- Use `json.loads` to decode pruner configuration loaded from environment variables (3114)
- Show progress bar based on `timeout` (3115, thanks xadrianzetx!)
- Support `IntDistribution` across codebase (3126, thanks nyanhi!)
- Make progress bar available with n_jobs!=1 (3138, thanks masap!)
- Wrap `RedisStorage` in `CachedStorage` (3204, thanks masap!)
- Use `functools.wraps` in `track_in_mlflow` decorator (3216)
- Make `RedisStorage` fast when running multiple trials (3262, thanks masap!)
- Reduce database query result for `Study.ask()` (3274, thanks masap!)

Bug Fixes

- Fix bug of nondeterministic behavior of `TPESampler` when `group=True` (3187, thanks xuzijian629!)
- Handle non-numerical params in `matplotlib.contour_plot` (3213, thanks xadrianzetx!)
- Fix log scale axes padding in `matplotlib.contour_plot` (3218, thanks xadrianzetx!)
- Handle `-inf` and `inf` values in `RDBStorage` (3238, thanks xadrianzetx!)
- Skip limiting the value if it is `nan` (3286)

Installation

- Bump to `torch` related packages (3156)
- Use `pytorch-lightning>=1.5.0` (3157)
- Remove testoutput from doctest of `mlflow` integration (3170)
- Restrict `nltk` version (3201)
- Add version constraints of `setuptools` (3207)
- Remove version constraint of `setuptools` (3231)
- Remove Sphinx version constraint (3237)

Documentation

- Add a note `logging_callback` only works in single process situation (3143)
- Correct `FrozenTrial`'s docstring (3161)
- Promote to use of v3.0.0a0 in `README.md` (3167)
- Mention tutorial of callback for `Study.optimize` from API page (3171, thanks xuzijian629!)
- Add reference to tutorial page in `study.enqueue_trial` (3172, thanks knshnb!)
- Fix typo in specify_params (3174, thanks knshnb!)
- Guide to tutorial of Multi-objective Optimization in visualization tutorial (3182, thanks xuzijian629!)
- Add explanation about Parallelize Optimization at FAQ (3186, thanks MasahitoKumada!)
- Add order in tutorial (3193, thanks makinzm!)
- Fix inconsistency in `distributions` documentation (3222, thanks xadrianzetx!)
- Add FAQ entry for heartbeat (3229)
- Replace AUC with accuracy in docs (3242)
- Fix `Raises` section of `FloatDistribution` docstring (3248, thanks xadrianzetx!)
- Add `{Float,Int}Distribution` to docs (3252)
- Update explanation for metrics of `AllenNLPExecutor` (3253)
- Add missing cli methods to the list (3268)
- Add docstring for property `DiscreteUniformDistribution.q` (3279)

Examples

- Add pytorch-lightning DDP example (https://github.com/optuna/optuna-examples/pull/43, thanks tohmae!)
- Install latest AllenNLP (https://github.com/optuna/optuna-examples/pull/73)
- Restrict `nltk` version (https://github.com/optuna/optuna-examples/pull/75)
- Add version constraints of `setuptools` (https://github.com/optuna/optuna-examples/pull/76)
- Remove constraint of `setuptools` (https://github.com/optuna/optuna-examples/pull/79)

Tests

- Add tests for transformer with upper bound parameter (3163)
- Add tests in `visualization_tests/matplotlib_tests/test_slice.py` (3175, thanks keisukefukuda!)
- Add test case of the value in optimization history with matplotlib (3176, thanks TakuyaInoue-github!)
- Add tests for generated plots of `matplotlib.plot_edf` (3178, thanks makinzm!)
- Improve pareto front figure tests for matplotlib (3183, thanks akawashiro!)
- Add tests for generated plots of `plot_edf` (3188, thanks makinzm!)
- Match contour tests between Plotly and Matplotlib (3192, thanks belldandyxtq!)
- Implement missing `matplotlib.contour_plot` test (3232, thanks xadrianzetx!)
- Unify the validation function of edf value between visualization backends (3233)
- Add test for default grace period (3263, thanks masap!)
- Add the missing tests of Plotly's `plot_parallel_coordinate` (3266, thanks MasahitoKumada!)
- Switch function order progbar tests (3280, thanks BasLaa!)

Code Fixes

- Black fix (3147)
- Switch to `FloatDistribution` (3166, thanks xadrianzetx!)
- Remove `deprecated` decorator of the feature of `n_jobs` (3173, thanks MasahitoKumada!)
- Fix black and blackdoc errors (3260, thanks masap!)
- Remove experimental label from `MaxTrialsCallback` (3261, thanks knshnb!)
- Remove redundant `_check_trial_id` (3264, thanks masap!)
- Make existing int/float distributions wrapper of `{Int,Float}Distribution` (3244)

Continuous Integration

- Use python 3.8 for CI and docker (3026)
- Add performance benchmarks using `kurobako` (3155)
- Use Python 3.7 in checks CI job (3239)

Other

- Bump up version to v3.0.0a1.dev (3142)
- Introduce a form to make TODOs explicit when creating issues (3169)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

BasLaa, HideakiImamura, MasahitoKumada, TakuyaInoue-github, akawashiro, belldandyxtq, g-votte, himkt, hvy, keisuke-umezawa, keisukefukuda, knshnb, kstoneriv3, makinzm, masap, not522, nyanhi, nzw0301, semiexp, tohmae, toshihikoyanase, tupui, xadrianzetx, xuzijian629