Optuna

$ python dask_simple.py


A brand-new Redis storage

We have replaced the Redis storage backend with a JournalStorage-based one. The experimental `RedisStorage` class has been removed in v3.1. The following example shows how to use the new `JournalRedisStorage` class.

python
import optuna
from optuna.storages import JournalStorage, JournalRedisStorage

def objective(trial):
…

storage = JournalStorage(JournalRedisStorage("redis://localhost:6379"))
study = optuna.create_study(storage=storage)
study.optimize(objective)


Sampler for brute-force search

`BruteForceSampler`, a new sampler for brute-force search, tries all combinations of parameters. In contrast to `GridSampler`, it does not require passing the search space as an argument and works even with branches. This sampler constructs the search space with the define-by-run style, so it works by just adding `sampler=optuna.samplers.BruteForceSampler()`.

python
import optuna

def objective(trial):
c = trial.suggest_categorical("c", ["float", "int"])
if c == "float":
return trial.suggest_float("x", 1, 3, step=0.5)
elif c == "int":
a = trial.suggest_int("a", 1, 3)
b = trial.suggest_int("b", a, 3)
return a + b

study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler())
study.optimize(objective)


Breaking Changes

- Allow users to call `study.optimize()` in multiple threads (4068)
- Use all trials in `TPESampler` even when `multivariate=True` (4079)
- Drop Python 3.6 (4150)
- Remove `RedisStorage` (4156)
- Deprecate `set_system_attr` in `Study` and `Trial` (4188)
- Deprecate `system_attrs` in `Study` class (4250)

New Features

- Add Dask integration (2023, thanks jrbourbeau!)
- Add journal-style log storage (3854)
- Support CMA-ES with margin in `CmaEsSampler` (4016)
- Add journal redis storage (4086)
- Add device argument to `BoTorchSampler` (4101)
- Add the feature to `JournalStorage` of Redis backend to resume from a snapshot (4102)
- Added `user_attrs` to print by optuna studies in `cli.py` (4129, thanks gonzaload!)
- Add `BruteForceSampler` (4132, thanks semiexp!)
- Add `__getstate__` and `__setstate__` to `RedisStorage` (4135, thanks shu65!)
- Support pickle in `JournalRedisStorage` (4139, thanks shu65!)
- Support for `qNoisyExpectedHypervolumeImprovement` acquisition function from `BoTorch` (Issue4014) (4186)

Enhancements

- Change the log message format for failed trials (3857, thanks erentknn!)
- Move default logic of `get_trial_id_from_study_id_trial_number()` method to BaseStorage (3910)
- Fix the data migration script for v3 release (4020)
- Convert `search_space` values of `GridSampler` explicitly (4062)
- Add single exception catch to study optimize (4098)
- Add validation in `enqueue_trial` (4126)
- Speed up `tests/samplers_tests/test_nsgaii.py::test_fast_non_dominated_sort_with_constraints` (4128, thanks mist714!)
- Add getstate and setstate to journal storage (4130, thanks shu65!)
- Support `None` in slice plot (4133, thanks belldandyxtq!)
- Add marker to matplotlib `plot_intermediate_value` (4134, thanks belldandyxtq!)
- Cache `study.directions` to reduce the number of `get_study_directions()` calls (4146)
- Add an in-memory cache in `Trial` class (4240)

Bug Fixes

- Fix infinite loop bug in `TPESampler` (3953, thanks gasin!)
- Fix `GridSampler` (3957)
- Fix an import error of `sqlalchemy.orm.declarative_base` (3967)
- Skip to add `intermediate_value_type` and `value_type` columns if exists (4015)
- Fix duplicated sampling of `SkoptSampler` (4023)
- Avoid parse errors of `datetime.isoformat` strings (4025)
- Fix a concurrency bug of JournalStorage `set_trial_state_values` (4033)
- Specify object type to numpy array init to avoid unintended str cast (4035)
- Make `TPESampler` reproducible (4056)
- Fix bugs in `constant_liar` option (4073)
- Add a flush to `JournalFileStorage.append_logs` (4076)
- Add a lock to `MLflowCallback` (4097)
- Reject deprecated distributions in `OptunaSearchCV` (4120)
- Stop using hash function in `_get_bracket_id` in `HyperbandPruner` (4131, thanks zaburo-ch!)
- Validation for the parameter enqueued in `to_internal_repr` of `FloatDistribution` and `IntDistribution` (4137)
- Fix `PartialFixedSampler` to handle `None` correctly (4147, thanks halucinor!)
- Fix the bug of JournalFileStorage on Windows (4151)
- Fix CmaEs system attribution key (4184)

Installation

- Replace `thop` with `fvcore` (3906)
- Use the latest stable scipy (3959, thanks gasin!)
- Remove GPyTorch version constraint (3986)
- Make typing_extensions optional (3990)
- Add version constraint on `importlib-metadata` (4036)
- Add a version constraint of `matplotlib` (4044)

Documentation

- Update cli tutorial (3902)
- Replace `thop` with `fvcore` (3906)
- Slightly improve docs of `FrozenTrial` (3943)
- Refine docs in `BaseStorage` (3948)
- Remove "Edit on GitHub" button from readthedocs (3952)
- Mention restoring sampler in saving/resuming tutorial (3992)
- Use `log_loss` instead of deprecated `log` since `sklearn` 1.1 (3993)
- Fix script path in benchmarks/README.md (4021)
- Ignore `ConvergenceWarning` in the ask-and-tell tutorial (4032)
- Update docs to let users know the concurrency problem on SQLite3 (4034)
- Fix the time complexity of `NSGAIISampler` (4045)
- Fix sampler comparison table (4082)
- Add `BruteForceSampler` in the samplers' list (4152)
- Remove markup from NaN in FAQ (4155)
- Remove the document of the `multi_objective` module (4167)
- Fix a typo in `QMCSampler` (4179)
- Introduce Optuna Dashboard in tutorial docs (4226)
- Remove `RedisStorage` from docstring (4232)
- Add the `BruteForceSampler` example to the document (4244)
- Improve the document of `BruteForceSampler` (4245)
- Fix an inline markup in distributed tutorial (4247)

Examples

- Add Dask example (https://github.com/optuna/optuna-examples/pull/46, thanks jrbourbeau!)
- Hotfix for botorch example (https://github.com/optuna/optuna-examples/pull/134)
- Replace `thop` with `fvcore` (https://github.com/optuna/optuna-examples/pull/136)
- Add `Optuna-distributed` to external projects (https://github.com/optuna/optuna-examples/pull/137)
- Remove the version constraint of GPyTorch (https://github.com/optuna/optuna-examples/pull/138)
- Fix a file path in `CONTRIBUTING.md` (https://github.com/optuna/optuna-examples/pull/139)
- Install `scikit-learn` instead of `sklearn` (https://github.com/optuna/optuna-examples/pull/141)
- Add constraint on `tensorflow` to `<2.11.0` (https://github.com/optuna/optuna-examples/pull/146)
- Specify botorch version (https://github.com/optuna/optuna-examples/pull/151)
- Pin numpy version to `1.23.x` for mxnet examples (https://github.com/optuna/optuna-examples/pull/154)

Tests

- Suppress warnings in `tests/test_distributions.py` (3912)
- Suppress warnings and minor code fixes in `tests/trial_tests` (3914)
- Reduce warning messages by `tests/study_tests/` (3915)
- Remove dynamic search space based objective from a parallel job test (3916)
- Remove all warning messages from `tests/integration_tests/test_sklearn.py` (3922)
- Remove out-of-range related warning messages from `MLflowCallback` and `WeightsAndBiasesCallback` (3923)
- Ignore `RuntimeWarning` when `nanmin` and `nanmax` take an array only containing nan values from `pruners_tests` (3924)
- Remove warning messages from test files for `pytorch_distributed` and `chainermn` modules (3927)
- Remove warning messages from `tests/integration_tests/test_lightgbm.py` (3944)
- Resolve warnings in `tests/visualization_tests/test_contour.py` (3954)
- Reduced warning messages from `tests/visualization_tests/test_slice.py` (3970, thanks jmsykes83!)
- Remove warning from a few visualizaiton tests (3989)
- Deselect integration tests in Tests CI (4013)
- Remove warnings from `tests/visualization_tests/test_optimization_history.py` (4024)
- Unset `PYTHONHASHSEED` for the hash-depedenet test (4031)
- Test: calling `study.tell` from another process (4039, thanks Abelarm!)
- Improve test for heartbeat: Add test for the case that trial state should be kept running (4055)
- Remove warnings in the test of Paretopereto front (4072)
- Remove matplotlib `get_cmap` warning from `tests/visualization_tests/test_param_importances.py` (4095)
- Reduce tests' `n_trials` for CI time reduction (4117)
- Skip `test_pop_waiting_trial_thread_safe` on RedisStorage (4119)
- Simplify the test of `BruteForceSampler` for infinite search space (4153)
- Add sep-CMA-ES in `parametrize_sampler` (4154)
- Fix a broken test for `dask.distributed` integration (4170)
- Add `DaskStorage` to existing storage tests (4176, thanks jrbourbeau!)
- Fix a test error in `test_catboost.py` (4190)
- Remove `test/integration_tests/test_sampler.py` (4204)

Code Fixes

- Refactor `_tell.py` (3841)
- Make log message user-friendly when objective returns a sequence of unsupported values (3868)
- Gather mask of None parameter in `TPESampler` (3886)
- Update cli tutorial (3902)
- Migrate CLI from `cliff` to `argparse` (4100)
- Enable mypy `--no-implicit-reexport` option (4110)
- Remove unused function: `find_any_distribution` (4127)
- Remove object inheritance from base classes (4161)
- Use mlflow 2.0.1 syntax (4173)
- Simplify implementation of `_preprocess_argv` in CLI (4187)
- Move `_solve_hssp` to `_hypervolume/utils.py` (4227, thanks jpbianchi!)
- Avoid to decode log string in `JournalRedisStorage` (4246)

Continuous Integration

- Hotfix `botorch` module by adding the version constraint of `gpytorch` (3950)
- Drop python 3.6 from integration CIs (3983)
- Use PyTorch 1.11 for consistency and fix a typo (3987)
- Support Python 3.11 (4018)
- Remove ` type: ignore` for mypy 0.981 (4019)
- Fix metric inconsistency between bayesmark plots and report (4077)
- Pin Ubuntu version to 20.04 in `Tests` and `Tests (Storage with server)` (4118)
- Add workflow to test Optuna with lower versions of constraints (4125)
- Mark some tests slow and ignore in pull request trigger (4138, thanks mist714!)
- Allow display names to be changed in benchmark scripts (Issue 4017) (4145)
- Disable scheduled workflow runs in forks (4159)
- Remove the CircleCI job `document` (4160)
- Stop running reproducibility tests on CI for PR (4162)
- Stop running reproducibility tests for coverage (4163)
- Add `workflow_dispatch` trigger to the integration tests (4166)
- [hotfix] Fix CI errors when using `mlflow==2.0.1` (4171)
- Add `fakeredis` in benchmark deps (4177)
- Fix `asv` speed benchmark (4185)
- Skip tests with minimum version for Python 3.10 and 3.11 (4199)
- Split normal tests and tests with minimum versions (4200)
- Update action/checkoutv2 -> v3 (4206)
- Update actions/stalev5 -> v6 (4208)
- Pin `botorch` to avoid CI failure (4228)
- Add the `pytest` dependency for asv (4243)

Other

- Bump up version number to 3.1.0.dev (3934)
- Remove the news section on README (3940)
- Add issue template for code fix (3968)
- Close stale issues immediately after labeling `stale` (4071)
- Remove `tox.ini` (4078)
- Replace gitter with GitHub Discussions (4083)
- Deprecate description-checked label (4090)
- Make `days-before-issue-stale` 300 days (4091)
- Unnecessary space removed (4109, thanks gonzaload!)
- Add note not to share pickle files in bug reports (4212)
- Update the description of optuna-dashboard on README (4217)
- Remove `optuna.TYPE_CHECKING` (4238)
- Bump up version to v3.1.0-b0 (4262)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

Abelarm, Alnusjaponica, HideakiImamura, amylase, belldandyxtq, c-bata, contramundum53, cross32768, erentknn, eukaryo, g-votte, gasin, gen740, gonzaload, halucinor, himkt, hvy, jmsykes83, jpbianchi, jrbourbeau, keisuke-umezawa, knshnb, mist714, ncclementi, not522, nzw0301, rene-rex, semiexp, shu65, sile, toshihikoyanase, wattlebirdaz, xadrianzetx, zaburo-ch

For fair comparison, all experiments were repeated 10 times, and the mean execution time was compared. Additional detailed benchmark settings include the following:
- Objective Function: Each trial consists of 10 parameters and 10 user attributes
- Storage: MySQL 8.0 (with PyMySQL)
- Sampler: [RandomSampler](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.RandomSampler.html#optuna.samplers.RandomSampler)
- Execution Environment: Kubernetes Pod with 5 cpus and 8Gi RAM

Please note, due to extensive execution time, the figure for v4.0.0 with 50,000 trials represents the average of 7 runs instead of 10.

<details>
<summary>Benchmark Script</summary>

python
import optuna
import time
import os
import numpy as np

optuna.logging.set_verbosity(optuna.logging.ERROR)
storage_url = "mysql+pymysql://user:password<ipaddr>:<port>/<dbname>"
n_repeat = 10

def objective(trial: optuna.Trial) -> float:
s = 0
for i in range(10):
trial.set_user_attr(f"attr{i}", "dummy user attribute")
s += trial.suggest_float(f"x{i}", -10, 10) ** 2
return s


def bench(n_trials):
elapsed = []
for i in range(n_repeat):
start = time.time()
study = optuna.create_study(
storage=storage_url,
sampler=optuna.samplers.RandomSampler()
)
study.optimize(objective, n_trials=n_trials, n_jobs=10)
elapsed.append(time.time() - start)
optuna.delete_study(study_name=study.study_name, storage=storage_url)
print(f"{np.mean(elapsed)=} {np.std(elapsed)=}")


for n_trials in [1000, 10000, 50000]:
bench(n_trials)


</details>


Five New Algorithms in OptunaHub (MO-CMA-ES, MOEA/D, etc.)

The following five new algorithms were added to OptunaHub!

- [Multi-objective CMA-ES (MO-CMA-ES)](https://hub.optuna.org/samplers/mocma/)by y0z
- [MOEA/D sampler](https://hub.optuna.org/samplers/moead/) by hrntsm
- [MAB Epsilon-Greedy Sampler](https://hub.optuna.org/samplers/mab_epsilon_greedy/) by ryota717
- [NSGAII sampler with Initial Trials](https://hub.optuna.org/samplers/nsgaii_with_initial_trials/) by hrntsm
- [CMA-ES with User Prior](https://hub.optuna.org/samplers/user_prior_cmaes/) by nabenabe0928

[MO-CMA-ES](https://hub.optuna.org/samplers/mocma/) is an extension of CMA-ES for multi-objective optimization. Its search mechanism is based on multiple (1+1)-CMA-ES and inherits good invariance properties from CMA-ES, such as invariance against rotation of the search space.

<img width="400" alt="mocmaes" src="https://github.com/user-attachments/assets/5289d2fd-81c9-40a4-b575-daf5d3cedc3f">

[MOEA/D](https://hub.optuna.org/samplers/moead/) solves a multi-objective optimization problem by decomposing it into multiple single-objective problems. It allows for the maintenance of a good diversity of solutions during optimization. Please take a look at [the article](https://medium.com/optuna/an-introduction-to-moea-d-and-examples-of-multi-objective-optimization-comparisons-8630565a4e89) from Hiroaki NATSUME(hrntsm) for more details.

<img width="400" alt="moead" src="https://github.com/user-attachments/assets/3ab83a64-b789-425b-800c-2c1f09e5a752">

Enhancements

- Update sklearn.py by addind catch to OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/163, thanks muhlbach!)
- Reduce `SELECT` statements by passing `study_id` to `check_and_add` in `TrialParamModel` (5702)
- Introduce `UPSERT` in `set_trial_user_attr` (5703)
- Reduce `SELECT` statements of `_CachedStorage.get_all_trials` by fixing filtering conditions (5704)
- Reduce `SELECT` statements by removing unnecessary distribution compatibility check in `set_trial_param()` (5709)
- Introduce `UPSERT` in `set_trial_system_attr` (5741)

Bug Fixes

- Accept `Mapping` as `param_distributions` in `OptunaSearchCV` (https://github.com/optuna/optuna-integration/pull/172, thanks yu9824!)
- Allow use of `OptunaSearchCV` with `cross_val_predict` (https://github.com/optuna/optuna-integration/pull/174, thanks yu9824!)
- Fix `GPSampler`'s suggestion failure within `torch.no_grad()` context manager (5671, thanks kAIto47802!)
- Fix a concurrency issue in `GPSampler` (5737)
- Fix a concurrency issue in `QMCSampler` (5740)

Installation

- Drop the support for Python 3.7 and update package metadata for Python 3.13 support (5727)
- Remove dependency specifier for installing SciPy (5736)

Documentation

- Add badges of PyPI and Conda Forge (https://github.com/optuna/optuna-integration/pull/167)
- Update news (5655)
- Update news section in `README.md` (5657)
- Add `InMemoryStorage` to document (5672, thanks kAIto47802!)
- Add the news about blog of `JournalStorage` to `README.md` (5674)
- Fix broken link in the artifacts tutorial (5677, thanks kAIto47802!)
- Add `pandas` installation guide to RDB tutorial (5685, thanks kAIto47802!)
- Add installation guide to multi-objective tutorial (5686, thanks kAIto47802!)
- Update document and notice interoperability between NSGA-II and Pruners (5688)
- Fix typo in `EMMREvaluator` (5694)
- Fix `RegretBoundEvaluator` document (5696)
- Update news section in `README.md` (5705)
- Add link to related blog post in `emmr.py` (5707)
- Update FAQ entry for model preservation using Optuna Artifact (5716, thanks chitvs!)
- Escape `\D` for Python 3.12 with sphinx build (5735)
- Avoid using functions in `sphinx_gallery_conf` to remove document build error with Python 3.12 (5738)
- Remove all `generated` directories in `docs/source` recursively (5739)
- Add information about `AutoSampler` to the docs (5745)

Examples

- Update CI config for `fastai` (https://github.com/optuna/optuna-examples/pull/279)
- Introduce `optuna.artifacts` to the PyTorch checkpoint example (https://github.com/optuna/optuna-examples/pull/280, thanks kAIto47802!)
- Fix inline code in README files in `kubernetes` directory (https://github.com/optuna/optuna-examples/pull/282)
- Test ray example with Python 3.12 (https://github.com/optuna/optuna-examples/pull/284)
- Add a link to molecule LLM notebook (https://github.com/optuna/optuna-examples/pull/285)
- Fix syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/286)
- Another fix for syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/287)
- Add checking for Python 3.12 (https://github.com/optuna/optuna-examples/pull/288, thanks kAIto47802!)
- Add Python 3.12 to tfkeras CI and remove warning message (https://github.com/optuna/optuna-examples/pull/289)
- Add Python 3.12 to `lightgbm` CI (https://github.com/optuna/optuna-examples/pull/290)
- Remove Python 3.7 from the workflow (https://github.com/optuna/optuna-examples/pull/291)

Code Fixes

- Add a comment for an unexpected bug in `CategoricalDistribution` (5683)
- Add more information about the hack in `WFG` (5687)
- Simplify type annotations to `_imports.py` (5692, thanks Prabhat-Thapa45!)
- Use `__future__.annotations` in `optuna/_experimental.py` (5714, thanks Jonathan43!)
- Use `__future__.annotations` in `tests/importance_tests/fanova_tests/test_tree.py` (5731, thanks guisp03!)
- Resolve TODO comments related to dropping Python 3.7 support (5734)

Continuous Integration

- Fix CI for `fastaiv2` (https://github.com/optuna/optuna-integration/pull/164)
- Fix for mypy (https://github.com/optuna/optuna-integration/pull/166)
- Fix mlflow integration for CI (https://github.com/optuna/optuna-integration/pull/168)
- Update CI to support Python 3.12 (https://github.com/optuna/optuna-integration/pull/170, thanks kAIto47802!)
- Update artifact version to v4 (https://github.com/optuna/optuna-integration/pull/176)
- Add a tri-objective problem to speed benchmarking (5635)
- Bump `actions/download-artifact` from 2 to 4.1.7 in `/.github/workflows` (5660)
- Update MySQL version in CI from 5.7 to 8 (5673)
- Remove performance benchmarks (5675)
- Update PostgreSQL version in CI to latest (5676)
- Update the version of Python used in the checks from 3.8 to 3.11 (5684)
- Run tests with Python 3.13 (5691)
- Fix for mypy (5706)
- [hotfix] Add a version constraint on fakeredis (5726)
- Update Python versions used in CI workflow (5728)
- Update sphinx build's `upload-artifact` version (5744)

Other

- Bump up version number to 4.1.0.dev (https://github.com/optuna/optuna-integration/pull/159)
- Update supported Python versions (https://github.com/optuna/optuna-integration/pull/175)
- Bump up to v4.1.0 (https://github.com/optuna/optuna-integration/pull/179)
- Add pre-commit config (5408)
- Add link to MOEA/D blog post in News (5719)
- Include all test files in `sdist` by updating `MANIFEST.in` (5720)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, Jonathan43, Prabhat-Thapa45, c-bata, chitvs, contramundum53, eukaryo, gen740, guisp03, kAIto47802, muhlbach, nabenabe0928, not522, nzw0301, porink0424, toshihikoyanase, y0z, yu9824

This is the release note of v4.2.1. This release includes a bug fix addressing an issue where Optuna was unable to import if an older version of the grpcio package was installed.

Bug

- [backport] Use `_LazyImport` for grpcio package (5965)

Other

- Bump up version number to v4.2.1 (5964)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.
c-bata HideakiImamura nabenabe0928

This is the release note of [v4.2.0](https://github.com/optuna/optuna/releases/tag/v4.2.0). In conjunction with the Optuna release, OptunaHub 0.2.0 is released. Please refer to [the release note of OptunaHub 0.2.0](https://github.com/optuna/optunahub/releases/tag/v0.2.0) for more details.

Highlights of this release include:

- 🚀gRPC Storage Proxy for Scalable Hyperparameter Optimization
- 🤖 SMAC3: Support for New State-of-the-art Optimization Algorithm by AutoML.org (automl)
- 📁 OptunaHub Now Supports Benchmark Functions
- 🧑‍💻 Gaussian Process-Based Bayesian Optimization with Inequality Constraints
- 🧑‍💻 c-TPE: Support Constrained TPESampler

Highlights

gRPC Storage Proxy for Scalable Hyperparameter Optimization


The gRPC storage proxy is a feature designed to support large-scale distributed optimization. As shown in the diagram below, gRPC storage proxy sits between the optimization workers and the database server, proxying the calls of Optuna’s storage APIs.

<img width="1241" alt="grpc-proxy" src="https://github.com/user-attachments/assets/00220bc6-75da-4ee3-b9ff-a2b3440ac207" />


In large-scale distributed optimization settings where hundreds to thousands of workers are operating, placing a gRPC storage proxy for every few tens can significantly reduce the load on the RDB server which would otherwise be a single point of failure. The gRPC storage proxy enables sharing the cache about Optuna studies and trials, which can further mitigate load. Please refer to [the official documentation](https://optuna.readthedocs.io/en/latest/reference/generated/optuna.storages.GrpcStorageProxy.html#optuna.storages.GrpcStorageProxy) for further details on how to utilize gRPC storage proxy.

SMAC3: Random Forest-Based Bayesian Optimization Developed by AutoML.org
[SMAC3](https://github.com/automl/SMAC3) is a hyperparameter optimization framework developed by [AutoML.org](http://automl.org/), one of the most influential AutoML research groups. The Optuna-compatible SMAC3 sampler is now available thanks to the contribution to OptunaHub by Difan Deng (dengdifan), one of the core members of AutoML.org. We can now use the method widely used in AutoML research and real-world applications from Optuna.

python
pip install optunahub smac
import optuna
import optunahub
from optuna.distributions import FloatDistribution

def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", -10, 10)
y = trial.suggest_float("y", -10, 10)
return x**2 + y**2

smac_mod = optunahub.load_module("samplers/smac_sampler")
n_trials = 100
sampler = smac_mod.SMACSampler(
{"x": FloatDistribution(-10, 10), "y": FloatDistribution(-10, 10)},
n_trials=n_trials,
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=n_trials)


Please refer to https://hub.optuna.org/samplers/smac_sampler/ for more details.

OptunaHub Now Supports Benchmark Functions
Benchmarking the performance of optimization algorithms is an essential process indispensable to the research and development of algorithms. The newly added OptunaHub Benchmarks in the latest version v0.2.0 of [optunahub](https://hub.optuna.org/) is a new feature for Optuna users to conduct benchmarks conveniently.

python
pip install optunahub>=4.2.0 scipy torch
import optuna
import optunahub

bbob_mod = optunahub.load_module("benchmarks/bbob")
smac_mod = optunahub.load_module("samplers/smac_sampler")
sphere2d = bbob_mod.Problem(function_id=1, dimension=2)

n_trials = 100
studies = []
for study_name, sampler in [
("random", optuna.samplers.RandomSampler(seed=1)),
("tpe", optuna.samplers.TPESampler(seed=1)),
("cmaes", optuna.samplers.CmaEsSampler(seed=1)),
("smac", smac_mod.SMACSampler(sphere2d.search_space, n_trials, seed=1)),
]:
study = optuna.create_study(directions=sphere2d.directions,
sampler=sampler, study_name=study_name)
study.optimize(sphere2d, n_trials=n_trials)
studies.append(study)

optuna.visualization.plot_optimization_history(studies).show()


In the above sample code, we compare and display the performance of the four kinds of samplers using a two-dimensional Sphere function, which is part of a group of benchmark functions widely used in the black-box optimization research community known as [Blackbox Optimization Benchmarking (BBOB)](https://hub.optuna.org/benchmarks/bbob/).

<img width="1250" alt="bbob" src="https://github.com/user-attachments/assets/89b09090-dd37-4679-9fa8-af987f138dcc" />


Gaussian Process-Based Bayesian Optimization with Inequality Constraints
We worked on its extension and adapted `GPSampler` to constrained optimization in Optuna v4.2.0 since Gaussian process-based Bayesian optimization is a very popular method in various research fields such as aircraft engineering and materials science. We show the basic usage below.

python
pip install optuna>=4.2.0 scipy torch
import numpy as np
import optuna

def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", 0.0, 2 * np.pi)
y = trial.suggest_float("y", 0.0, 2 * np.pi)
c = float(np.sin(x) * np.sin(y) + 0.95)
trial.set_user_attr("c", c)
return float(np.sin(x) + y)

def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
return (trial.user_attrs["c"],)

sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)


Please try out `GPSampler` for constrained optimization especially when only a small number of trials are available!

c-TPE: Support Constrained TPESampler


Although Optuna has supported constrained optimization for `TPESampler`, which is the default Optuna sampler, since v3.0.0, its algorithm design and performance comparison have not been verified academically. OptunaHub now supports [c-TPE](https://arxiv.org/abs/2211.14411), which is another constrained optimization method for `TPESampler`. Importantly, the algorithm design and its performance comparison are publicly reviewed to be accepted to IJCAI, a top-tier AI international conference. Please refer to https://hub.optuna.org/samplers/ctpe/ for details.

New Features

- Enable `GPSampler` to support constraint functions (5715)
- Update output format options in CLI to include the `value` choice (5822, thanks iamarunbrahma!)
- Add gRPC storage proxy server and client (5852)

Enhancements

- Introduce client-side cache in `GrpcStorageProxy` (5872)

Bug Fixes

- Fix CI (https://github.com/optuna/optuna-integration/pull/185)
- Fix ticks in Matplotlib contour plot (5778, thanks sulan!)
- Adding check in `cli.py` to handle an empty database (5828, thanks willdavidson05!)
- Avoid the input validation fail in Wilcoxon signed ranked test for Scipy 1.15 (5912)
- Fix the default sampler of `load_study` function (5924)

Documentation

- Update OptunaHub example in README (5763)
- Update `distributions.rst` to list deprecated distribution classes (5764)
- Remove deprecation comment for `step` in `IntLogUniformDistribution` (5767)
- Update requirements for OptunaHub in README (5768)
- Use inline code rather than italic for `step` (5769)
- Add notes to `ask_and_tell` tutorial - batch optimization recommendations (5817, thanks SimonPop!)
- Fix the explanation of returned values of `get_trial_params` (5820)
- Introduce `sphinx-notfound-page` for better 404 page (5898)
- Follow-up 5872: Update the docstring of `run_grpc_proxy_server` (5914)
- Modify doc-string of gRPC-related modules (5916)

Examples

- Adapt docker recipes to Python 3.11 (https://github.com/optuna/optuna-examples/pull/292)
- Add version constraint for `wandb` (https://github.com/optuna/optuna-examples/pull/293)

Tests

- Add unit tests for `retry_history` method in `RetryFailedTrialCallback` (5865, thanks iamarunbrahma!)
- Add tests for value format in CLI (5866)
- Import grpc lazy to fix the CI (5878)

Code Fixes

- Fix annotations for `distributions.py` (5755, thanks KannanShilen!)
- Simplify type annotations for `tests/visualization_tests/test_pareto_front.py` (5756, thanks boringbyte!)
- Fix type annotations for `test_hypervolume_history.py` (5760, thanks boringbyte!)
- Simplify type annotations for `tests/test_cli.py` (5765, thanks boringbyte!)
- Simplify type annotations for `tests/test_distributions.py` (5773, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/test_qmc.py` (5775, thanks boringbyte!)
- Simplify type annotations for `tests/sampler_tests/tpe_tests/test_sampler.py` (5779, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py` (5781, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/tpe_tests/test_parzen_estimator.py` (5782, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/journal_tests/test_journal.py` (5783, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/rdb_tests/create_db.py` (5784, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/rdb_tests/` (5785, thanks boringbyte!)
- Simplify type annotations for `tests/test_deprecated.py` (5786, thanks boringbyte!)
- Simplify type annotations for `tests/test_convert_positional_args.py` (5787, thanks boringbyte!)
- Simplify type annotations for `tests/importance_tests/test_init.py` (5790, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_storages.py` (5791, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_heartbeat.py` (5792, thanks boringbyte!)
- Simplify type annotations `optuna/cli.py` (5793, thanks willdavidson05!)
- Simplify type annotations for `tests/trial_tests/test_frozen.py` (5794, thanks boringbyte!)
- Simplify type annotations for `tests/trial_tests/test_trial.py` (5795, thanks boringbyte!)
- Simplify type annotations for `tests/trial_tests/test_trials.py` (5796, thanks boringbyte!)
- Refactor some funcs in NSGA-III (5798)
- Simplify type annotations `optuna/_transform.py` (5799, thanks JLX0!)
- Make the assertion messages in `test_trial.py` readable (5800)
- Simplify type annotations for `tests/pruners_tests/test_hyperband.py` (5801, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_median.py` (5802, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_patient.py` (5803, thanks boringbyte!)
- Use `study.ask()` in tests instead of `create_new_trial` (5807, thanks unKnownNG!)
- Simplify type annotations for `tests/pruners_tests/test_percentile.py` (5808, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_successive_halving.py` (5809, thanks boringbyte!)
- Simplify type annotations for `tests/study_tests/test_optimize.py` (5810, thanks boringbyte!)
- Simplify type annotations for `tests/hypervolume_tests/test_hssp.py` (5812, thanks boringbyte!)
- Refactor the MOTPE split (5813)
- Simplify type annotations for `optuna/_callbacks.py` (5818, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/_random.py` (5819, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/_gp/sampler.py` (5823, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_sbx.py` (5824, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_spx.py` (5825, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_undx.py` (5826, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_vsbx.py` (5827, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossover.py` (5831, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/testing/threading.py` (5832, thanks boringbyte!)
- Simplify type annotations for `optuna/pruners/_patient.py` (5833, thanks boringbyte!)
- Use `study.ask()` in `tests/pruners_tests/test_percentile.py` (5834, thanks fusawa-yugo!)
- Simplify type annotations for `optuna/search_space/group_decomposed.py` (5836, thanks boringbyte!)
- Simplify type annotations for `optuna/search_space/intersection.py` (5837, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_base.py` (5838, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_redis.py` (5840, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_storage.py` (5841, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_dataframe.py` (5842, thanks boringbyte!)
- Fix logger.warn() deprecation warning in GP module (5843, thanks iamarunbrahma!)
- Simplify type annotations for `optuna/study/_optimize.py` (5844, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_tell.py` (5845, thanks boringbyte!)
- Fix `mypy` errors due to `numpy` 2.2.0 (5848)
- Simplify type annotations for `optuna/visualization/matplotlib/_contour.py` (5851, thanks boringbyte!)
- Refactor the fix for MyPy errors due to NumPy v2.2.0 (5853)
- Simplify type annotations for `optuna/visualization/matplotlib/_parallel_coordinate.py` (5854, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_param_importances.py` (5855, thanks boringbyte!)
- Use `study.ask()` in `tests/pruners_tests/test_successive_halving.py` (5856, thanks willdavidson05!)
- Simplify type annotations for `optuna/visualization/matplotlib/_pareto_front.py` (5857, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_rank.py` (5858, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_slice.py` (5859, thanks boringbyte!)
- Refactor plot contour (5867)
- Refactor MOTPE weighting (5871)
- Simplify type annotations for `optuna/visualization/_utils.py` (5876, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_contour.py` (5877, thanks boringbyte!)
- Simplify type annotations for `optuna/_gp/gp.py` (5879, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_tell.py` (5880, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_heartbeat.py` (5882, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/storage.py` (5883, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.a.py` (5884, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.c.py` (5885, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.d.py` (5886, thanks boringbyte!)
- Simplify type annotations for `optuna/_deprecated.py` (5887, thanks boringbyte!)
- Simplify type annotations for `optuna/_experimental.py` (5888, thanks boringbyte!)
- Simplify type annotations for `optuna/_imports.py` (5889, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_slice.py` (5894, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_parallel_coordinate.py` (5895, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_rank.py` (5896, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_param_importances.py` (5897, thanks boringbyte!)
- Simplify type annotations for `optuna/_callbacks.py` (5899, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_file.py` (5900, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_with_server.py` (5901, thanks boringbyte!)
- Simplify type annotations for `tests/test_multi_objective.py` (5902, thanks boringbyte!)
- Simplify type annotations for `tests/artifacts_tests/test_gcs.py` (5903, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/test_grid.py` (5904, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_dataframe.py` (5905, thanks boringbyte!)
- Simplify type annotations for `tests/visualization_tests/test_optimization_history.py` (5906, thanks boringbyte!)
- Simplify type annotations for `tests/visualization_tests/test_intermediate_plot.py` (5907, thanks boringbyte!)
- Simplify type annotations for multiple test files in `tests/visualization_tests/` (5908, thanks boringbyte!)
- Avoid the port number conflict in the copy study tests (5913)
- Simplify annotations in `tests/study_tests/test_study.py` (5923, thanks sawa3030!)

Continuous Integration

- Fix a GitHub action workflow for publishing to PyPI (https://github.com/optuna/optuna-integration/pull/181)
- Fix CI by adding a version constraint (https://github.com/optuna/optuna-integration/pull/186)
- Rename `test_cache_is_invalidated` and remove `assert study._thread_local.cached_all_trials is None` (5733)
- Use Python 3.12 for docs build CI (5742)
- Fix a GitHub action workflow for publishing to PyPI (5759)
- Install older `kaleido` to fix CI errors (5771)
- Add Python 3.12 to the Docker image CI (5789)
- Move `mypy` related entries in setup.cfg to `pyproject.toml` (5861)

Other

- Bump up version number to v4.2.0.dev (https://github.com/optuna/optuna-integration/pull/178)
- Bump up version number to 4.2.0 (https://github.com/optuna/optuna-integration/pull/191)
- Add `CITATION.cff` (5746)
- Update news to add the autosampler article (5748)
- Bump the version up to v4.2.0.dev (5752)
- Update the news section for Optuna 4.1 release (5758)
- Update pre-commit configuration file (5847)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, JLX0, KannanShilen, SimonPop, boringbyte, c-bata, fusawa-yugo, gen740, himkt, iamarunbrahma, kAIto47802, ktns, mist714, nabenabe0928, not522, nzw0301, porink0424, sawa3030, sulan, unKnownNG, willdavidson05, y0z