Optuna

$ python dask_simple.py


A brand-new Redis storage

We have replaced the Redis storage backend with a JournalStorage-based one. The experimental `RedisStorage` class has been removed in v3.1. The following example shows how to use the new `JournalRedisStorage` class.

python
import optuna
from optuna.storages import JournalStorage, JournalRedisStorage

def objective(trial):
…

storage = JournalStorage(JournalRedisStorage("redis://localhost:6379"))
study = optuna.create_study(storage=storage)
study.optimize(objective)


Sampler for brute-force search

`BruteForceSampler`, a new sampler for brute-force search, tries all combinations of parameters. In contrast to `GridSampler`, it does not require passing the search space as an argument and works even with branches. This sampler constructs the search space with the define-by-run style, so it works by just adding `sampler=optuna.samplers.BruteForceSampler()`.

python
import optuna

def objective(trial):
c = trial.suggest_categorical("c", ["float", "int"])
if c == "float":
return trial.suggest_float("x", 1, 3, step=0.5)
elif c == "int":
a = trial.suggest_int("a", 1, 3)
b = trial.suggest_int("b", a, 3)
return a + b

study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler())
study.optimize(objective)


Breaking Changes

- Allow users to call `study.optimize()` in multiple threads (4068)
- Use all trials in `TPESampler` even when `multivariate=True` (4079)
- Drop Python 3.6 (4150)
- Remove `RedisStorage` (4156)
- Deprecate `set_system_attr` in `Study` and `Trial` (4188)
- Deprecate `system_attrs` in `Study` class (4250)

New Features

- Add Dask integration (2023, thanks jrbourbeau!)
- Add journal-style log storage (3854)
- Support CMA-ES with margin in `CmaEsSampler` (4016)
- Add journal redis storage (4086)
- Add device argument to `BoTorchSampler` (4101)
- Add the feature to `JournalStorage` of Redis backend to resume from a snapshot (4102)
- Added `user_attrs` to print by optuna studies in `cli.py` (4129, thanks gonzaload!)
- Add `BruteForceSampler` (4132, thanks semiexp!)
- Add `__getstate__` and `__setstate__` to `RedisStorage` (4135, thanks shu65!)
- Support pickle in `JournalRedisStorage` (4139, thanks shu65!)
- Support for `qNoisyExpectedHypervolumeImprovement` acquisition function from `BoTorch` (Issue4014) (4186)

Enhancements

- Change the log message format for failed trials (3857, thanks erentknn!)
- Move default logic of `get_trial_id_from_study_id_trial_number()` method to BaseStorage (3910)
- Fix the data migration script for v3 release (4020)
- Convert `search_space` values of `GridSampler` explicitly (4062)
- Add single exception catch to study optimize (4098)
- Add validation in `enqueue_trial` (4126)
- Speed up `tests/samplers_tests/test_nsgaii.py::test_fast_non_dominated_sort_with_constraints` (4128, thanks mist714!)
- Add getstate and setstate to journal storage (4130, thanks shu65!)
- Support `None` in slice plot (4133, thanks belldandyxtq!)
- Add marker to matplotlib `plot_intermediate_value` (4134, thanks belldandyxtq!)
- Cache `study.directions` to reduce the number of `get_study_directions()` calls (4146)
- Add an in-memory cache in `Trial` class (4240)

Bug Fixes

- Fix infinite loop bug in `TPESampler` (3953, thanks gasin!)
- Fix `GridSampler` (3957)
- Fix an import error of `sqlalchemy.orm.declarative_base` (3967)
- Skip to add `intermediate_value_type` and `value_type` columns if exists (4015)
- Fix duplicated sampling of `SkoptSampler` (4023)
- Avoid parse errors of `datetime.isoformat` strings (4025)
- Fix a concurrency bug of JournalStorage `set_trial_state_values` (4033)
- Specify object type to numpy array init to avoid unintended str cast (4035)
- Make `TPESampler` reproducible (4056)
- Fix bugs in `constant_liar` option (4073)
- Add a flush to `JournalFileStorage.append_logs` (4076)
- Add a lock to `MLflowCallback` (4097)
- Reject deprecated distributions in `OptunaSearchCV` (4120)
- Stop using hash function in `_get_bracket_id` in `HyperbandPruner` (4131, thanks zaburo-ch!)
- Validation for the parameter enqueued in `to_internal_repr` of `FloatDistribution` and `IntDistribution` (4137)
- Fix `PartialFixedSampler` to handle `None` correctly (4147, thanks halucinor!)
- Fix the bug of JournalFileStorage on Windows (4151)
- Fix CmaEs system attribution key (4184)

Installation

- Replace `thop` with `fvcore` (3906)
- Use the latest stable scipy (3959, thanks gasin!)
- Remove GPyTorch version constraint (3986)
- Make typing_extensions optional (3990)
- Add version constraint on `importlib-metadata` (4036)
- Add a version constraint of `matplotlib` (4044)

Documentation

- Update cli tutorial (3902)
- Replace `thop` with `fvcore` (3906)
- Slightly improve docs of `FrozenTrial` (3943)
- Refine docs in `BaseStorage` (3948)
- Remove "Edit on GitHub" button from readthedocs (3952)
- Mention restoring sampler in saving/resuming tutorial (3992)
- Use `log_loss` instead of deprecated `log` since `sklearn` 1.1 (3993)
- Fix script path in benchmarks/README.md (4021)
- Ignore `ConvergenceWarning` in the ask-and-tell tutorial (4032)
- Update docs to let users know the concurrency problem on SQLite3 (4034)
- Fix the time complexity of `NSGAIISampler` (4045)
- Fix sampler comparison table (4082)
- Add `BruteForceSampler` in the samplers' list (4152)
- Remove markup from NaN in FAQ (4155)
- Remove the document of the `multi_objective` module (4167)
- Fix a typo in `QMCSampler` (4179)
- Introduce Optuna Dashboard in tutorial docs (4226)
- Remove `RedisStorage` from docstring (4232)
- Add the `BruteForceSampler` example to the document (4244)
- Improve the document of `BruteForceSampler` (4245)
- Fix an inline markup in distributed tutorial (4247)

Examples

- Add Dask example (https://github.com/optuna/optuna-examples/pull/46, thanks jrbourbeau!)
- Hotfix for botorch example (https://github.com/optuna/optuna-examples/pull/134)
- Replace `thop` with `fvcore` (https://github.com/optuna/optuna-examples/pull/136)
- Add `Optuna-distributed` to external projects (https://github.com/optuna/optuna-examples/pull/137)
- Remove the version constraint of GPyTorch (https://github.com/optuna/optuna-examples/pull/138)
- Fix a file path in `CONTRIBUTING.md` (https://github.com/optuna/optuna-examples/pull/139)
- Install `scikit-learn` instead of `sklearn` (https://github.com/optuna/optuna-examples/pull/141)
- Add constraint on `tensorflow` to `<2.11.0` (https://github.com/optuna/optuna-examples/pull/146)
- Specify botorch version (https://github.com/optuna/optuna-examples/pull/151)
- Pin numpy version to `1.23.x` for mxnet examples (https://github.com/optuna/optuna-examples/pull/154)

Tests

- Suppress warnings in `tests/test_distributions.py` (3912)
- Suppress warnings and minor code fixes in `tests/trial_tests` (3914)
- Reduce warning messages by `tests/study_tests/` (3915)
- Remove dynamic search space based objective from a parallel job test (3916)
- Remove all warning messages from `tests/integration_tests/test_sklearn.py` (3922)
- Remove out-of-range related warning messages from `MLflowCallback` and `WeightsAndBiasesCallback` (3923)
- Ignore `RuntimeWarning` when `nanmin` and `nanmax` take an array only containing nan values from `pruners_tests` (3924)
- Remove warning messages from test files for `pytorch_distributed` and `chainermn` modules (3927)
- Remove warning messages from `tests/integration_tests/test_lightgbm.py` (3944)
- Resolve warnings in `tests/visualization_tests/test_contour.py` (3954)
- Reduced warning messages from `tests/visualization_tests/test_slice.py` (3970, thanks jmsykes83!)
- Remove warning from a few visualizaiton tests (3989)
- Deselect integration tests in Tests CI (4013)
- Remove warnings from `tests/visualization_tests/test_optimization_history.py` (4024)
- Unset `PYTHONHASHSEED` for the hash-depedenet test (4031)
- Test: calling `study.tell` from another process (4039, thanks Abelarm!)
- Improve test for heartbeat: Add test for the case that trial state should be kept running (4055)
- Remove warnings in the test of Paretopereto front (4072)
- Remove matplotlib `get_cmap` warning from `tests/visualization_tests/test_param_importances.py` (4095)
- Reduce tests' `n_trials` for CI time reduction (4117)
- Skip `test_pop_waiting_trial_thread_safe` on RedisStorage (4119)
- Simplify the test of `BruteForceSampler` for infinite search space (4153)
- Add sep-CMA-ES in `parametrize_sampler` (4154)
- Fix a broken test for `dask.distributed` integration (4170)
- Add `DaskStorage` to existing storage tests (4176, thanks jrbourbeau!)
- Fix a test error in `test_catboost.py` (4190)
- Remove `test/integration_tests/test_sampler.py` (4204)

Code Fixes

- Refactor `_tell.py` (3841)
- Make log message user-friendly when objective returns a sequence of unsupported values (3868)
- Gather mask of None parameter in `TPESampler` (3886)
- Update cli tutorial (3902)
- Migrate CLI from `cliff` to `argparse` (4100)
- Enable mypy `--no-implicit-reexport` option (4110)
- Remove unused function: `find_any_distribution` (4127)
- Remove object inheritance from base classes (4161)
- Use mlflow 2.0.1 syntax (4173)
- Simplify implementation of `_preprocess_argv` in CLI (4187)
- Move `_solve_hssp` to `_hypervolume/utils.py` (4227, thanks jpbianchi!)
- Avoid to decode log string in `JournalRedisStorage` (4246)

Continuous Integration

- Hotfix `botorch` module by adding the version constraint of `gpytorch` (3950)
- Drop python 3.6 from integration CIs (3983)
- Use PyTorch 1.11 for consistency and fix a typo (3987)
- Support Python 3.11 (4018)
- Remove ` type: ignore` for mypy 0.981 (4019)
- Fix metric inconsistency between bayesmark plots and report (4077)
- Pin Ubuntu version to 20.04 in `Tests` and `Tests (Storage with server)` (4118)
- Add workflow to test Optuna with lower versions of constraints (4125)
- Mark some tests slow and ignore in pull request trigger (4138, thanks mist714!)
- Allow display names to be changed in benchmark scripts (Issue 4017) (4145)
- Disable scheduled workflow runs in forks (4159)
- Remove the CircleCI job `document` (4160)
- Stop running reproducibility tests on CI for PR (4162)
- Stop running reproducibility tests for coverage (4163)
- Add `workflow_dispatch` trigger to the integration tests (4166)
- [hotfix] Fix CI errors when using `mlflow==2.0.1` (4171)
- Add `fakeredis` in benchmark deps (4177)
- Fix `asv` speed benchmark (4185)
- Skip tests with minimum version for Python 3.10 and 3.11 (4199)
- Split normal tests and tests with minimum versions (4200)
- Update action/checkoutv2 -> v3 (4206)
- Update actions/stalev5 -> v6 (4208)
- Pin `botorch` to avoid CI failure (4228)
- Add the `pytest` dependency for asv (4243)

Other

- Bump up version number to 3.1.0.dev (3934)
- Remove the news section on README (3940)
- Add issue template for code fix (3968)
- Close stale issues immediately after labeling `stale` (4071)
- Remove `tox.ini` (4078)
- Replace gitter with GitHub Discussions (4083)
- Deprecate description-checked label (4090)
- Make `days-before-issue-stale` 300 days (4091)
- Unnecessary space removed (4109, thanks gonzaload!)
- Add note not to share pickle files in bug reports (4212)
- Update the description of optuna-dashboard on README (4217)
- Remove `optuna.TYPE_CHECKING` (4238)
- Bump up version to v3.1.0-b0 (4262)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

Abelarm, Alnusjaponica, HideakiImamura, amylase, belldandyxtq, c-bata, contramundum53, cross32768, erentknn, eukaryo, g-votte, gasin, gen740, gonzaload, halucinor, himkt, hvy, jmsykes83, jpbianchi, jrbourbeau, keisuke-umezawa, knshnb, mist714, ncclementi, not522, nzw0301, rene-rex, semiexp, shu65, sile, toshihikoyanase, wattlebirdaz, xadrianzetx, zaburo-ch

For fair comparison, all experiments were repeated 10 times, and the mean execution time was compared. Additional detailed benchmark settings include the following:
- Objective Function: Each trial consists of 10 parameters and 10 user attributes
- Storage: MySQL 8.0 (with PyMySQL)
- Sampler: [RandomSampler](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.RandomSampler.html#optuna.samplers.RandomSampler)
- Execution Environment: Kubernetes Pod with 5 cpus and 8Gi RAM

Please note, due to extensive execution time, the figure for v4.0.0 with 50,000 trials represents the average of 7 runs instead of 10.

<details>
<summary>Benchmark Script</summary>

python
import optuna
import time
import os
import numpy as np

optuna.logging.set_verbosity(optuna.logging.ERROR)
storage_url = "mysql+pymysql://user:password<ipaddr>:<port>/<dbname>"
n_repeat = 10

def objective(trial: optuna.Trial) -> float:
s = 0
for i in range(10):
trial.set_user_attr(f"attr{i}", "dummy user attribute")
s += trial.suggest_float(f"x{i}", -10, 10) ** 2
return s


def bench(n_trials):
elapsed = []
for i in range(n_repeat):
start = time.time()
study = optuna.create_study(
storage=storage_url,
sampler=optuna.samplers.RandomSampler()
)
study.optimize(objective, n_trials=n_trials, n_jobs=10)
elapsed.append(time.time() - start)
optuna.delete_study(study_name=study.study_name, storage=storage_url)
print(f"{np.mean(elapsed)=} {np.std(elapsed)=}")


for n_trials in [1000, 10000, 50000]:
bench(n_trials)


</details>


Five New Algorithms in OptunaHub (MO-CMA-ES, MOEA/D, etc.)

The following five new algorithms were added to OptunaHub!

- [Multi-objective CMA-ES (MO-CMA-ES)](https://hub.optuna.org/samplers/mocma/)by y0z
- [MOEA/D sampler](https://hub.optuna.org/samplers/moead/) by hrntsm
- [MAB Epsilon-Greedy Sampler](https://hub.optuna.org/samplers/mab_epsilon_greedy/) by ryota717
- [NSGAII sampler with Initial Trials](https://hub.optuna.org/samplers/nsgaii_with_initial_trials/) by hrntsm
- [CMA-ES with User Prior](https://hub.optuna.org/samplers/user_prior_cmaes/) by nabenabe0928

[MO-CMA-ES](https://hub.optuna.org/samplers/mocma/) is an extension of CMA-ES for multi-objective optimization. Its search mechanism is based on multiple (1+1)-CMA-ES and inherits good invariance properties from CMA-ES, such as invariance against rotation of the search space.

<img width="400" alt="mocmaes" src="https://github.com/user-attachments/assets/5289d2fd-81c9-40a4-b575-daf5d3cedc3f">

[MOEA/D](https://hub.optuna.org/samplers/moead/) solves a multi-objective optimization problem by decomposing it into multiple single-objective problems. It allows for the maintenance of a good diversity of solutions during optimization. Please take a look at [the article](https://medium.com/optuna/an-introduction-to-moea-d-and-examples-of-multi-objective-optimization-comparisons-8630565a4e89) from Hiroaki NATSUME(hrntsm) for more details.

<img width="400" alt="moead" src="https://github.com/user-attachments/assets/3ab83a64-b789-425b-800c-2c1f09e5a752">

Enhancements

- Update sklearn.py by addind catch to OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/163, thanks muhlbach!)
- Reduce `SELECT` statements by passing `study_id` to `check_and_add` in `TrialParamModel` (5702)
- Introduce `UPSERT` in `set_trial_user_attr` (5703)
- Reduce `SELECT` statements of `_CachedStorage.get_all_trials` by fixing filtering conditions (5704)
- Reduce `SELECT` statements by removing unnecessary distribution compatibility check in `set_trial_param()` (5709)
- Introduce `UPSERT` in `set_trial_system_attr` (5741)

Bug Fixes

- Accept `Mapping` as `param_distributions` in `OptunaSearchCV` (https://github.com/optuna/optuna-integration/pull/172, thanks yu9824!)
- Allow use of `OptunaSearchCV` with `cross_val_predict` (https://github.com/optuna/optuna-integration/pull/174, thanks yu9824!)
- Fix `GPSampler`'s suggestion failure within `torch.no_grad()` context manager (5671, thanks kAIto47802!)
- Fix a concurrency issue in `GPSampler` (5737)
- Fix a concurrency issue in `QMCSampler` (5740)

Installation

- Drop the support for Python 3.7 and update package metadata for Python 3.13 support (5727)
- Remove dependency specifier for installing SciPy (5736)

Documentation

- Add badges of PyPI and Conda Forge (https://github.com/optuna/optuna-integration/pull/167)
- Update news (5655)
- Update news section in `README.md` (5657)
- Add `InMemoryStorage` to document (5672, thanks kAIto47802!)
- Add the news about blog of `JournalStorage` to `README.md` (5674)
- Fix broken link in the artifacts tutorial (5677, thanks kAIto47802!)
- Add `pandas` installation guide to RDB tutorial (5685, thanks kAIto47802!)
- Add installation guide to multi-objective tutorial (5686, thanks kAIto47802!)
- Update document and notice interoperability between NSGA-II and Pruners (5688)
- Fix typo in `EMMREvaluator` (5694)
- Fix `RegretBoundEvaluator` document (5696)
- Update news section in `README.md` (5705)
- Add link to related blog post in `emmr.py` (5707)
- Update FAQ entry for model preservation using Optuna Artifact (5716, thanks chitvs!)
- Escape `\D` for Python 3.12 with sphinx build (5735)
- Avoid using functions in `sphinx_gallery_conf` to remove document build error with Python 3.12 (5738)
- Remove all `generated` directories in `docs/source` recursively (5739)
- Add information about `AutoSampler` to the docs (5745)

Examples

- Update CI config for `fastai` (https://github.com/optuna/optuna-examples/pull/279)
- Introduce `optuna.artifacts` to the PyTorch checkpoint example (https://github.com/optuna/optuna-examples/pull/280, thanks kAIto47802!)
- Fix inline code in README files in `kubernetes` directory (https://github.com/optuna/optuna-examples/pull/282)
- Test ray example with Python 3.12 (https://github.com/optuna/optuna-examples/pull/284)
- Add a link to molecule LLM notebook (https://github.com/optuna/optuna-examples/pull/285)
- Fix syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/286)
- Another fix for syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/287)
- Add checking for Python 3.12 (https://github.com/optuna/optuna-examples/pull/288, thanks kAIto47802!)
- Add Python 3.12 to tfkeras CI and remove warning message (https://github.com/optuna/optuna-examples/pull/289)
- Add Python 3.12 to `lightgbm` CI (https://github.com/optuna/optuna-examples/pull/290)
- Remove Python 3.7 from the workflow (https://github.com/optuna/optuna-examples/pull/291)

Code Fixes

- Add a comment for an unexpected bug in `CategoricalDistribution` (5683)
- Add more information about the hack in `WFG` (5687)
- Simplify type annotations to `_imports.py` (5692, thanks Prabhat-Thapa45!)
- Use `__future__.annotations` in `optuna/_experimental.py` (5714, thanks Jonathan43!)
- Use `__future__.annotations` in `tests/importance_tests/fanova_tests/test_tree.py` (5731, thanks guisp03!)
- Resolve TODO comments related to dropping Python 3.7 support (5734)

Continuous Integration

- Fix CI for `fastaiv2` (https://github.com/optuna/optuna-integration/pull/164)
- Fix for mypy (https://github.com/optuna/optuna-integration/pull/166)
- Fix mlflow integration for CI (https://github.com/optuna/optuna-integration/pull/168)
- Update CI to support Python 3.12 (https://github.com/optuna/optuna-integration/pull/170, thanks kAIto47802!)
- Update artifact version to v4 (https://github.com/optuna/optuna-integration/pull/176)
- Add a tri-objective problem to speed benchmarking (5635)
- Bump `actions/download-artifact` from 2 to 4.1.7 in `/.github/workflows` (5660)
- Update MySQL version in CI from 5.7 to 8 (5673)
- Remove performance benchmarks (5675)
- Update PostgreSQL version in CI to latest (5676)
- Update the version of Python used in the checks from 3.8 to 3.11 (5684)
- Run tests with Python 3.13 (5691)
- Fix for mypy (5706)
- [hotfix] Add a version constraint on fakeredis (5726)
- Update Python versions used in CI workflow (5728)
- Update sphinx build's `upload-artifact` version (5744)

Other

- Bump up version number to 4.1.0.dev (https://github.com/optuna/optuna-integration/pull/159)
- Update supported Python versions (https://github.com/optuna/optuna-integration/pull/175)
- Bump up to v4.1.0 (https://github.com/optuna/optuna-integration/pull/179)
- Add pre-commit config (5408)
- Add link to MOEA/D blog post in News (5719)
- Include all test files in `sdist` by updating `MANIFEST.in` (5720)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, Jonathan43, Prabhat-Thapa45, c-bata, chitvs, contramundum53, eukaryo, gen740, guisp03, kAIto47802, muhlbach, nabenabe0928, not522, nzw0301, porink0424, toshihikoyanase, y0z, yu9824

This is the release note of [v4.1.0](https://github.com/optuna/optuna-integration/milestone/9).

See the [release note of Optuna v4.1.0](https://github.com/optuna/optuna/releases/tag/v4.1.0) for more details.

This is the release note of [v4.0.0](https://github.com/optuna/optuna-integration/milestone/8).

See the [release note of Optuna v4.0.0](https://github.com/optuna/optuna/releases/tag/v4.0.0) for more details.