Optuna

Human-in-the-loop Optimization



$ pip install Pillow optuna==3.1.0 optuna-dashboard==0.9.0b4


python
from __future__ import annotations

import os
import textwrap
import threading
import time
from typing import NoReturn
from wsgiref.simple_server import make_server

import optuna
from PIL import Image
from optuna.trial import TrialState

from optuna_dashboard import ObjectiveChoiceWidget, save_note, ObjectiveSliderWidget
from optuna_dashboard import register_objective_form_widgets
from optuna_dashboard import set_objective_names
from optuna_dashboard import wsgi
from optuna_dashboard.artifact import upload_artifact
from optuna_dashboard.artifact.file_system import FileSystemBackend


storage = optuna.storages.InMemoryStorage()
base_path = os.path.join(os.path.dirname(__file__), "artifact")
artifact_backend = FileSystemBackend(base_path=base_path)


def suggest_and_generate_image(study: optuna.Study) -> None:
Ask new parameters
trial = study.ask()
r = trial.suggest_int("r", 0, 255)
g = trial.suggest_int("g", 0, 255)
b = trial.suggest_int("b", 0, 255)

Generate image
image_path = f"tmp/sample-{trial.number}.png"
image = Image.new("RGB", (320, 240), color=(r,g,b))
image.save(image_path)

Upload Artifact
artifact_id = upload_artifact(artifact_backend, trial, image_path)

Save Note
note = textwrap.dedent(f'''\
Trial {trial.number}

''')
save_note(trial, note)


def start_preferential_optimization() -> NoReturn:
Create Study
study = optuna.create_study(
study_name="Human-in-the-loop Optimization",
storage=storage,
directions=["minimize", "maximize"]
)
set_objective_names(study, ["Human Perception Score", "Looks Yellow?"])
register_objective_form_widgets(study, widgets=[
ObjectiveChoiceWidget(
choices=["Good 👍", "Bad 👎"],
values=[-1, 1],
description="Please input your score!",
),
ObjectiveSliderWidget(
min=1,
max=10,
step=1,
description="Higher is better.",
),
])

Start Human-in-the-loop Optimization
n_batch = 3
while True:
running_trials = study.get_trials(deepcopy=False, states=(TrialState.RUNNING,))
if len(running_trials) >= n_batch:
time.sleep(5)
continue
suggest_and_generate_image(study)


def main() -> None:
if not os.path.exists(base_path):
os.mkdir(base_path)

Start Dashboard server on background
app = wsgi(storage, artifact_backend=artifact_backend)
httpd = make_server("127.0.0.1", 8080, app)
thread = threading.Thread(target=httpd.serve_forever)
thread.start()

Run optimize loop
try:
start_preferential_optimization()
except KeyboardInterrupt:
httpd.shutdown()
httpd.server_close()
thread.join()


if __name__ == "__main__":
main()


What's Changed
* Fix bug of custom user widgets and enhance artifact support. by c-bata in https://github.com/optuna/optuna-dashboard/pull/375
* Update best trials when state is updated by c-bata in https://github.com/optuna/optuna-dashboard/pull/376
* Bump the version up to v0.9.0b4 by c-bata in https://github.com/optuna/optuna-dashboard/pull/377


**Full Changelog**: https://github.com/optuna/optuna-dashboard/compare/v0.9.0b3...v0.9.0b4

What's Changed
* Add state filters in TrialList by c-bata in https://github.com/optuna/optuna-dashboard/pull/363
* Hide user_attrs on GraphParallelCoordinate by default by c-bata in https://github.com/optuna/optuna-dashboard/pull/365
* Update `TrialList` component by c-bata in https://github.com/optuna/optuna-dashboard/pull/364
* Support user attributes on GraphSlice by c-bata in https://github.com/optuna/optuna-dashboard/pull/366
* Remove TODOs in CachedExtraStudyProperty by c-bata in https://github.com/optuna/optuna-dashboard/pull/367
* Set `key` attributes in `TrialList` by c-bata in https://github.com/optuna/optuna-dashboard/pull/368
* Avoid to calculate importance when only available single trial by c-bata in https://github.com/optuna/optuna-dashboard/pull/369
* Add trial tell feature by keisuke-umezawa in https://github.com/optuna/optuna-dashboard/pull/355
* Introduce `Artifact` by c-bata in https://github.com/optuna/optuna-dashboard/pull/314
* Introducing Custom User Widget by c-bata in https://github.com/optuna/optuna-dashboard/pull/370
* Save reload interval on localStorage by c-bata in https://github.com/optuna/optuna-dashboard/pull/372
* Bump the version up to v0.9.0b3 by c-bata in https://github.com/optuna/optuna-dashboard/pull/373


**Full Changelog**: https://github.com/optuna/optuna-dashboard/compare/v0.9.0b2...v0.9.0b3

What's Changed
* Add Python API to save the note by c-bata in https://github.com/optuna/optuna-dashboard/pull/337
* Add API to set objective names by c-bata in https://github.com/optuna/optuna-dashboard/pull/338
* Handle exceptions in optuna-fast-fanova by c-bata in https://github.com/optuna/optuna-dashboard/pull/339
* Fix the link to feedback survey by c-bata in https://github.com/optuna/optuna-dashboard/pull/340
* Use type instead of interface by c-bata in https://github.com/optuna/optuna-dashboard/pull/342
* Change Chip's colors and show durations by c-bata in https://github.com/optuna/optuna-dashboard/pull/343
* Cutomize TrialTable for new ui by c-bata in https://github.com/optuna/optuna-dashboard/pull/344
* Support trial user attributes on `GraphHistory` by c-bata in https://github.com/optuna/optuna-dashboard/pull/341
* Add button to rename study by c-bata in https://github.com/optuna/optuna-dashboard/pull/345
* Fix target arg for importance by c-bata in https://github.com/optuna/optuna-dashboard/pull/346
* Select multiple trials by Shift + Click by c-bata in https://github.com/optuna/optuna-dashboard/pull/347
* Make plot components faster and robust. by c-bata in https://github.com/optuna/optuna-dashboard/pull/348
* Fix seed visual regression tests by c-bata in https://github.com/optuna/optuna-dashboard/pull/349
* Simplify Python API to save notes by c-bata in https://github.com/optuna/optuna-dashboard/pull/350
* Improve graph components by c-bata in https://github.com/optuna/optuna-dashboard/pull/351
* Fix the link to trial detail page by c-bata in https://github.com/optuna/optuna-dashboard/pull/352
* Support fixed_params by c-bata in https://github.com/optuna/optuna-dashboard/pull/353
* Filter TrialState for GraphIntermediateValues by c-bata in https://github.com/optuna/optuna-dashboard/pull/310
* Fix bug of multiple select trials by c-bata in https://github.com/optuna/optuna-dashboard/pull/356
* Plot the hyperparameter importances for each objective value on a single bar chart. by c-bata in https://github.com/optuna/optuna-dashboard/pull/357
* Add global state filters for History and IntermediateValue plots by c-bata in https://github.com/optuna/optuna-dashboard/pull/358
* Use `theme.typography.fontWeightBold` instead of 600 by c-bata in https://github.com/optuna/optuna-dashboard/pull/359
* Remove paddingTop from CardAction on StudyList by c-bata in https://github.com/optuna/optuna-dashboard/pull/361
* Bump the version up to v0.9.0b2 by c-bata in https://github.com/optuna/optuna-dashboard/pull/360


**Full Changelog**: https://github.com/optuna/optuna-dashboard/compare/v0.9.0b1...v0.9.0b2

The new UI for Optuna Dashboard

| TOP PAGE (NEW UI) | STUDY DETAIL PAGE (NEW UI) |
| --- | --- |
| <img width="2024" alt="Screenshot 2023-01-04 11 21 44" src="https://user-images.githubusercontent.com/5564044/210472991-559d3fd0-959e-4130-8c4a-92a8fc750f81.png"> | <img width="2066" alt="Screenshot 2023-01-04 11 22 40" src="https://user-images.githubusercontent.com/5564044/210473003-8244f569-85bb-46e2-bbb7-46e49282cbc8.png"> |

We are considering replacing the current UI of Optuna Dashboard with the new UI that is available from this release. Please try it out and share your thoughts with us via [this post](https://github.com/optuna/optuna-dashboard/discussions/332) 👇


What's Changed
* Upgrade to React 18 by c-bata in https://github.com/optuna/optuna-dashboard/pull/309
* Use `skip_table_creation` when using Optuna v3 or later by c-bata in https://github.com/optuna/optuna-dashboard/pull/320
* Remove `typing_extensions` from the dependencies by c-bata in https://github.com/optuna/optuna-dashboard/pull/321
* Use type hinting generics. by c-bata in https://github.com/optuna/optuna-dashboard/pull/322
* Update js dependencies by c-bata in https://github.com/optuna/optuna-dashboard/pull/323
* Run npm audit fix by c-bata in https://github.com/optuna/optuna-dashboard/pull/324
* Fix warning for the use of `ReactDOM.render()` by c-bata in https://github.com/optuna/optuna-dashboard/pull/325
* Remove `e.preventDefault` by c-bata in https://github.com/optuna/optuna-dashboard/pull/327
* New Dashboard UI. by c-bata in https://github.com/optuna/optuna-dashboard/pull/328
* Make a lot of improvements in the new Dashboard UI by c-bata in https://github.com/optuna/optuna-dashboard/pull/329
* Add follow-up changes for 329 by c-bata in https://github.com/optuna/optuna-dashboard/pull/330
* Fix bug of markdown editor and add mathjax support. by c-bata in https://github.com/optuna/optuna-dashboard/pull/331
* Link to the GitHub Discussion for the feedback survey by c-bata in https://github.com/optuna/optuna-dashboard/pull/333
* Bump the version up to 0.9.0b1 by c-bata in https://github.com/optuna/optuna-dashboard/pull/334


**Full Changelog**: https://github.com/optuna/optuna-dashboard/compare/v0.8.1...v0.9.0b1

What's Changed
* Fix a bug of is_sortable inference for trial user attrs by c-bata in https://github.com/optuna/optuna-dashboard/pull/308
* Avoid to call deprecated system_attrs property by c-bata in https://github.com/optuna/optuna-dashboard/pull/311
* Fix typo by Alnusjaponica in https://github.com/optuna/optuna-dashboard/pull/312
* Update the getting started section of README by c-bata in https://github.com/optuna/optuna-dashboard/pull/313
* Interface change for set study directions by c-bata in https://github.com/optuna/optuna-dashboard/pull/315
* Interface change for `set_study_directions` by gen740 in https://github.com/optuna/optuna-dashboard/pull/292
* Fix a pypi publish action by c-bata in https://github.com/optuna/optuna-dashboard/pull/317
* Bump the version up to v0.8.1 by c-bata in https://github.com/optuna/optuna-dashboard/pull/318

New Contributors
* Alnusjaponica made their first contribution in https://github.com/optuna/optuna-dashboard/pull/312
* gen740 made their first contribution in https://github.com/optuna/optuna-dashboard/pull/292

**Full Changelog**: https://github.com/optuna/optuna-dashboard/compare/v0.8.0...v0.8.1

This is the release note of v0.8.0. See [here](https://github.com/pfnet/optuna/milestone/4?closed=1) for the complete list of solved issues and merged PRs.

Enhancements
------------
- Support pruning when using ChainerMN. (286, thanks MannyKayy, rcalland!)
- Add queries to retrieve information of trials. (284)

Documents
---------
- Update configurations.rst with grammar fix. (316, thanks d1vanloon!)
- Fix PEP8 errors on README.md. (314, thanks nai62!)
- Add a blank line to show code example. (311)