Clearml

New Features

- Add support for recursive list, dict, and tuple ref parsing for pipeline `controller.add step()` parameter overrides (1099, thanks natephysics!)
- Update PyNVML to the latest NVIDIA version for better GPU monitoring
- Add `force_single_script_file` argument to `Task.create()` to avoid git repository auto detection
- Use `os.register_at_fork` instead of monkey patching fork for python > 3.6
- Add support for programmatically archive and unarchive models from the model registry using the `Model.archive()` and `Mode.unarchive()` methods (1096)
- Improve documentation

Bug Fixes

- Fix Hydra overrides integration (552)
- Fix model reporting in TensorFlow 2.13 does not work properly (1112)
- Fix GPU memory used reports 0 when memory can not be queried per process
- Fix FastAI binding not reporting scalars if TensorBoard is installed
- Fix JSON serialization error does not raise exception
- Fix OutputModel should prefer connected task name over filename stem
- Fix dataset zip extraction might fail when creating folders concurrently (with `Exception [Errno 17] File exists` warning)
- Fix import `sys` should not appear in pipeline component auto detected packages
- Fix compatibility issue with requests < 2.28

New Features and Improvements

- Add `always_create_from_code` argument to `PipelineController` (default True) to allow remote execution to create the Pipeline DAG at runtime (as opposed to adhering to the existing structure stored on the task when running locally)

Bug Fixes

- Fix using `OutputModel` reporting and other methods fail if `OutputModel.update_weights()` is not called before that (1078)
- Fix hash for Pandas objects artifacts
- Fix Hydra/OmegaConf `--overrides` support
- Fix Pipeline metrics not forwarded to pipeline controller at the end of a pipeline step

New Features and Improvements

- Raise error if dataset name is empty (1083, thanks mathematicalmichael!)
- Add support for `spawn` start method for Python `multiprocess`. This should help circumvent issues like [this](https://github.com/python/cpython/issues/84559)
- Improve docstrings


Bug Fixes

- Fix S3 uploads fail with `LazyEvalWrapper` type error (1081)

Breaking Changes

- As a result of fixing 1051 and similar issues with retrieving task data/models/artifacts/samples stored in GCS we introduced some breaking changes. To access resources registered to GCS from ClearML SDK versions 1.11.0 to 1.11.1, see the instructions [here](https://github.com/allegroai/clearml/tree/master/docs/errata_breaking_change_gcs_sdk_1_11_x.md).


New Features and Improvements

- Add `include_archive` parameter to `Dataset.list_datasets()`, now possible to include archived datasets in the returned list (1069, thanks natephysics!)
- Add support to specify the multipart threshold and chunk size using the `aws.boto3.multipart_chunksize` and `aws.boto3.multipart_threshold` configuration options (1059, thanks cgaudreau-ubisoft!)
- Add `PipelineController.get_pipeline()` for retrieving previously run pipelines


Bug Fixes

- Fix AWS driver issues (1000, thanks achaiah!)
* Fix credential authentication failure when attempting to use token
* Fix instantiation within VPC without AvailabilityZones
- Fix `continue_last_task=0` is ignored in pipelines run with `retry_on_failure` (1054)
- Fix `Task.connect_configuration()` doesn’t handle dictionaries with special characters
- Fix pipeline steps created with PipelineDecorator aren't cached
- Fix `Task.get_by_name()` doesn't return the most recent task when multiple tasks have same name
- Fix `boto3` certificate verification ignores custom self-signed certificates

New Features and Improvements

- Support `lightning>=2.0` (1033, thanks aweinmann!)
- Add `clearml-init` support for standard HTTP and HTTPS ports in webapp in conjunction with non-default api/files server ports (1031, thanks pktiuk!)
- Improve docstrings

Bug Fixes

- Fix remote execution does not log configuration values from modified `jsonargparse` configuration files
- Fix `ValueError` when setting task properties at the start of a pipeline component
- Fix OS exceptions on passing `Path` where `str` expected
- Fix infinite recursion when remotely updating a `dict` added to task with `Task.connect()`
- Fix `Task.connect` race condition overwriting task description with connected input model

New Features and Improvements

- Add offline mode for ClearML Datasets (854)
- Improve performance of `Dataset.add_external_files()` (962, thanks john-zielke-snkeos!)
- Add `Task.launch_multi_node()` for distributed experiment execution
- Add `Task.get_all_reported_scalars()` to fetch all available scalar data
- Add `Task.remove_input_models()` to disassociate input models from a task
- Improve `Dataset.list_datasets()` to include dataset version in the result
- Add support for PyTorch Lightning CLI >=2.0
- Add support for automatic Dataset content preview for JSON files
- Improve docstrings

Bug Fixes

- Fix multiple local parallel experiments conflicting on storage access validation (897)
- Fix code running with `joblib` hangs (1009)
- Fix comment for storage scheme `gs://` (1018, thanks pzarfos!)
- Fix typo "Categories per column" (1021, thanks antonaleks!)
- Fix downloading artifacts from GCP when the download URI includes unsafe characters
- Fix pipeline step cache is not cleared when the only change to the step is that the docker image changed
- Fix `Task.report_text()` sometimes reporting to an incorrect task when multiple tasks run simultaneously
- Fix `Task.set_offline(offline_mode=False)` raising an `UnboundLocalError`
- Fix `scikit-image` package (`skimage`) is sometimes not detected as a dependency
- Fix `PipelineDecorator` sometimes causing a race condition when starting a remote execution



v1.10.4-fix
New Features and Improvements

- Update references to `GetAllRequest` in `Task` docstrings (982, thanks Make42!)

Bug Fixes

- Fix issue with JSON conversion when using `jsonargparse` during remote execution ([clearml-agent 153](https://github.com/allegroai/clearml-agent/issues/153) and #1010)
- Fix `ProxyDictPostWrite.update()` not triggering a write-back to the backend (985)
- Fix standard streams being patched in subtasks, which was causing the streams to be reported to the wrong task
- Fix `TypeError` when using `Task.query_tasks()` without specifying the `task_filter` parameter
- Fix Tensorboard negative iterations, they are now zeroed
- Fix `Task.init(continue_last_task=0)` when running remotely, now no longer ignored
- Fix `urllib3` import error when using `urllib3>=2.0.0`