Clearml

Breaking Changes

- As a result of fixing 1051 and similar issues with retrieving task data/models/artifacts/samples stored in GCS we introduced some breaking changes. To access resources registered to GCS from ClearML SDK versions 1.11.0 to 1.11.1, see the instructions [here](https://github.com/allegroai/clearml/tree/master/docs/errata_breaking_change_gcs_sdk_1_11_x.md).


New Features and Improvements

- Add `include_archive` parameter to `Dataset.list_datasets()`, now possible to include archived datasets in the returned list (1069, thanks natephysics!)
- Add support to specify the multipart threshold and chunk size using the `aws.boto3.multipart_chunksize` and `aws.boto3.multipart_threshold` configuration options (1059, thanks cgaudreau-ubisoft!)
- Add `PipelineController.get_pipeline()` for retrieving previously run pipelines


Bug Fixes

- Fix AWS driver issues (1000, thanks achaiah!)
* Fix credential authentication failure when attempting to use token
* Fix instantiation within VPC without AvailabilityZones
- Fix `continue_last_task=0` is ignored in pipelines run with `retry_on_failure` (1054)
- Fix `Task.connect_configuration()` doesn’t handle dictionaries with special characters
- Fix pipeline steps created with PipelineDecorator aren't cached
- Fix `Task.get_by_name()` doesn't return the most recent task when multiple tasks have same name
- Fix `boto3` certificate verification ignores custom self-signed certificates

New Features and Improvements

- Support `lightning>=2.0` (1033, thanks aweinmann!)
- Add `clearml-init` support for standard HTTP and HTTPS ports in webapp in conjunction with non-default api/files server ports (1031, thanks pktiuk!)
- Improve docstrings

Bug Fixes

- Fix remote execution does not log configuration values from modified `jsonargparse` configuration files
- Fix `ValueError` when setting task properties at the start of a pipeline component
- Fix OS exceptions on passing `Path` where `str` expected
- Fix infinite recursion when remotely updating a `dict` added to task with `Task.connect()`
- Fix `Task.connect` race condition overwriting task description with connected input model

New Features and Improvements

- Add offline mode for ClearML Datasets (854)
- Improve performance of `Dataset.add_external_files()` (962, thanks john-zielke-snkeos!)
- Add `Task.launch_multi_node()` for distributed experiment execution
- Add `Task.get_all_reported_scalars()` to fetch all available scalar data
- Add `Task.remove_input_models()` to disassociate input models from a task
- Improve `Dataset.list_datasets()` to include dataset version in the result
- Add support for PyTorch Lightning CLI >=2.0
- Add support for automatic Dataset content preview for JSON files
- Improve docstrings

Bug Fixes

- Fix multiple local parallel experiments conflicting on storage access validation (897)
- Fix code running with `joblib` hangs (1009)
- Fix comment for storage scheme `gs://` (1018, thanks pzarfos!)
- Fix typo "Categories per column" (1021, thanks antonaleks!)
- Fix downloading artifacts from GCP when the download URI includes unsafe characters
- Fix pipeline step cache is not cleared when the only change to the step is that the docker image changed
- Fix `Task.report_text()` sometimes reporting to an incorrect task when multiple tasks run simultaneously
- Fix `Task.set_offline(offline_mode=False)` raising an `UnboundLocalError`
- Fix `scikit-image` package (`skimage`) is sometimes not detected as a dependency
- Fix `PipelineDecorator` sometimes causing a race condition when starting a remote execution



v1.10.4-fix
New Features and Improvements

- Update references to `GetAllRequest` in `Task` docstrings (982, thanks Make42!)

Bug Fixes

- Fix issue with JSON conversion when using `jsonargparse` during remote execution ([clearml-agent 153](https://github.com/allegroai/clearml-agent/issues/153) and #1010)
- Fix `ProxyDictPostWrite.update()` not triggering a write-back to the backend (985)
- Fix standard streams being patched in subtasks, which was causing the streams to be reported to the wrong task
- Fix `TypeError` when using `Task.query_tasks()` without specifying the `task_filter` parameter
- Fix Tensorboard negative iterations, they are now zeroed
- Fix `Task.init(continue_last_task=0)` when running remotely, now no longer ignored
- Fix `urllib3` import error when using `urllib3>=2.0.0`

Bug Fixes

- Fix mixed-up logs when using pipelines. Now pipeline controller logs won't be visible in pipeline steps' logs.

New Features and Improvements
- Allow task filtering with `Task.query_tasks()` using `TaskTypes` (938 )
- Allow adding tags to individual pipeline steps and to the `PipelineController`
- Restrict `Dataset.create()` usage for old server API versions
- Improve docstrings

Bug Fixes
- Fix `Task.get_tasks` docstring (937)
- Fix `Task.get_reported_scalars()` docstring for `x_axis` parameter when set to `"timestamp"` (964, thanks jday1!)
- Fix missing plots on models in Windows
- Fix Pipeline version auto-bumping
- Fix `APIClient.events.debug_images()` for latest server API version

New Features and Improvements
- Add `cast` parameter to `Task.get_parameter` method (958, thanks harry-optimised!)

Bug Fixes
- Fix backward compatibility issue when using datetime filtering with `APIClient().models.get_all` and `APIClient().tasks.get_all` requests
- Fix docstrings explaining tags filtering on `Model` and `Task`