Clearml

New Features and Bug Fixes

- Add hash check for pickle files (resolves CVE-2024-24590)
- Fix `clearml-task` relative paths calculation when `--cwd`, `--folder` and `--script` are provided (1161, thanks dimidagd!)
- Fix regression causing an error when `secure` is provided in AWS S3 bucket settings

New Features and Improvements

- Report upload/download with `tqdm` if installed
- Add support for a `PipelineController.version` property
- Add support for specifying an AWS profile when providing bucket storage credentials
- Add an `ignore_remote_overrides` argument to `Task.connect()` and `Task.connect_configuration()` allowing users to disable overriding values when running remotely
- Add support for setting a custom certificate path using the `CLEARML_API_HOST_VERIFY_CERT` environment variable (instead of just allowing the user to disable verification)
- Add dataset reporting example (1180)
- Update README GIFs (1191)

Bug Fixes

- Fix missing component callbacks on multiple step calls (1195, thanks materight!)
- Fix unsafe usage of `tempfile.mktemp`
- Fix potential path traversal on file download (CVE-2024-24591)
- Fix Python 3.12 support by not using `distutils`
- Fix `TaskScheduler` does not save local state before running remotely
- Fix pipeline should continue instead of aborting when `PipelineController.abort_on_failure` is set to False and a step has been aborted
- Fix HPO crashes when its corresponding task is initialized with `continue_last_task=True`

Bug Fixes

- Fix created pipelines do not appear in the UI pipelines section

New Features and Improvements

- Add support for decorated pipeline steps (1154)
- Improve `torch.distributed` behavior and auto-detect Huggingface `accelerate` distributed execution
- Improve `jsonargparse` integration: `allow_config_file_override_from_ui` argument added for enabling/disabling overriding configuration file entries
- Add `raise_on_error` parameter to `Model.update_weights()` and `Model.update_weights_package()`
- Add HPO multi-objective optimization support by specifying multiple optimization objectives to the `HyperParameterOptimizer` class
- Improve pipelines from functions: libraries used by pipeline steps no longer have to be imported in the step itself and can be imported globally instead
- Support clearing repository details in `task.set_repo()`
- Auto detect `tensorflow-intel` as `tensorflow`
- Add support for auto detecting torch and transformers accelerate distributed execution
- Add tabular data multi-stage pipeline process example

Bug Fixes

- Fix `langchain` integration (1124, 1126)
- Fix using `target_project` with `TaskScheduler.add_task()` corrupts project if used with pipelines (1137)
- Fix `PipelineController.add_function_step()` fails when adding custom decorated step (1154)
- Fix `jsonargparse` binding does not capture parameters before `Task.init()` is called (1164)
- Fix invalid `jsonargparse` arguments should be deleted
- Fix handling of appended parameters to Hydra defaults list
- Fix `Task.set_repo()` and `Task.set_packages()` do not wait for automatic asynchronous repository detection
- Fix active GPU filtering in resource monitor to support GPU ID and not just GPU idx
- Fix misleading error raised when using the SDK with no configuration file or an invalid configuration file
- Fix process sometimes hangs issue by improving exit and exception handlers, unregistering signal handling in child processes except for the `BackgroundMonitor`

New Features
- Support parallel dataset verification (1131, thanks charlienewey-odin!)
- Support `lightgbm` model loading from a string (1136, thanks bedapisl!)
- Add support for Hydra command-line syntax for modifying `omegaconf`
- Add support for setting `output_uri` in pipeline decorators and functions
- Add more visibility when overriding `jsonargparse` arguments
- Use StorageManager cache when downloading models
- Support parsing GS credentials as a JSON string if they do not point to a JSON file
- Support ignoring `default_output_uri` on remote runs when `Task.init(output_uri=False)` was specified and the backend task data has no `output_uri` specified
- Add configuration files logging [example](https://github.com/allegroai/clearml/blob/e6f1cc8a5c65f7007cf6452745d09795e776a548/examples/reporting/config_files.py)
- Support raising `LoginError` when initializing
- Add HuggingFace Transformers Example (1146)
- Improve docstrings for tag filters
- Add a security policy file (1148)


Bug Fixes
- Fix Google Storage subdir path in bucket URL breaks validation (1117, thanks materight!)
- Fix jsonschema `DeprecationWarning` in newer versions (1128, thanks xquyvu!)
- Fix `task.set_script(diff="")` does not clear the uncommitted changes (1106)
- Fix parallel dataset verification fails on older Python versions (1144)
- Fix `attrs.__version__` deprecation warning (1145)
- Fix `Task.get_parameters_as_dict(cast=True)` raises a YAML parser error
- Fix potential threading issues when setting `self.lock` to `None`
- Fix pipeline version is not displayed in the table view and version list
- Fix displayed pipeline run time in pipeline components is not the same as that shown in the info tab in full detail
- Fix `Model.connect()` in remote execution might result in the wrong model being connected
- Fix exception raised on `ConnectionError` (which actually indicates that a read has timed out, and should be retried)
- Fix missing configuration error not raised in certain cases
- Fix loading queues containing metadata fails due to an incorrect queue metadata scheme
- Fix `pyjwt` requirements for Python < 2.9
- Fix pipeline examples

Bug Fixes

- Fix Hydra overrides integration (1120)
- Fix GPU memory used reports 0 when memory can not be queried per process (edge case)