Clearml

Bug Fixes

- Fix auto-scaler should recheck that the worker is still IDLE before shutting it down (1240, thanks cthorey!)
- Fix resource monitor fails to get GPU stats in some edge cases

New Features
- Add draft option to pipeline steps (1226, thanks CharlesFrankum!)
- Add support for custom working directory for pipelines (1194)
- Add `Task.get_requirements()` method returning the task’s requirements
- Allow controlling the number of threads used by `StorageManager.download_folder()` using the `max_workers` argument
- Update examples dependencies
- Improve auto populate in `Task.init()`
- Documentation:
* Add docstrings for model properties
* Improve docstring for `force_requirements_env_freeze`
* Add `Task.set_packages()` docstring notes


Bug Fixes
- Fix UTF-8 script code encoding issue (1208, thanks ae-ae!)
- Fix Colab docs (1220, thanks tkukurin!)
- Fix metrics reporting with `OutputModel` while in offline mode results in an error (1172)
- Fix task running in Google Colab doesn't properly get the notebook diff (1204)
- Fix hydra binds break in offline mode (1215)
- Fix HPO crashes when optimizing for single value scalars (1221)
- Fix GPU info such as `gpu_memory` and `gpu_type` is not being collected in some cases
- Fix `clearml-data` CLI tool will move non-dataset tasks to a `.dataset` project if the respective task is not a dataset
- Fix pandas `DataFrame` artifacts with hierarchical indices get mangled by the CSV round-trip
- Fix `urllib3` sends deprecation warning when setting `ssl_version`
- Fix `Task.connect(dict)` return value is not dict-compatible
- Fix `jsonargparse` sub-command config parsing
- Fix Lightning integration crashes when a config entry contains `.` in its name
- Fix Python 3.5 compatibility

Bug Fixes

- Fix AWS S3 configuration `secure` property not forwarded in `StorageHelper` (1205, thanks odulcy-mindee!)
- Fix `NVIDIA_VISIBLE_DEVICES=all` env var value will result in no GPU metrics being reported
- Fix setting `output_uri` when an AWS S3 config section with `secure=false` exists raises an error
- Fix `CacheManager` attempts to clear cache when file limit has not been reached resulting in slow unnecessary checks

New Features and Bug Fixes

- Add hash check for pickle files (resolves CVE-2024-24590)
- Fix `clearml-task` relative paths calculation when `--cwd`, `--folder` and `--script` are provided (1161, thanks dimidagd!)
- Fix regression causing an error when `secure` is provided in AWS S3 bucket settings

New Features and Improvements

- Report upload/download with `tqdm` if installed
- Add support for a `PipelineController.version` property
- Add support for specifying an AWS profile when providing bucket storage credentials
- Add an `ignore_remote_overrides` argument to `Task.connect()` and `Task.connect_configuration()` allowing users to disable overriding values when running remotely
- Add support for setting a custom certificate path using the `CLEARML_API_HOST_VERIFY_CERT` environment variable (instead of just allowing the user to disable verification)
- Add dataset reporting example (1180)
- Update README GIFs (1191)

Bug Fixes

- Fix missing component callbacks on multiple step calls (1195, thanks materight!)
- Fix unsafe usage of `tempfile.mktemp`
- Fix potential path traversal on file download (CVE-2024-24591)
- Fix Python 3.12 support by not using `distutils`
- Fix `TaskScheduler` does not save local state before running remotely
- Fix pipeline should continue instead of aborting when `PipelineController.abort_on_failure` is set to False and a step has been aborted
- Fix HPO crashes when its corresponding task is initialized with `continue_last_task=True`

Bug Fixes

- Fix created pipelines do not appear in the UI pipelines section