Clearml

Bug Fixes

- Fix pipeline breaks when `continue_on_abort` is set to true
- Fix Pycharm Plugin Windows/Linux interoperability

New Features

- Add additional warning instructing on how to install in case we failed detecting a Jupyter notebook with an import error
- Add `Task.get_executed_queue()` to get the name/ID of the queue a task was executed in
- Move `Task.set_resource_monitor_iteration_timeout()` to a class method, add `wait_for_first_iteration_to_start_sec` and `max_wait_for_first_iteration_to_start_sec` arguments (also add `sdk.development.worker.wait_for_first_iteration_to_start_sec` and `sdk.development.worker.max_wait_for_first_iteration_to_start_sec` configuration options)
- Add support for better pipeline continue behavior including control of children using the `continue_behaviour` argument
- Add Python 3.12 support

Bug Fixes

- Fix issue 1249 pytorch-lightning patches (1254, thanks a-gardner1!)
- Fix parameter overrides are converted to string when using HPO (975)
- Fix FastAI performance (1234)
- Fix MIG GPU support
- Fix AMD GPU metrics collection
- Fix Jupyter password might not be used in some protected Jupyterlab instances
- Fix URL substitution was not applied to registered uploaded files when reporting an event

Bug Fixes

- Fix auto-scaler should recheck that the worker is still IDLE before shutting it down (1240, thanks cthorey!)
- Fix resource monitor fails to get GPU stats in some edge cases

New Features
- Add draft option to pipeline steps (1226, thanks CharlesFrankum!)
- Add support for custom working directory for pipelines (1194)
- Add `Task.get_requirements()` method returning the task’s requirements
- Allow controlling the number of threads used by `StorageManager.download_folder()` using the `max_workers` argument
- Update examples dependencies
- Improve auto populate in `Task.init()`
- Documentation:
* Add docstrings for model properties
* Improve docstring for `force_requirements_env_freeze`
* Add `Task.set_packages()` docstring notes


Bug Fixes
- Fix UTF-8 script code encoding issue (1208, thanks ae-ae!)
- Fix Colab docs (1220, thanks tkukurin!)
- Fix metrics reporting with `OutputModel` while in offline mode results in an error (1172)
- Fix task running in Google Colab doesn't properly get the notebook diff (1204)
- Fix hydra binds break in offline mode (1215)
- Fix HPO crashes when optimizing for single value scalars (1221)
- Fix GPU info such as `gpu_memory` and `gpu_type` is not being collected in some cases
- Fix `clearml-data` CLI tool will move non-dataset tasks to a `.dataset` project if the respective task is not a dataset
- Fix pandas `DataFrame` artifacts with hierarchical indices get mangled by the CSV round-trip
- Fix `urllib3` sends deprecation warning when setting `ssl_version`
- Fix `Task.connect(dict)` return value is not dict-compatible
- Fix `jsonargparse` sub-command config parsing
- Fix Lightning integration crashes when a config entry contains `.` in its name
- Fix Python 3.5 compatibility

Bug Fixes

- Fix AWS S3 configuration `secure` property not forwarded in `StorageHelper` (1205, thanks odulcy-mindee!)
- Fix `NVIDIA_VISIBLE_DEVICES=all` env var value will result in no GPU metrics being reported
- Fix setting `output_uri` when an AWS S3 config section with `secure=false` exists raises an error
- Fix `CacheManager` attempts to clear cache when file limit has not been reached resulting in slow unnecessary checks

New Features and Bug Fixes

- Add hash check for pickle files (resolves CVE-2024-24590)
- Fix `clearml-task` relative paths calculation when `--cwd`, `--folder` and `--script` are provided (1161, thanks dimidagd!)
- Fix regression causing an error when `secure` is provided in AWS S3 bucket settings