Clearml

Breaking Changes

- When using MinIO or Azure storage, downloading using `StorageManager.download_file()` and `StorageManager.download_folder()` will not create a subfolder with the bucket name in the generated download path (709)

New Features and Improvements

- Improve the `Task.mark_completed`, `Task.close`, and the hyperparameter example (927, thanks Make42!)
- Add `Task.get_debug_samples()` to retrieve task's debug samples (761)
- Add custom artifact serialization for pipelines using the new `artifact_{serialization,deserialization}_function` parameters in `PipelineController` and `PipelineDecorator` (689)
- Allow registering offline models (653)
- Add table layout configuration to `Logger.report_table()` through `extra_data` parameter, solving (796)
- Add support for attaching plots and scalars to models using the new `Model.report_*` methods
- Add `force_download` argument to `Model.get_local_copy()` to force the downloading of a new copy of a model even if already in the cache
- Add `PipelineController.connect_configuration()` to add configuration dictionaries to a pipeline
- Add `Dataset.delete` configurable hosted contents deletion support, allowing deletion of both ClearML file server hosted files, and external storage hosted files
- Add `PipelineController.is_successful()` criteria customization
- Adjust `CLEARML_VCS_DIFF` environment variable behavior, setting it to an empty string now forces the task to not log uncommitted changes
- Add support for API v2.23
- Add Getting Started Notebooks
- Add the `output_uri=True` argument in the hook config for the OpenMMLab example
- Add local HPO command line execution with the `--local` argument
- Add support for python package namespace detection

Bug Fixes

- Fix logging dependencies that use the `subdirectory` argument when `pip install`ing from a git repo (947, thanks jday1!)
- Fix missing comma leading to a `SyntaxError` (959, thanks jday1!)
- Fix auto-logging for multiple models trained by the same framework in the same task (767)
- Fix wrong overriding of custom server URLs when running `clearml-init` ([clearml-server 181](https://github.com/allegroai/clearml-server/issues/181) and #910)
- Fix parallel coordinates plot display of categorical variables (907)
- Fix `task.connect()` docstring ([ClearML Docs issue 473](https://github.com/allegroai/clearml-docs/issues/473))
- Fix auto-publishing in `Dataset.finalize()` raises exception due to backward compatibility issue (908)
- Fix auto-detection of Jupyter Notebooks in Sagemaker
- Fix Gradio binding being triggered even when `gradio` is not imported
- Fix reconnection following an offline session. Only allow closed tasks to be reconnected
- Fix `python-fire` support
- Fix mishandling nested dictionaries in `Task.connect()` inside `PipelineDecorator`
- Fix `get_or_create_project` crashes when run in parallel
- Fix `task.get_user_properties(value_only=True)`, previously raising an exception
- Fix `silence_errors` parameter not working as expected in `StorageManager.get_file_size_bytes()`
- Fix setting `Dataset` metadata name to a string prefixed with `data_` breaking `Dataset.get`
- Fix Pipeline fails when creating tasks from multi-argument functions, due to a parsing error when working with Python comma slices

Bug Fixes
- Fix broken `Task._get_status()` which was breaking `clearml-session` in the latest version
- Fix path substitution, making it possible now to store unsubstituted URL for models (935 thanks john-zielke-snkeos!)

New Features and Improvements

- Support parsing queue name when providing execution queue in pipelines code (857, thanks Anton-Cherepkov!)
- Ignore `None` values for keys in the `click` argument parser (903, thanks chengzegang!)
- Improve docstrings for `Task.mark_completed()` and `Task.close()` (921, 920, thanks Make42!)
- Add pre/post execution callbacks to pipeline steps using `PipelineDecorator.component`
- Add status-change callback to pipeline steps using `PipelineController.add_step()`, `PipelineController.add_function_step()`, and `PipelineDecorator.component`

Bug Fixes

- Fix missing debug samples when reporting using TensorBoard (924, thanks jday1!)
- Fix wrong Jupyter API token during repository detection (904, thanks RoseGoldIsntGay!)
- Fix typo in the warning for very large git diffs (932, thanks yiftachbeer!)
- Fix pipelines from tasks don't propagate `parameter_override` values in `PipelineController.add_step()`
- Fix folders and files uploaded to S3 and Azure with `StorageManager.upload_file()` have wrong MIME types
- Fix CSV file preview in Datasets
- Fix `Task.connect_configuration()` doesn't work with non-string dictionary keys
- Fix LightGBM example deprecation warning
- Fix potential race condition in `get_or_create_project()`

New Features and Improvements

- Add signature version to `boto3` configuration (884, thanks cgaudreau-ubisoft!)
- Allow requesting custom token expiration using the `api.auth.req_token_expiration_sec` configuration setting
- Add Python 3.11 support

Bug Fixes

- Fix `UniformParameterRange.tolist()` throws error when step size is not defined (859, thanks davyx8!)
- Fix `StorageManager.list()` does not return size metadata (865)
- Fix storage with path substitutions (877, thanks john-zielke-snkeos!)
- Fix extras in ClearML installation prevents `clearml` from being included in requirements (885, thanks cajewsa!)
- Fix metadata set on an uploaded model object is not accessible (890, thanks supritmkulkarni!)
- Fix `TriggerScheduler` docstrings (881)
- Fix Azure storage upload not working (868)
- Fix connect list of dicts parsed incorrectly in remote
- Fix casting `None` to `int` fails uploads and permission checks
- Fix numpy 1.24 support
- Fix `clearml-data` previews are saved on file server even when `output_uri` is specified
- Fix connecting a dictionary to task sometimes raises an exception
- Fix authentication headers are not set on substituted fileserver URLs
- Fix `Task.get_project_id()` cannot find hidden projects

New Features and Improvements

- Add `r` prefix to re.match() strings (837, thanks daugihao!)
- Add `path_substitution` to `clearml.conf` example file (842)
- Clarify `deferred_init` usage in `Task.init()` (855)
- Add pipeline decorator argument to control docker image (856)
- Add `StorageManager.set_report_upload_chunk_size()` and `StorageManager.set_report_download_chunk_size()` to set chunk size for upload and download
- Add `allow_archived` argument in `Task.get_tasks()`
- Support querying model metadata in `Model.query_models()`
- Add `Dataset.set_metadata()` and `Dataset.get_metadata()`
- Add `delete_from_storage` (default `True`) to `Task.delete_artifacts()`

Bug Fixes

- Fix jsonargparse and pytorch lightning integration broken for remote execution (403)
- Fix error when using `TaskScheduler` with 'limit_execution_time' (648)
- Fix dataset not synced if the changes are only modified files (835, thanks fjean!)
- Fix `StorageHelper.delete()` does not respect path substitutions (838)
- Fix can't write more than 2 GB to a file
- Fix `StorageManager.get_file_size_bytes()` returns `ClientError` instead of `None` for invalid S3 links
- Fix Dataset lineage view is broken with multiple dataset dependencies
- Fix `tensorflow_macos` support
- Fix crash when calling `task.flush(wait_for_uploads=True)` while executing remotely
- Fix `None` values get casted to empty strings when connecting a dictionary

Bug fixes

- Set GCS credentials to `None` if invalid service account credentials are provided (841, thanks freddessert!)
- Fix a sync issue when loading deferred configuration