Clearml

New Features and Improvements

- Switch services agent entrypoint shell from `sh` to `bash` (141, thanks InCogNiTo124!)
- Improve poetry support
* Add poetry cwd support (142, thanks nielstenboom!)
* Add `agent.package_manager.poetry_install_extra_args` configuration option
- Do not allow request exceptions (keep retrying, throw error only on the initial login call)

Bug Fixes

- Fix agent update version (132, thanks achaiah!)
- Fix login uses GET with payload which breaks when trying to connect a server running in GCP
- Fix `clearml-agent build --docker` stuck on certain containers
- Fix build fails when target is relative path
- Fix pinging running task (change default to once a minute)
- Fix `_` is allowed in k8s label names
- Fix k8s glue does not delete pending pods if the tasks they represent were aborted
- Reintroduce `CLEARML_AGENT_SERVICES_DOCKER_RESTART` accidentally reverted by a previous merge
- Fix `git+ssh://` links inside installed packages not being converted properly to HTTPS-authenticated links

New Features and Improvements
- Upgrade requirements for `attrs`, `jsonschema`, `pyparsing`, `six` and `pyjwt` (129)
- Add default output URI selection to `clearml-agent init`
- Add `agent.disable_task_docker_override` configuration option to disable docker override specified in executing tasks
- Add `CLEARML_AGENT_FORCE_SYSTEM_SITE_PACKAGES` env var (default `true`) to allow overriding default `system_site_packages: true` behavior when running tasks in containers (docker mode and k8s-glue)

Bug Fixes
- Fix using deprecated types validator argument raises an error (deprecated even before `jsonschema` 3.0.0 and unsupported since 4.0.0)
- Fix pip support allowing multiple pip version constraints (by default, one for < Python 3.10 and one for >= Python 3.10)

New Features and Improvements
- Add support for single value metric reporting (400)
- Add support for specifying parameter sections in `PipelineDecorator` (629)
- Add support for parallel uploads and downloads (upload \ download and zip \ unzip of artifacts) ([ClearML Slack](https://clearml.slack.com/archives/CTK20V944/p1650467326569479))
- Add support for specifying execution details (repository, branch, commit, packages, image) in `PipelineDecorator`
- Bump PyJWT version due to _"Key confusion through non-blocklisted public key formats"_ vulnerability
- Add support for AWS Session Token (using boto3's `aws_session_token` argument)

Bug Fixes
- Fix `Task.get_projects()` retrieves only the first 500 results (612)
- Fix failure to delete artifacts stored in Azure (660)
- Fix Process Pool hangs at exit (674)
- Fix number of unpacked values when syncing a dataset (682)
- Fix FastAI DeprecationWarning (683)
- Fix `StorageManager.download_folder()` crash
- Fix pipelines can't handle `None` return value
- Fix pre-existing pipeline raises an exception
- Fix deprecation warning in the `image_reporting` example
- Fix patches are kept binded after`Task.close()` is called
- Fix running pipeline code remotely without first running it locally (i.e. no configuration on the Task)
- Fix local task execution with empty working directory
- Fix permission check fails when using local storage folder that does not exist
- Fix pipeline `add_function_step` breaks in remote execution
- Fix wrong mimetype used for any file or folder uploaded to S3 using `StorageManager`
- Add missing default `default_cache_manager_size` in configuration files

Bug Fixes

- Fix Process Pool hangs at exit (674)

New Features
- Add OpenMMLab example 655 (thanks zhouzaida!)
- Add support for saving artifacts with different formats 634
- Add support for setting reported values for `NaN` and `Inf` 604
- Support more than 500 results in `Task.get_tasks()` using the `fetch_only_first_page` argument 612
- Support links in `clearml-data` 585
- Support deferred task initialization using `Task.init()` argument `deferred_init` (beta feature)
- Support resuming experiments when importing an Offline session
- Add `--import-offline-session` command line option to `clearml-task`
- Support automatically logging Tensorboard Hparams
- Add wildcard support for model auto-logging, see `Task.init()` ([ClearML Slack](https://clearml.slack.com/archives/CTK20V944/p1644931337863039))
- Add support for Lightning CLI
- Support `None` values in `Task.connect()`
- Add Model.project getter/setter
- Add support for Task progress indication
- Datasets
* Improve Dataset version table
* Add warning to Dataset creation on current Task
- Examples and documentation
* Add manual seaborn logging example 628
* Change package author
* Change pipeline example to run locally 642
* Update Pytorch Lightning example for `pytorch-lightning>=v1.6.0` 650

Bug Fixes
- Fix Keras model config serialization in `PatchKerasModelIO` 616 (thanks bzamecnik!)
- Fix `task.get_parameters_as_dict(cast=True)` casts `False` to True 622 (thanks bewt85!)
- Fix Fire integration is not compatible with typing library 610
- Fix remote execution with `argparse` mutually exclusive groups raises "required" error even when no argument is required
- Fix Hydra tasks never fail and are only set to completed (fix handling return code)
- Fix `clearml-data` wildcard support
- Fix HPO randomly aborts running tasks before the time limit
- Fix matplotlib capture
- Fix issue with accessing images in projects containing `/`
- AutoScaler
* Fix resource name with a prefix matching a resource type may cause the auto-scaler to avoid spinning down idle instances
* Fix Idle workers should contain resource name and not instance type
- Fix backwards compatibility issue when using `abstractmethod`
- Matplotlib
* Fix uploading 3D plots with matplotlib plt shows 2D plot on task results page
* Fix wrong Histogram plotting using when matplotlib
- Fix PyTorch `ScriptModule` autobind
- Fix PyTorch auto-magic logging torchscript models
- Fix forked process will not call `_at_exit` and flush all outstanding reports
- Fix matplotlib to plotly conversion fails on subplots (convert as image if figure has subplots)
- Fix Windows sub process might end up waiting forever for uploads to finish if subprocess is very shot lived
- Fix `StorageManager.get_local_copy()` returning `None` for a valid path in Windows
- Fix Jupyter notebook cannot be detected
- Fix PipelineController does not change node Task name, only pipeline step name
- Fix `Task.query_tasks()` specifying page size or page number

New Features and Improvements
- Add support for setting reported values for `NaN` and `Inf` 604
- Add reserved OS environments warning
- Add git credentials to colab example 621 (thanks thepycoder!)
- Add jsonargparse support 403 (thanks ajecc and mauvilsa!)
- Update autokeras example

Bug Fixes
- Fix sub-project separators are incorrectly quoted in generated URLs 584
- Revert Optuna deprecation fix 613
- Fix HPO randomly aborts running tasks before the time limit
- Fix cloud driver overwrites `agent.extra_docker_arguments`
- Fix Pipeline Controller auto-magic framework connect
- Fix unused scroll is not cleared in `Task.get_reported_plots()`