Clearml

New Features and Improvements
- Support running jupyter notebook inside a git repository (repository will be referenced without uncommitted changes and jupyter notebook will be stored om plain code as uncommitted changes)
- Add jupyter notebook fail warning
- Allow pipeline steps to return string paths without them being treated as a folder artifact and zipped (780)
- Remove `future` from Python 3 requirements

Bug Fixes
- Fix exception raised when using `ThreadPool` (790)
- Fix Pyplot/Matplotlib binding reports incorrect line labels and colors (791)
- Pipelines
- Fix crash when running cloned pipeline that invokes a step twice (770, related to 769, thanks tonyd!)
- Fix pipeline argument becomes None if default value is not set
- Fix `retry_on_failure` callback does nothing when specified on `PipelineController.add_step()`
- Fix pipeline clone logic
- Jupyter Notebook
- Fix support for multiple jupyter servers running on the same machine
- Fix issue with old/new notebook packages installed
- Fix local cache with access rules disabling partial local access
- Fix `Task.upload_artifact()` fails uploading pandas `DataFrame`
- Fix relative paths in examples (787, thanks mendrugory!)

New Features and Improvements
- Add callback option for pipeline step retry

Bug Fixes
- Fix Python Fire binding
- Fix Dataset failing to load helper packages should not crash
- Fix `Dataset.get_local_copy()` is allowed for a non-finalized dataset
- Fix `Task.upload_artifact()` does not upload empty lists/tuples
- Fix pipeline retry mechanism interface
- Fix Python <3.5 compatibility
- Fix local cache warning (should be a debug message)

New Features
- Add `agent.docker_args_extra_precedes_task` and `agent.protected_docker_extra_args` configuration settings to prevent the same switch to be used by both `agent.extra_docker_args` and the a Task's docker args
- Add `agent.resource_monitoring.disk_use_path` configuration option to allow monitoring a different volume than the one containing the home folder
- Change default `agent.enable_git_ask_pass` to true
- Add example and support for pre-built containers including services-mode support with overrides `CLEARML_AGENT_FORCE_CODE_DIR` and `CLEARML_AGENT_FORCE_EXEC_SCRIPT`
- Add `CLEARML_AGENT_SERVICE_TASK=1` environment variable in case we're running a service task
- Add `CLEARML_AGENT_TEMP_STDOUT_FILE_DIR` to allow specifying temp dir used for storing agent log files and temporary log files (daemon and execution)
- Update GPU stats and pynvml support
- Add git clone verbosity using `CLEARML_AGENT_GIT_CLONE_VERBOSE` environment variable
- k8s glue
* Add status reason when aborting before moving to k8s_scheduler queue
* When cleaning up pending pods, verify task is still aborted and pod is still pending before deleting the pod
* Set worker ID in k8s pod execution


Bug Fixes
- Fix `agent.package_manager.poetry_install_extra_args` are used in all Poetry commands and not just in `install` (173)
- Fix if process return code is SIGKILL (-9 or 137) and abort callback was called, do not mark as failed but as aborted
- Fix `agent.git_host` setting will cause gitdomain URLs to not be replaced by SSH URLs since furl cannot parse them to obtain host
- Fix an environment variable that should be set with a numerical value of `0` (i.e. end up as `"0"` or `"0.0"`) is set to an empty string
- Fix `agent.package_manager.extra_index_url` URLs are not sanitized in configuration printout
- Fix recursion issue when deep-copying a session
- k8s glue
* Fix k8s glue configuration might be contaminated when changed during apply
* Fix `KeyError` if container does not contain the arguments field

Bug Fixes

- Fix `APIClient` fails when calling `get_all` endpoints with API 2.20 (affects CLI tools such as `clearml-session`)

New Features and Improvements

- Add option to specify an endpoint URL when creating S3 resource service (679, thanks AndolsiZied!)
- Add support for providing `ExtraArgs` to boto3 when uploading files using the `sdk.aws.s3.extra_args` configuration option
- Add support for Server API 2.20
- Add `Task.get_num_enqueued_tasks()` to get the number of tasks enqueued in a specific queue
- Add support for updating model metadata using `Model.set_metadata()`, `Model.get_metadata()`, `Model.get_all_metadata()`, `Model.get_all_metadata_casted()` and `Model.set_all_metadata()`
- Add `Task.get_reported_single_value()`
- Add a retry mechanism for models and artifacts upload
- Pipelines with empty configuration takes it from code
- Add support for running pipeline steps on preemptible instances
- Datasets
- Add description to Datasets
- Add wild-card support in `clearml-data`

Bug Fixes

- Fix dataset download (713, thanks dankirsdot!)
- Fix lock is not released after dataset cache is downloaded (708, thanks mralgos!)
- Fix deadlock might occur when using process pool large number processes (674)
- Fix 'series' not appearing on UI when using `logger.report_table()` (684)
- Fix `Task.init()` docstring to include behavior when executing remotely (737, thanks mmiller-max!)
- Fix `KeyError` when running remotely and no params were passed to click (https://github.com/allegroai/clearml-agent/issues/111)
- Fix full path is stored when uploading a single artifact file
- Fix passing non-alphanumeric filename in `sdk.development.detect_with_pip_freeze`
- Fix Python 3.6 and 3.10 support
- Fix mimetype cannot be `None` when uploading to S3
- Pipelines
- Fix pipeline DAG
- Add support for pipelines with spot instances
- Fix pipeline proxy object is always resolved in main pipeline logic
- Fix pipeline steps with empty configuration should try and take it from code
- Fix wait for jobs based on local/remote pool frequency
- Fix `UniformIntegerParameterRange.to_list()` ignores min value
- Fix pipeline component returning a list of length 1
- Datasets
- Fix `Dataset.get()` does not respect `auto_create`
- Fix getting datasets fails with new ClearML Server v1.6
- Fix datasets can't be queried by project/name alone
- Fix adding child dataset to older parent dataset without stats
- Fix error when connecting an input model
- Fix deadlocks, including:
- Change thread Event/Lock to a process fork safe threading objects
- Use file lock instead of process lock to avoid future deadlocks since python process lock is not process safe (killing a process holding a lock will Not release the lock)
- Fix `StorageManager.list()` on a local Windows path
- Fix model not created in the current project
- Fix `keras_tuner_cifar` example raises `DeprecationWarning` and `ValueError`

Bug Fixes

- Fix format string construction sometimes causing delayed evaluation errors (706)