Clearml

Features and Bug Fixes
- Add `HyperParameterOptimizer` parallel coordinates 279
- Add `Task.init()` argument `tags`
- Change HPO parallel coordinates color scale, yellow is low
- Change `HyperParameterOptimizer` `spawn_task_project` to `spawn_project`
- Use only lower limit constraint for numpy
- Fix argparse `nargs` support was broken
- Fix argparse with `action="append"`
- Fix PyJWT v2.0 token parsing
- Fix python package detection should not list `file://` links

Features
- Add `Task.delete()` support
- Add `Task.debug_simulate_remote_task()` to simulate task execution by ClearML Agent
- Add warning on archived Task in pipeline 274
- Add `Task.init(..., output_uri=True)` will use the default files_server as output uri
- Make `clearml-data` CLI stateful, remember last dataset ID as default dataset
- Added `HyperParameterOptimizer.get_optimizer_top_experiments()` for querying post execution optimization pipeline
- Add `Task.set_archived()` and `Task.get_archived()`
- Add `Task.set_credentials()` option to store into credentials file
- Add `clearml-data` close now auto uploads
- Add `HyperParameterOptimizer` arguments `spawn_task_project` and `save_top_k_tasks_only`


Bug Fixes
- Fix `PipelineController` running remotely without configuration does not execute the default code pipeline 273
- Fix reusing task after its project was deleted 274
- Fix `Task.archived_tag` read-only property does not work 274
- Fix argparse support to store consistent str representation of custom objects. Avoid changing default value if remote value matches
- Fix argsparse type as function
- Fix Dataset add single and multiple file(s)
- Fix get project name from parent dataset if not specified
- Fix mpl exporter. Added support for legend
- Fix model upload
- Fix optimizer callback best experiment
- Fix Optuna optimizer failing on tasks with `None` value in scalar query
- Fix auto python package detection installed directly from URLs
- Fix dataset upload aborted on server watchdog
- Fix dataset genealogy, graph and restoring data
- Fix numpy dependency for python versions

Features

- Add virtual environment caching
* Supports venv caching both in standard and docker mode
* Configurable using the `agent.venvs_cache` configuration section
* Disabled by default, enable [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L101)
- Add support for `--services-mode` with venvs
- Add `agent.force_git_ssh_user` configuration value (default `git`, see [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L29)) 42
- Add `agent.ignore_requested_python_version` configuration option for multi python environments (default `false`)
- Add `agent.enable_task_env` configuration option to set the OS environment based on the Environment section of the Task (default `false`, see [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L151))
- K8s glue
* Add support for detecting and deleting k8s pods that fail to start
* Allow providing namespace in k8s glue and k8s glue example
* Add base-pod-number parameter to k8s glue and example
- Change `agent.default_docker.image` to `nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04` (see [here](https://github.com/allegroai/clearml-agent/blob/8a46dc6b03860bc1cc022fd19e86bf4524986a0e/docs/clearml.conf#L144))
- Use shared git cache for multiple agents on the same machine
- Upgrade pynvml add detect CUDA version from driver level
- Update agent and services docker files
- Update documentation

Bug Fixes

- Fix `docker --network` returns `None`
- Fix docker mode without venvs cache dir
- Fix applying git diff on a newly added file
- Fix environment variables `CLEARML_WEB_HOST`/`CLEARML_FILES_HOST` not passed to running tasks (or updated on the config object)
- Fix `--detached` command line option not supported on Windows (ignore and issue warning)
- Fix file not found error (errno 2) interpreted as aborted (i.e. `Ctrl-C`)
- Fix `from clearml` runtime diff patching
- Fix cache to take cuda version into account
- Fix CPU mode
- Fix multi instances on Windows
- Fix conda support for `git+http` links
- Fix k8s glue does not pass docker environment variables, remove deprecated flags

ClearML-Agent (formerly allegro trains-agent)

Features and Bug Fixes

- Fix support for pip virtual-environment on Windows
- Fix support for conda using repository requirements.txt (empty "Installed Packages" section)

ClearML-Agent (*formerly allegro trains-agent*)

**Breaking Changes**
Package renamed to `clearml-agent`

bash
pip install clearml-agent
clearml-agent daemon --docker ...

Features
- Add Hydra support 219
- Add cifar ignite example 237
- Add auto extraction of `tar.gz` files when using `StorageManager` 237
- Add `Task.init()` argument `auto_connect_streams` controlling stdout/stderr/logging capture 181
- Add carriage return flush support using the `sdk.development.worker.console_cr_flush_period` configuration setting 181
- Add `Task.create_function_task()` to allow creating a new task using a function and arguments to be executed remotely 230
- Allow disabling SSL certificates verification using `Task.setup_upload()` argument `verify` or AWS S3 bucket configuration `verify` property 256
- Add `StorageManager.get_files_server()`
- Add `Task.get_project_id()` using project name
- Add `project_name` argument to `Task.set_project()`
- Add `Task.connect()` support for class / instance objects
- Add `Task get_configuration_object() and `Task.set_configuration_object()` for easier automation
- Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings
- Use a built-in matplotlib convertor
- Add reporting text as debug sample example

Bug Fixes
- Fix Optuna HPO parameter serializing 254
- Fix connect dictionary `''` cast to `None` 258
- Fix lightgbm binding keyword argument issue 251
- Fix artifact preview if artifact body is remote URI 239
- Fix infinite recursion in `StorageManager` upload 253
- Fix keras reusing model object only if the filename is the same 252
- Fix running remotely with no configuration should not crash but output a warning 243
- matplotlib
- Fix matplotlib 3.3.3 support
- Fix global figure enumeration
- Fix binding without a title reported a single plot (`untitled 00`) instead of increasing the counter
- Fix Python 2.7/3.5 support
- Fix quote issue when reporting debug images
- Fix replace quote safe characters in upload file to include `;=$`
- Fix `at_exit` called from another process should be ignored
- Fix `Task.set_tags()` for completed / published tasks
- Fix `Task.add_tags()` not working when running remotely
- Fix `Task.set_user_properties()` docstring and interface
- Fix preview with JSON (dict) artifacts did not store the artifact
- Fix `Logger.report_text()` on task created using `Task.create()` was not supported
- Fix initialization for torch: only call torch `get_worker_info` if torch was loaded
- Fix flush (wait) on auxiliary task (obtained using` Task.get_task()`) should wait on all upload events
- Fix server was not updated with the defaults from the code when running remotely and configuration section is missing
- Fix connect dict containing `None` default values, blocked the remote execution from passing string instead of None
- Fix `Task.upload_artifact()` argument `delete_after_upload=True` used in conjunction with `wait_for_upload=True` was not supported