Clearml

Bug Fixes
- `clearml-task` CLI link fixed

ClearML (*formerly allegro trains*)

**Breaking Changes**
Package renamed to `clearml`
python
from clearml import Task
...


Features
- `clearml-data` CLI for [Differential Dataset](https://github.com/allegroai/clearml/blob/master/docs/datasets.md) Management & Versioning
- `clearml-task` CLI for creating (importing) code into ClearML
- [Task.create](https://github.com/allegroai/clearml/blob/c8e0e6b7407a9801e9029cea94a3ec552fa85a32/clearml/task.py#L584) supports importing a code / repository into ClearML, including remote execution with `clearml-agent`

Bug Fixes
- PyJWT v2.0 breaks the interface

Features
- Add Hydra support 219
- Add cifar ignite example 237
- Add auto extraction of `tar.gz` files when using `StorageManager` 237
- Add `Task.init()` argument `auto_connect_streams` controlling stdout/stderr/logging capture 181
- Add carriage return flush support using the `sdk.development.worker.console_cr_flush_period` configuration setting 181
- Add `Task.create_function_task()` to allow creating a new task using a function and arguments to be executed remotely 230
- Allow disabling SSL certificates verification using `Task.setup_upload()` argument `verify` or AWS S3 bucket configuration `verify` property 256
- Add `StorageManager.get_files_server()`
- Add `Task.get_project_id()` using project name
- Add `project_name` argument to `Task.set_project()`
- Add `Task.connect()` support for class / instance objects
- Add `Task get_configuration_object() and `Task.set_configuration_object()` for easier automation
- Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings
- Use a built-in matplotlib convertor
- Add reporting text as debug sample example

Bug Fixes
- Fix Optuna HPO parameter serializing 254
- Fix connect dictionary `''` cast to `None` 258
- Fix lightgbm binding keyword argument issue 251
- Fix artifact preview if artifact body is remote URI 239
- Fix infinite recursion in `StorageManager` upload 253
- Fix keras reusing model object only if the filename is the same 252
- Fix running remotely with no configuration should not crash but output a warning 243
- matplotlib
- Fix matplotlib 3.3.3 support
- Fix global figure enumeration
- Fix binding without a title reported a single plot (`untitled 00`) instead of increasing the counter
- Fix Python 2.7/3.5 support
- Fix quote issue when reporting debug images
- Fix replace quote safe characters in upload file to include `;=$`
- Fix `at_exit` called from another process should be ignored
- Fix `Task.set_tags()` for completed / published tasks
- Fix `Task.add_tags()` not working when running remotely
- Fix `Task.set_user_properties()` docstring and interface
- Fix preview with JSON (dict) artifacts did not store the artifact
- Fix `Logger.report_text()` on task created using `Task.create()` was not supported
- Fix initialization for torch: only call torch `get_worker_info` if torch was loaded
- Fix flush (wait) on auxiliary task (obtained using` Task.get_task()`) should wait on all upload events
- Fix server was not updated with the defaults from the code when running remotely and configuration section is missing
- Fix connect dict containing `None` default values, blocked the remote execution from passing string instead of None
- Fix `Task.upload_artifact()` argument `delete_after_upload=True` used in conjunction with `wait_for_upload=True` was not supported

Features

- Add LightGBM support
- Add initial Hydra support 219
- Add synchronous support for `Task.upload_artifact()` 231
- Add `sdk.development.store_code_diff_from_remote` (default `false`) to store diff from remote HEAD instead of local HEAD 222
- Add `sdk.development.detect_with_conda_freeze` (default `true`) for full conda freeze (requires trains-agent >= 16.2)
- Add user properties support using `Task.get_user_properties`, `Task.set_user_properties()` and `Task.delete_user_properties()`
- Add `Logger.report_table()` support for table as list of lists
- Add support to split DAG and Table in pipeline DAG plot. Pipeline DAG single nodes are now round circles below the DAG graph.
- Add Pipeline/Optimization can be attached to any Task (not just the current task)
- Add `force_download` flag to `StorageManager.get_local_copy()`
- Add control over the artifact preview using `Task.upload_artifact()` `preview` argument
- Add `Logger.report_matplotlib_figure()` with examples
- Add `Task.set_task_type()`
- AWS auto-scaler
- Add key pair and security groups support
- Add multi-line support for both extra bash script and extra `trains.conf` data
- Update examples

Bug Fixes

- Fix `Task.update_output_model()` wrong argument order 220
- Fix initializing task on argparse parse in remote mode. Do not call `Task.init()` to avoid auto connect, use `Task.get_task()` instead
- Fix detected task cwd outside of repository root folder
- Fix `Task.connect(dict)` to place non-existing entries on the section name instead of General
- Fix `Task.clone()` support for trains-server < 0.16
- Fix `StorageManager` cache extract zipped artifacts. Use modified time instead of access time for cached files
- Fix diff command output was stripped
- Make sure local packages with multi-files are marked as `package`
- Fix `Task.set_base_docker()` should be skipped when running remotely
- Fix ArgParser binding handling of string argument with boolean default value (affects Pytorch Lightning integration)
- When using `detect_with_pip_freeze` make sure that `package file://` lines are replaced with `package==x.y.z` as local file will probably not be available
- Fix git packages to new pip standard `package git+`
- Improve conda package naming `_` and `-` support
- Do not add specific setuptools version to requirements (pip can't install it anyway)
- Fix image URL quoting when uploading from a file path

Features

- Add `Task.set_resource_monitor_iteration_timeout()` to set ResourceMonitor iteration wait duration timeout 208
- Add PyTorch Lightning save/restore model binding 212
- Add `git diff` for repository submodule (requires git 2.14 or above)
- Add `TrainsJob.is_completed()` and `TrainsJob.is_aborted()`
- Add `Task.logger` property
- Add Pipeline Controller automation and example (see ***here***)
- Add improved trace filtering capabilities in `trains.debugging.trace.trace_trains()`
- Add default help per argument (if not provided) in ArgParser binding
- Deprecate `Task.reporter`
- Update PyTorch example
- Remove warning on skipped auto-magic model logging 206
- Support Keras restructuring for Network, Model and Sequential
- Update autokeras requirements according to https://github.com/keras-team/autokeras#installation

Bug Fixes

- Fix joblib auto logging models failing on compressed streams 203
- Fix sending empty reports 205
- Fix scatter2d sub-sampling and rounding
- Fix plots reporting
* `NaN` representation (matplotlib conversion)
* Limit the number of digits in a plot to reduce plot size (using `sdk.metrics.plot_max_num_digits` configuration value)
- Fix `Task.wait_for_status()` to reload after it ends
- Fix thread wait Ctrl-C interrupt did not exit process
- Improve Windows support for installed packages analysis
- Fix auto model logging using relative path
- Fix Hyper-parameter Optimization example
- Fix `Task.clone()` when working with TrainsServer < 0.16.0
- Fix pandas artifact handling
* Avoid adding `unnamed:0` column
* Return original pandas object
- Fix `TrainsJob` hyper-params overriding order was not guaranteed
- Fix ArgParse auto-connect to support default function type

Features and Bug Fixes

- Fix `typing` dependency for Python<3.5 184
- Fix `git+https` requirements handling, resolve `top_level.txt` package name (`kerastuner` from git was not detected)
- Fix `Task.get_reported_console_output()` for new Trains Server API v2.9
- Fix cache handling for different partitions/drives/devices
- Disable offline mode when running remotely (i.e. executed by Trains Agent)
- Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) 189
- Fix double-escaped model design text when connecting OutputModel
- Enhance HyperParameter optimizer