Clearml

Features

- Use default `boto` credential chain if no keys are provided in the configuration file or environment variables 342
- Support `DummyModel` configuration [ClearML Slack thread](https://clearml.slack.com/archives/CTK20V944/p1621469235085400)
- Add `report_matplotlib_figure(..., report_interactive=False)` allowing to upload a matplotlib as a non-interactive (high quality png) plot
- Add `Logger.matplotlib_force_report_non_interactive()`
- Remove matplotlib axis range (`plotly.js` auto-range can adjust it in real-time)
- Add object-storage support in cleanup-service
- Add `dataset_tags` argument to `Dataset.create()`
- Expose `docker_args` and `docker_bash_setup_script` in `clearml-task` CLI
- Add logging for Nvidia driver and Cuda version
- Add optional ignored packages in script requirements (currently used for `pywin32`)
- Update examples
* Increase channel result to support max of 1K channels for finding slack channel and use cursor in Slack Alerts monitoring service
* Add `csv` data sample to `data_samples`
* Remove deprecated examples

Bug Fixes

- Fix Hydra should not store the full resolved OmegaConf 327
- Fix direct import of keras save/load model functions 355
- Fix run as module 359
- Fix Python 2.7 support 366
- Fix `Task.add_requirements()` passing `package_version` starting with ``, `;` or ``
- Fix import keras from TF
- Fix support for Hydra's `run_job()` change in parameter order by passing `config` and `task_function` as keyword arguments
- Fix background upload retries with Google Storage (`gs://`)
- Fix Python 3.8 race condition in `Task.close()`
- Fix shutting down a Task immediately after creation might block
- Fix `Task.execute_remotely()` from Jupyter notebook
- Fix Jupyter Notebook inside VSCode
- Fix support for `Dataset.create()` argument `use_current_task`
- Fix `Dataset.finalize()` can hang in extreme scenario
- Protect against wrong file object type when auto-binding models
- Fix matplotlib date convertor
- Fix automation controller overrides nodes clone

Bug Fixes

* Fix in rare scenarios process stuck on exit, again 🙁 🙂

Bug Fixes
- Fix in rare scenarios process stuck on exit

1. Take down the serving containers (docker-compose or k8s)
2. Update the clearml-serving CLI `pip3 install -U clearml-serving`
3. Re-add a single existing endpoint with `clearml-serving model add ...` (press yes when asked)
(it will upgrade the clearml-serving session definitions)
4. Pull latest serving containers (`docker-compose pull ...` or k8s)
5. Re-spin serving containers (docker-compose or k8s)

Features
- Add conda and pip environment debug prints (using `--debug`)
- Add support for PyJWT v2
- Change the default conda channel order, so it pulls the correct `pytorch` package
- Improve k8s glue support
* Support k8s glue container env vars merging
* Add number of pods limit to k8s glue using the `max_pods_limit` argument (use `--max-pods` switch in the k8s glue example)
* Add k8s glue default `restartPolicy=Never` to template to prevent pods from restarting
- Add `--stop` switch support for dynamic gpus
- Verify `docker` command exists when running in docker mode
- Add support for terminating dockers on `sig_term` in dynamic mode
- Add stopping message on Task process termination
- Add `agent.docker_install_opencv_libs` configuration option to enable automatic opencv libs install for faster docker spin-up (default: `true`, see [here](https://github.com/allegroai/clearml-agent/blob/4f18bb7ea0600db6ee63ecb5ad0ea8d048a272ca/docs/clearml.conf#L139))
- Add support for new container base setup script feature
- Bump virtualenv dependency version (support `v>=16,<21`)
- Add support for dynamic gpus opportunistic scheduling (with min/max gpus per queue)
- Deprecate `venv_update` in configuration (replaced by the more robust `venvs_cache`)
- Add Python 3.9 to the support table

Bug Fixes
- Fix agent can return non-zero error code and pods will end up restarting forever 56
- Fix poetry support 57
- Fix cuda version from driver does not return minor version
- Fix requirements local path replace back when using cache
- Fix k8s glue
* Fix broken k8s glue docker args parsing
* Fix empty env prevents override when merging template
- Fix venv cache crash on bad symbolic links
- Fix no docker arguments provided

Features

- Add `force_download` argument to `Artifact.get()` and `Artifact.get_local_copy()` 319
- Support all reporting using subprocesses instead of threads (default `sdk.development.report_use_subprocess=True`)
- Improve Datasets support
* Add `clearml-data publish` to allow publishing a dataset task
* Add `clearml-data` sync can now create, sync and close a dataset with a single command
* Allow Dataset to be created on a running Task
* Add `dataset_tags` argument to `Dataset.get()`
* Add `Dataset.get_logger()`
- Add `Task.add_requirements()` support for specifying version `<>=~` etc.
- Add `StorageManager.upload_folder()` and `StorageManager.download_folder()`
- Add progress report logging for `StorageHelper.upload_from_stream()`, `StorageHelper.upload()` and `StorageHelper.upload_from_stream()`
- Add jupyter auto-magic store jupyter notebook as an artifact on the Task (default `sdk.development.store_jupyter_notebook_artifact=True`)
- Add upload HTML preview of jupyter notebook as an artifact
- Add `PipelineController` disable clone base task option
- Add links to Tasks in optimization summary table (not just Task IDs)
- Add support for datetime in scatter plots + matplotlib support
- Improve plotly value type conforming
- Improve PyTorch `DataLoader` speed 207
- Update Auto Scaler default values and configuration
- Examples
* Add Hydra example
* Add artifacts retrieval example
* Update various examples

Bug Fixes

- Fix warning or error message if requirements parsing failed 291
- Fix pytorch-lighting multi node store 292
- Fix strip remote diff 295
- Fix python package detection `sklearn` -> `scikit-learn` 296
- Fix argparse issues
* Fix argparse with `[None]` in default parameter 297
* Fix parsing of arguments in scientific notation 313
* Fix argparser logging always captures defaults (Windows only, ignored cmd)
* Fix argparse `nargs` passed in command line `--nargs 1 2` should be stored as `[1, 2]` not as `['1', '2']`
* Fix support for nonstandard argparse with default value that is not of defined type
* Fix server updated with the argparse in remote before Task.init() is called (respect skipped args)
- Fix Dataset support
* Fix `Dataset.remove_files()` can't find files when files are in dataset root 303
* Fix closing a dataset with only files removed
* Fix Dataset generate removed / modified / added of summary table
- Fix hydra multi-run support 306
- Fix TF/TensorBoard support
* Fix TensorBoard multiple `Task.init()`/`Task.close()` calls within the same process 312
* Fix TensorBoard 2+ `pr_curve`
* Fix TF `pr_curve` should not be inverted
* Fix TF +2.3 mixed eager mode execution summary metrics not reported
* Fix TF bind keyboard interrupt
* Fix TF 2.4 keras load/save model
- Fix `clearml-task`
* Fix error when script cannot be found
* Fix `--docker` flag not passed
* Fix patching local git diff
- Fix `clearml-data`
* Fix `clearml-data sync` requires `--name`
* Fix missing required argument `--files` in `clearml-data remove`
- Fix `Task.execute_remotely()` from Jupyter Notebook
- Fix populate Task called from Jupyter Notebook (use `Task.create(packages=True)` to auto populate based on locally installed packages)
- Fix plotly plot with numpy containing `NaN`/`datetime`
- Fix matplotlib with Agg backend (or in remote execution)
- Fix trying to upload model file as a folder (automatically package the folder)
- Fix broken packages on package `importlib` detection failed the entire requirements detection
- Fix `Task.connect(object)` should always return the same object instance
- Fix `Task.create()` with repo and script that exists locally
- Fix crash in case `Logger.get_logger()` cannot get the file name
- Fix exception at exit in python 3.8+ on MacOS/Windows
- Fix make pipeline summary table link to Task step logs
- Fix Hydra 1.1 support (argparse description)
- Fix close task after logger is closed
- Fix `Task.set_base_docker()` in remote execution
- Fix artifact preview limit to 64Kb
- Fix JupyterLab Notebook detection
- Fix Python 2.7 support