Clearml-agent

Features

- Add virtual environment caching
* Supports venv caching both in standard and docker mode
* Configurable using the `agent.venvs_cache` configuration section
* Disabled by default, enable [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L101)
- Add support for `--services-mode` with venvs
- Add `agent.force_git_ssh_user` configuration value (default `git`, see [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L29)) 42
- Add `agent.ignore_requested_python_version` configuration option for multi python environments (default `false`)
- Add `agent.enable_task_env` configuration option to set the OS environment based on the Environment section of the Task (default `false`, see [here](https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L151))
- K8s glue
* Add support for detecting and deleting k8s pods that fail to start
* Allow providing namespace in k8s glue and k8s glue example
* Add base-pod-number parameter to k8s glue and example
- Change `agent.default_docker.image` to `nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04` (see [here](https://github.com/allegroai/clearml-agent/blob/8a46dc6b03860bc1cc022fd19e86bf4524986a0e/docs/clearml.conf#L144))
- Use shared git cache for multiple agents on the same machine
- Upgrade pynvml add detect CUDA version from driver level
- Update agent and services docker files
- Update documentation

Bug Fixes

- Fix `docker --network` returns `None`
- Fix docker mode without venvs cache dir
- Fix applying git diff on a newly added file
- Fix environment variables `CLEARML_WEB_HOST`/`CLEARML_FILES_HOST` not passed to running tasks (or updated on the config object)
- Fix `--detached` command line option not supported on Windows (ignore and issue warning)
- Fix file not found error (errno 2) interpreted as aborted (i.e. `Ctrl-C`)
- Fix `from clearml` runtime diff patching
- Fix cache to take cuda version into account
- Fix CPU mode
- Fix multi instances on Windows
- Fix conda support for `git+http` links
- Fix k8s glue does not pass docker environment variables, remove deprecated flags

ClearML-Agent (formerly allegro trains-agent)

Features and Bug Fixes

- Fix support for pip virtual-environment on Windows
- Fix support for conda using repository requirements.txt (empty "Installed Packages" section)

ClearML-Agent (*formerly allegro trains-agent*)

**Breaking Changes**
Package renamed to `clearml-agent`

bash
pip install clearml-agent
clearml-agent daemon --docker ...

Features and Bug Fixes
- Update PyJWT requirement (v2.0.0 breaks interface)
- Update other requirements constraints
- Change k8s pod naming scheme in k8s glue to include queue name, conform queue name to k8s standard

Features
- conda
- Add `agent.package_manager.conda_env_as_base_docker` allowing "docker_cmd" to contain link to a full pre-packaged conda environment (`tar.gz` created by `conda-pack`). Use `TRAINS_CONDA_ENV_PACKAGE` environment variable to specify `conda tar.gz` file.
- Add conda support for read-only pre-built environment (pass conda folder as `docker_cmd` on Task)
- Improve trying to find conda executable
- k8s glue
- Add support for limited number of services exposing ports
- Add support for k8s pod custom user properties
- Allow selecting external `trains.conf` file for the pod itself
- Allow providing pod template, extra bash init script, alternate SSH server port, gateway address (k8s ingress/ELB)
- Allow specifying `cudatoolkit` version in the "installed packages" section when using conda as package manager https://github.com/allegroai/trains/issues/229
- Add `agent.package_manager.force_repo_requirements_txt`. If True, "Installed Packages" on Task are ignored, and only repository `requirements.txt` is used
- Pass `TRAINS_DOCKER_IMAGE` into docker for interactive sessions
- Add `torchcsprng` and `torchtext` to PyTorch resolving

Bug Fixes
- When logging suppress "\r" when reading a current chunk of a file/stream. Add `agent.suppress_carriage_return` (default True) to support previous behavior
- Make sure `TRAINS_AGENT_K8S_HOST_MOUNT` is used only once per mount
- Fix k8s glue script to trains-agent default docker script
- Fix apply git diff from submodule only
- conda
- Fix conda pip freeze to be consistent with trains 0.16.3
- Fix conda environment support for trains 0.16.3 full env. Add `agent.package_manager.conda_full_env_update` to allow conda to update back the requirements (default False, to preserve previous behavior)
- Fix running from conda environment - `conda.sh` not found in first conda PATH match
- Fix docker mode ubuntu/debian support by making sure not to ask for input (fix `tzdata` install)
- Fix repository detection - ignore environment `SSH_AUTH_SOCK`, only check if git user/pass are configured
- git diff
- Fix support for non-ascii diff
- Fix diff with empty line at the end will cause corrupt diff apply message
- Allow zero context diffs (useful when blind patching repository)
- Fix `daemon --stop` when agent UID cannot be located
- Fix nvidia docker support on some linux distros (SUSE)
- Fix nvidia pytorch dockers support
- Fix torch CUDA 11.1 support
- Fix requirements dict with null entry in `pip` should be considered None install from repository's `requirements.txt`

Features

- Add `sdk.metrics.plot_max_num_digits` configuration option to reduce plot storage size
- Add `agent.package_manager.post_packages` and `agent.package_manager.post_optional_packages` configuration options to control packages install order (e.g. horovod)
- Add `agent.git_host` configuration option for limiting git credential usage for a specific host (overridable using `TRAINS_AGENT_GIT_HOST` environment variable)
- Add `agent.force_git_ssh_port` configuration option to control `https` to `ssh` link conversion for non standard `ssh` ports
- Add requirements detection features
* Improve support for detecting new pip version (20+) supporting `package scheme://link`

Bug Fixes

- Fix pre-installed packages are ignored when installing a git package wheel. Reinstalling a `git+http` link is enough to make sure all requirements are met/installed https://github.com/allegroai/trains/issues/196
- Fix incorrect check for spaces in current execution folder
- Fix requirements detection
* Update torch version after using downloaded / system pre-installed version
* Do not install git packages twice when a new pip version is used (pip freeze will detect the correct git link version)