Clearml-agent

Features

- Add `agent.docker_init_bash_script` configuration section to allow finer control over docker startup script
- Changed default docker image from `nvidia/cuda` to `nvidia/cuda:10.1-runtime-ubuntu18.04` to support `cudnn` frameworks (e.g. TF)
- Improve support for dockers with preinstalled `conda` environment
- Improve trains-agent-docker spinning
- Add `daemon --order-fairness` for round-robin queue pulling
- Add `daemon --stop` to terminate a running agent (assuming other arguments are the same)
- If no additional arguments, Agents are terminated in lexicographical order
- Support cleanup of all log files on termination unless executed with `--debug`
- Add error message when Trains API Server is not accessible on startup

Bug Fixes

- Fix GPU Windows monitoring support https://github.com/allegroai/trains/issues/177
- Fix `.git-credentials` and `.gitconfig` mapping into docker
- Fix non-root docker image usage
- Fix docker to use `UTF-8` encoding, so prints won't break it
- Fix `--debug` to set all loggers to `DEBUG`
- Fix task status change to `queued` should never happen during Task runtime
- Fix `requirement_parser` to support `package git+http` lines
- Fix GIT user/password in requirements and support for `-e git+http` lines
- Fix configuration wizard to generate `trains.conf` matching latest Trains definitions

Features

- Add Trains Agent Daemon and Services docker files

Bug Fixes

- Fix initialization wizard (allow at most two verification retries, then print error)
- Add warning on `--gpus` with no detected CUDA version 24
- Add `agent.force_git_ssh_protocol` configuration option to force all git links to `ssh://` 16
- Add git user/pass permission into pip package installation from Git repository 22

Features

- Add daemon Services Mode (`daemon --services-mode`) where the daemon spins a task in its own docker and verifies start-up and shut-down. This allows multiple tasks to be launched simultaneously on the same machine (currently in CPU mode only), where each task service will register itself as a worker for the lifetime of the task
- Enhance `build --docker` mode
- Add `--install-globally` option to install required packages in the docker's system python
- Add `--entry-point` option to allow automatic task cloning when running the docker
- Support PyTorch Nightly builds using the `agent.torch_nightly` configuration flag. If `true`, the agent looks for a nightly build when a stable torch wheel is not found
- Add environment variables support for git user/password
- Using `TRAINS_AGENT_GIT_USER`/`TRAINS_AGENT_GIT_PASS`
- Pass git credentials to dockerized experiment execution
- Support running code from module (i.e. `-m` in execution entry point)
- Add daemon `--create-queue` to automatically create a queue and use it if queue name doesn't exist in the server
- Move `--gpus` and `--cpu-only` to worker args (used by daemon, execute and build)

Bug Fixes

- Fix init wizard, correctly display the input servers 19
- Fix version control links in requirements when using `conda`
- Fix `build --docker` mode standalone docker execution
- Improve docker host-mount support, use `TRAINS_AGENT_DOCKER_HOST_MOUNT` environment variable
- Support `pip` v20.1 local/http package reference in `pip freeze`
- Fix detached mode to correctly use cache folder slots
- Fix `CUDA_VISIBLE_DEVICES` should never be set to "all" (Trains Slack channel [thread](https://allegroai-trains.slack.com/archives/CTK20V944/p1590563064305700))
- Do not monitor GPU when running with `--cpu-only`

Features and Bug Fixes

- Add daemon detached mode (`--detached`, `-d`) that runs the agent as daemon in the background and returns immediately
- Auto mount `~/.git-credentials` into docker container (if file exists)
- Add `TRAINS_AGENT_EXTRA_PYTHON_PATH` environment variable to allow adding additional python path during experiment execution (helpful when using extra un-tracked modules)
- Fix "run as user" feature (using `TRAINS_AGENT_EXEC_USER` environment variable)
- Fix PyTorch support to ignore minor versions when looking for package to install/download
- Fix experiment execution output handling

Features and Bug Fixes

- Add support for `trains-agent execute --id <experiment-id> --docker` that allows executing a specific experiment inside a docker container
- Add support for `trains-agent execute --id <template-experiment-id> --clone` that clones the provided experiment and executes the cloned experiment
- Add support for `APIClient.models.delete()` to allow programmatically deleting a model https://github.com/allegroai/trains-server/issues/32
- Add daemon support for passing storage-related OS environment variables to experiments executed inside a docker container (supported by `trains>=0.13.3`):
* AWS: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION`
* Azure: `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_KEY`
* Google: `GOOGLE_APPLICATION_CREDENTIALS`
- Fix git checkout with submodules https://github.com/allegroai/trains/issues/112
- Prefer docker image from command line over the one specified in experiment

Features and Bug Fixes

- Allow providing queue names instead of queue IDs in daemon mode
- Docker mode improvements
* Support running as a specific user inside a docker using the `TRAINS_AGENT_EXEC_USER` environment flag
* Pass correct GPU limit when skipping gpus flag
* Add `--force-current-version` daemon command-line flag
- Add K8s/trains glue service example
- Added K8s support in daemon mode
* Running inside a K8s pod
* Mounting dockerized experiment folders to host
* Allow a specific network for the docker
- Add default storage environment vars (for AWS, GS and Azure) to generated agent configuration
- Improve Unicode/UTF stdout handling