Clearml

New Features and Bug Fixes

- Add default pip version support for Python 3.12

New Features
- Add NO_DOCKER flag to clearml-agent-services entrypoint (206, thanks valentinschabschneider!)
- Use `venv` module if `virtualenv` is not supported
- Find the correct python version when using a pre-installed python environment
- Add `/bin/bash` support in the task's `script.binary` property
- Add support for `.ipynb` script entry files (install nbconvert in runtime, convert file to python and execute the python script). Includes `CLEARML_AGENT_FORCE_TASK_INIT` patching of `.ipynb` files (post python conversion)
- Add `CLEARML_MULTI_NODE_SINGLE_TASK` (values -1, 0, 1, 2) for easier multi-node single Task workloads
- Add default docker `agent.default_docker.match_rules` configuration option supported by enterprise servers (note: matching_rules are ignored if `--docker container` is passed in command line)
- Add `-m module args` in script entry now supports standalone script. Standalone script is placed in a file specified by the `working_dir` setting in the `<dir>:<target_file>` format (e.g `.:standalone.py`), or in _untitled.py_ if not specified
- Add `K8S_GLUE_POD_USE_IMAGE_ENTRYPOINT` env var to allow running k8s pods without overriding the image entrypoint (useful for agents using prebuilt images in k8s)
- Add venv cache mount override for non-root containers (use: `agent.docker_internal_mounts.venvs_cache`)
- Add `/bin/bash -c "command"` support. Task `binary` should be set to `/bin/bash` and `entry_point` should be set to `-c command`
- Add queue priority info to CLI help (211)
- Add support for tasks containing only bash script or python module command
- Add support for skipping container apt installs using `CLEARML_AGENT_SKIP_CONTAINER_APT` env var in k8s

Bug Fixes
- Fix git fetch did not update new tags 209
- Fix file mode should be optional in configuration `files` section
- Fix `-m module $env` to support parsing `$env` before launching
- Fix setting tasks that were just marked as `aborted` to `started` - only force task to `started` after dequeuing it otherwise do nothing
- Fix slurm multi-node rank detection
- Fix pass `--docker only` (i.e. no default container image) when using `--dynamic-gpus` feature
- Fix logger object was used even if `None`
- Fix a race condition where in rare conditions popping a task from a queue that was aborted did not set it to started before the watchdog killed it (not applicable in k8s/slurm)
- Fix multi-node support to only send pip freeze update, only set task as started and only update task status on exit for RANK 0
- Fix do not cache venv cache if venv/python skip env var was set
- Fix use same state transition in k8s if supported by the server (instead of stopping the task before re-enqueue)
- Fix failed Task in services mode logged as _"User aborted"_ instead of failed, add Task state reason string
- Fix remove task from pending queue and set to failed in k8s when applying the pod template fails

Bug fixes

- Set GCS credentials to `None` if invalid service account credentials are provided (841, thanks freddessert!)
- Fix a sync issue when loading deferred configuration

New Features and Improvements

- Added `VCS_ENTRY_POINT` environment variable that overrides ClearML's entrypoint auto-detection

Bug Fixes

- Fix all parameters returned from a pipeline are considered strings
- Fix `Task.set_parameters()` does not add parameter type when parameter exists but does not have a type

New Features and Bug Fixes
- Add option to set daemon polling interval (197, thanks ilouzl!)
- Add Python 3.12 support
- Fix git pulling on cached invalid git entry. On error, re-clone the entire repository again (enable using `agent.vcs_cache.clone_on_pull_fail: true`)
- Fix conda env should not be cached if installing into base conda or conda existing env exists
- Fix cached repositories were not passing user/token when pulling
- Fix when disabling vcs cache do not add vcs mount point to container

New Features

- Add `CLEARML_AGENT_FORCE_POETRY` environment variable to allow forcing poetry even when using pip requirements manager
- Add `CLEARML_AGENT_FORCE_TASK_INIT` environment variable to allow runtime patching of script even if no repository is specified and the code is running a preinstalled docker
- Improve venv cache handling:
* Add `FileLock` readonly mode, default is write mode (i.e. exclusive lock, preserving behavior)
* Add venv cache now uses readonly lock when copying folders from venv cache into target folder. This enables multiple read, single write operation
* Do not lock the cache folder if we do not need to delete old entries
* Add `agent.venvs_cache.lock_timeout` to control the venv cache folder lock timeout (in seconds, default 30)
- Add protection for `truncate()` call
- Move configuration sanitization settings to the default config file
- Add queue ID report before pulling task
- Improve GPU monitoring for MIGs

Bug Fixes

- Use correct Python version in Poetry init (179, thanks nfzd!)
- Fix queue handling in `K8sIntegration` and `k8s_glue_example.py` (183, thanks FeU-aKlos!)
- Fix `FileNotFoundException` crash in `find_python_executable_for_version` (192, thanks ae-ae!)
- Fix delete temp console pipe log files after Task execution is completed (important for long lasting services agents to avoid collecting temp files on host machine)
- Fix `agent.enable_git_ask_pass` does not show in configuration dump
- Fix `pippip` is returned as a pip version if no value exists in `agent.package_manager.pip_version`
- Fix Python 3.12 support by removing `distutil` imports
- Fix `IOError` on file lock when using shared folder
- Fix torch resolver settings applied to `PytorchRequirement` instance are not used
- Fix comment lines (starting with ``) are not ignored in docker startup bash script
- Fix dynamic GPU sometimes misses the initial print