Clearml

Bug Fixes

- Fix `CLEARML_AGENT_SKIP_PIP_VENV_INSTALL` fails to find python executable
- Fix `apt-get update` failure causes `apt-get install` not to be executed

New Features and Improvements

- Update S3 bucket verify option for minio 83 (thanks pshowbs!)
- Add environment variable for request method 91 (thanks mmiller-max!)
- Add additional k8s-glue dockerfiles 94 (thanks xadcoh!)
- Update default docker image to `nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04`
- Add support for custom docker image resolving using the `agent.default_docker.match_rules` configuration setting (see [here](https://github.com/allegroai/clearml-agent/blob/8712c5e636d9a02e939a9759348d29521a3939a9/docs/clearml.conf#L169))
- Add `agent.force_git_root_python_path` configuration setting to force adding the git repository root folder to the `PYTHONPATH` (if set working directory is not added to the `PYHTONPATH`)
- Add `build --force-docker` command line argument to the to allow ignoring task container data
- Add `agent.poetry_version` configuration setting to specify poetry version (and force installation of poetry if missing, see [here](https://github.com/allegroai/clearml-agent/blob/8712c5e636d9a02e939a9759348d29521a3939a9/docs/clearml.conf#L65))
- Add custom build script support
- Add extra configurations when starting daemon
- Add `agent.package_manager.force_original_requirements` configuration option, allowing to only use original requirements produced by local execution (note that using this configuration option prevents editing installed packages using the UI)
- Add support for the `CLEARML_AGENT_PROPAGATE_EXITCODE` environment variabe. Set this variable to `1` to allow ClearML Agent to return a nonzero exit code on failure
- Update `clearml-agent init` (use `app.clear.ml` as default server, add git token references)

Bug Fixes

- Fix virtualenv python interpreter used 98 (thanks idantene!)
- Fix typing package incorrectly required for Python>3.5 103 (thanks Honzys!)
- Fix symbolic links not copied from cached VCS into working copy (windows platform will result with default copy content instead of original symbolic link) 89
- Fix agent fails to check out code from main branch when branch/commit is not explicitly specified https://github.com/allegroai/clearml/issues/551
- Fix `git+git://` requirements
- Fix `default_python` calculation (and verbosity)
- Fix using deprecated `abc` support (Python 3.10 compatibility)
- Fix no default value for `CLEARML_API_DEFAULT_REQ_METHOD` causes `ValueError` if not specified
- Fix `agent.hide_docker_command_env_vars` mode to include URL passwords and handle environment vars containing docker commands
- Fix conda package manager listed packages with local links (` file://`) should ignore the local package if it does not exist
- Fix cuda patch version support in conda
- Fix agent attempts to check out code when in standalone mode
- Fix `FORCE_LOCAL_CLEARML_AGENT_WHEEL` environment variable handling when running from a Windows host
- Fix user-provided `"` is unnecessarily replaced to `\\"`
- Fix token is not propagated to docker in case credentials are not available
- Fix PyTorch aarch64 and windows support
- Fix VCS packages are reinstalled when the same commit version is already installed
- Fix git packages are installed even if commit is given and is preinstalled when using cached virtual environment

Stable Release

* Features
* GPU Performance improvements, 50%-300% improvement over vanilla Triton
* Performance improvements on CPU, optimize uvloop + multi-processing
* Huggingface Transformer example
* Binary input support, 37 , thanks Aleksandar1932

* Bug fixes
* stdout/stderr in inference service was not logged to dedicated Task

Features

- Add `Task.force_store_standalone_script()` to force storing standalone script instead of a Git repository reference (340)
- Add `Logger.set_default_debug_sample_history()` and `Logger.get_default_debug_sample_history()` to allow controlling maximum debug samples programmatically
- Add populate now stores function arg types as part of the hyper paremeters
- Add `status_message` argument to `Task.mark_stopped()`
- Change `HTTP` driver timeout and retry codes (connection timeout will now trigger a retry)

Bug Fixes

- Fix and upgrade the SlackMonitor (533)
- Fix network issues causing Task to stop on status change when no status change has occurred (535)
- Fix Pipeline controller function support for dict as input argument
- Fix uploading the same metric/variant from multiple processes in threading mode should create a unique file per process (since global counter is not passed between the subprocesses)
- Fix resource monitoring should only run in the main process when using threaded logging mode
- Fix fork patching so that the signal handler (`at_exit`) will be called on time
- Fix fork (process pool) hangs or drops reports when reports are at the end of the forked function in both threaded and subprocess mode reporting
- Fix Multi-pipeline support
- Fix delete artifacts after upload
- Fix artifact preview has no truth value
- Fix storage cache cleanup does not remove all entries on a silent fail
- Fix always store session cache in `~/.clearml` (regardless of the cache folder)
- Fix `StorageManager.download_folder()` fails on Windows path

Features
- Add support for `jsonargpraser` (403)
- Add `HyperParameterOptimizer.get_top_experiments_details()` returns the hparams and metrics of the top performing experiments of an HPO (473)
- Allow overriding initial iteration offset using environment variable (`CLEARML_SET_ITERATION_OFFSET`) or `Task.init(continue_last_task==<offset>)` (496)
- Add better input handling for `clearml-init` in colab (515)
- Add environment variable for default request method (521)
- Add `LocalClearmlJob` as possible option for HPO (525)
- Add convenience functionality to `clearml-data` (526)
- Add support for `vscode-jupyter` (https://github.com/microsoft/vscode-jupyter/pull/8531)
- Improve detection of running reporting subprocess (including zombie state)
- Support controlling S3/Google Cloud Storage `_stream_download_pool_connections` using the `stream_connections` configuration setting in `clearml.conf` (default 128)
- Add warning when loosing reporting subprocess
- Add `Model.remove()` to allow removing a model from the model repository
- Add HTTP download timeout control (change default connection timeout to 30 seconds)
- Add initial setup callback to monitoring class
- Add `Task.get_reported_plots()`
- Allow `Monitor.get_query_parameters` to override defaults
- Add support for Google Cloud Storage `pool_connections` and `pool_maxsize` overrides
- Add last worker time to `AutoScaler`
- Add warning when opening an aborted Dataset
- Store multi-pipeline execution plots on the master pipeline Task
- Support pipeline return value stored on pipeline Task
- Add `PipelineDecorator.multi_instance_support`
- Add `PipelineDecorator` to `clearml` and `clearml.automation` namespaces
- Documentation and examples
* Update docstrings (501)
* Add Markdown in pipeline jupyter notebooks (502)
* Update pipeline example (494)
* Add abseil example (509)
* Change README to dark theme (513)
* Update XGBoost example (524)
* Change example name (528)

Bug Fixes
- Fix `TriggerScheduler` on Dataset change (491)
- Fix links in Jupyter Notebooks (505)
- Fix `pandas` delta datetime conversion (510)
- Fix `matplotlib` auto-magic detect bar graph series name (518)
- Fix path limitation on storage services (posix, object storage) when storing target artifacts by limiting length of project name (full path) and task name used for object path (516)
- Fix multi-processing context block catching exception
- Fix Google Cloud Storage with no default project causes a crash
- Fix main process's reporting subprocess lost, switch back to thread mode
- Fix forked `StorageHelper` should use its own `ThreadExecuter`
- Fix local `StorageHelper.delete()` raising exception on non existing file instead of returning false
- Fix `StorageHelper` rename partial file throwing errors on multiple access
- Fix resource monitor fails on permission issues (skip over parts)
- Fix reusing Task does not reset it
- Fix support `clearml` PyCharm Plugin 1.0.2 (support partial pycharm git repo sync)
- Fix `Task.reset()` `force` argument ineffective
- Fix PY3.5 compatibility
- Fix validation error causes infinite loop
- Fix tasks schema prevents sending `null` container parts
- Fix missing `CLEARML_SET_ITERATION_OFFSET` definition
- Fix `Model.get_weights_package()` returns `None` on error
- Fix download progress bar based on `sdk.storage.log.report_download_chunk_size_mb` configuration
- Fix Conda lists the `CudaToolkit` version installed (for the agent to reproduce)
- Fix Jupyter kernel shutdown causing nested atexit callbacks leaving Task in running state
- Fix multi-subprocess can cause Task to hand at close
- Fix TF 2.7 support (get `logdir` on with multiple TB writers)

Bug Fixes

- Fix duplicate keyword argument (affects `clearml-data` `Dataset.get()`) https://github.com/allegroai/clearml/issues/490, [ClearML Slack Channel \#1](https://clearml.slack.com/archives/CTK20V944/p1635411658177900), [\#2](https://clearml.slack.com/archives/CTK20V944/p1635925696270900), [\#3](https://clearml.slack.com/archives/CTK20V944/p1635323823155700), [\#4](https://clearml.slack.com/archives/CTK20V944/p1635921934264300)
- Fix session raises missing host error when in offline mode 489
- Fix `Task.get_task()` does not load `output_uri` from stored Task
- Fix `Task.get_models()['input']` returns string instead of `clearml.Model`
- Fix `tf.saved_model.load()` binding for TensorFlow >=2.0
- Fix hyperparams with `None` value converted to empty string causes inferred type to change to `str` in consecutive `Task.connect()` calls