Clearml

Features

- Add support for MegEngine with examples (455)
- Add `TaskTypes` to main namespace (453)
- Add `LogUnifomParameterRange` for hyperparameter optimization with Optuna (462)
- Add joblib (equivalent to scikit) to `Task.init(auto_connect_frameworks)` argument
- Log environment variables starting with `*` in `environ_bind.py` (459)
- Pipeline
- Add eager decorated pipeline execution
- Support pipeline monitoring for scalers/models/artifacts
- Add `PipelineController.upload_model()`
- Add `PipelineController.add_step(configuration_overrides)` argument allowing to override Task configuration objects
- Change `PipelineController.start_locally()` default `run_pipeline_steps_locally=False`
- Add `PipelineController.stop(mark_failed, mark_aborted)` argguments
- Add `PipelineController.run_locally` decorator
- Add `PipelineController.continue_on_fail` property
- Add `PipelineController.__init__(abort_on_failure)` argument
- Add ClreamlJob state cache (refresh every second)
- Datasets
- Add `clearml-data` multi-chunk support
- Change `clearml-data` default chunk size to 512MB
- Change `Dataset.create()` now automatically reverts to using current Task if no project/name provided
- Add `Optimizer.get_top_experiments_id_metrics_pair()` for top performing experiments
- Add support for setting default value to auto connected argparse arguments
- Add `Task.get_script()` and `Task.set_script()` for getting and setting task's script prioerties for execution
- Add `Task.mark_completed()` `force` and `status_message` arguments
- Add `Task.stopped()` `reason` argument
- Add `Task.query_tasks()`, `Task.get_task()` and `Task.get_tasks()` `tags` argument

Bug Fixes

- Fix PyJWT resiliency support
- Fix xgb train overload (456)
- Fix `http://` throws `OSError` in Windows by using `pathlib2` instead of `os` (#463)
- Fix local diff should include staged commits, otherwise applying git diff fails (457)
- Fix `task.upload_artifact` non-standard dictionary will now revert to `pickle` (452)
- Fix `S3BucketConfig.is_valid()` for EC2 environments with `use_credentials_chain` (478)
- Fix audio classifier example when training with a custom dataset (484)
- Fix `clearml-task` diff was corrupted by Windows drive letter and separator (483)
- Fix TQDM "line cleanup" not using `CR` but rather arrow-up escape sequence (181)
- Fix `task.connect(dict)` value casting - if `None` is the default value, use backend stored type
- Fix Jupyter notebook should always set Task as completed/stopped, never failed (exceptions are caught in interactive session)
- Fix Pipeline support
- Fix `LocalClearmlJob` setting failed status
- Fix pipeline stopping all running steps
- Fix nested pipeline component parent point to pipeline Task
- Fix `PipelineController.start()` should not kill the process when done
- Fix pipeline failing to create Step Task should cause the pipeline to be marked failed
- Fix nested pipeline components missing pipeline tags
- Fix images reported over history size were not sent if frequency was too high
- Fix git detectors missing git repository without origin
- Fix support for upload `LazyEvalWrapper` artifacts
- Fix duplicate task dataset tags
- Fix FileLock create target folder
- Fix crash inside forked subprocess might leave SafeQueue in a locked state, causing `task.close()` to hang
- Fix PyTorch distributed example `TimeoutSocket` issue in Windows
- Fix broken `Dataset.finalize()`
- Fix Python 3.5 compatibility

Bug Fixes
- This release fixes the `six` conflict with the new `pathlib2` version 2.3.7 and up.

Features and Bug Fixes

- Add support for truncating task log file after reporting to server using `agent.truncate_task_output_files` configuration setting
- Fix PyJWT resiliency support
- Fix `--stop` checking default queue tag (80)
- Fix queue tag `default` does not exist and `--queue` not specified (try queue named `"default"`)
- Fix Python 3.5 compatibility
- Fix Python 2.7 support for PyTorch

Breaking Changes

- Disable default demo server (available by setting the `CLEARML_NO_DEFAULT_SERVER=0` environment variable)
- Change k8s glue default pod label to `CLEARML=agent` (instead of `TRAINS=agent`)

Features

- Add poetry cache into docker mapping 74
- Allow rewriting SSH URLs (see [here](https://github.com/allegroai/clearml-agent/commit/9456e493ac6d6495310ee084db906f9cdca8218c)), refers to #72, 42
- Add docker environment arguments log masking support, customizable using the `agent.hide_docker_command_env_vars` configuration value (see [here](https://github.com/allegroai/clearml-agent/blob/db57441c5dda43d8e38f01d7f52f047913e95ba5/docs/clearml.conf#L172)) 67
- Add support for naming docker containers using the `agent.docker_container_name_format` configuration option to set a name format (disabled by default) https://github.com/allegroai/clearml/issues/412
- k8s glue
- Remove queue name from pod name, add queue name and ID to pod labels 64
- Update task `status_message` for non-responsive or hanging pods
- Support the `agent.docker_force_pull` configuration option for scheduled pods
- Add docker example for running the k8s glue as a pod in a k8s cluster
- Add `agent.ignore_requested_python_version` configuration option to ignore any requested python version (default false, see [here](https://github.com/allegroai/clearml-agent/blob/db57441c5dda43d8e38f01d7f52f047913e95ba5/docs/clearml.conf#L45))
- Add `agent.docker_internal_mounts` configuration option to control containers internal mounts (non-root containers, see [here](https://github.com/allegroai/clearml-agent/blob/db57441c5dda43d8e38f01d7f52f047913e95ba5/docs/clearml.conf#L184))
- Add support for `-r requirements.txt` in the Installed Packages section
- Add support for `CLEARML_AGENT_INITIAL_CONNECT_RETRY_OVERRIDE` environment variable to override initial server connection behavior (defaults to true, allows boolean value or an explicit number specifying the number of connect retries)
- Add support for `CLEARML_AGENT_DISABLE_SSH_MOUNT` environment variable allowing to disable the auto `.ssh` mount into the docker
- Add support for `CLEARML_AGENT_SKIP_PIP_VENV_INSTALL` environment variable to skip Python virtual env installation on execute and allow providing a custom venv binary
- Add support for `CLEARML_AGENT_VENV_CACHE_PATH` environment variable to allow overriding venv cache folder configuration
- Add support for `CLEARML_AGENT_EXTRA_DOCKER_ARGS` environment variable to allow overriding extra docker args configuration
- Add support for environment variables containing bash-style string lists using shlex
- Add printout when using ClearML key/secret from environment variables
- Increase worker keep-alive timeout to 10 minutes instead of 1 minute
- Update documentation

Bug Fixes

- Fix auto mount `SSH_AUTH_SOCK` into docker 45
- Fix package manager configuration documentation 78
- Fix support for spaces in docker arguments https://github.com/allegroai/clearml/issues/358
- Fix standalone script with pre-exiting conda venv
- Fix PyYAML v5.4, v5.4.1 versions not supported
- Fix parsing VCS links starting with `git+git` (notice `git+git://` was already supported)
- Fix Python package with `git+git://` links or `git+ssh://` conversion
- Fix `--services-mode` if the execute agent fails when starting to run with error code 0
- Fix `--stop` with dynamic gpus
- Fix support for unicode standalone scripts, changing default `ascii` encoding to `UTF-8`
- Fix venv cache cannot reinstall package from git with http credentials
- Fix `PYTHONIOENCODING` environment variable is overwritten when already defined
- k8s glue
- Fix suppoer for multiple k8s glue instances with pod limits
- Fix task container handling fails parsing docker image
- Fix task container is not set when using default image/arguments
- Fix task container image arguments are used when no image is specified
- Fix task container arguments not supported in when template is not provided
- Fix `agent.extra_docker_bash_script` not applied correctly
- Fix task runtime properties are removed when re-enqueuing task
- Fix error is not thrown when failing to push task to queue

Features

- Add Click support and examples 386
- Add progress bar to SHA2 generation 396
- Add prefix to Task reported runtime info: `cpu_cores`, `gpu_driver_version` and `gpu_driver_cuda_version`
- Add support for `Logger.report_text()` explicit log-level reporting
- Add `return_full_path` argument to `StorageManager.list()`
- Support `Task.get_tasks()` passing multiple project names
- Add `TaskScheduler`
- Add `task_filter` argument to `Objective.get_top_tasks()`, allow `name` as a `task_filter` field
- Add `--output-uri` command-line option to `clearml-task`
- Add `requirements_file` argument to `Task.force_requirements_env_freeze()` to allow specifying a local requirements file
- Add support for `list` type argument in `Task.connect_configuration()` (previously only `dict` type was supported)
- Rename `TrainsTuner` to `ClearmlTuner`
- Update documentation links

Bug Fixes

- Fix Pandas with multi-index 399
- Fix check permissions fail in `HTTPDriver` 394
- Fix Dataset not setting system tag on existing `data_processing` Tasks
- Fix disable redundant resource monitoring in pipeline controller
- Fix `ClearMLJob` when both `project` and `target_project` are specified
- Fix `ClearMLJob` docker container info is not cached
- Fix no print logging after Python logging handlers are cleared
- Fix `PipelineController` callback returning `False`
- Fix machine specs when GPU is not supported
- Fix internal `logging.Logger` can't be pickled (only applicable to Python 3.6 or lower)
- Wait for reported events to flush to ensure `Task.flush()` with `wait_for_uploads=True` awaits background processes

Features

- Add Google Colab notebook tutorial 368 374
- Add support for GIF images in Tensorboard 372
- Add a tensorboardX example for `add_video` (creates GIFs in tensorboard) 372
- Add auto scaler customizable boot bash script
- Add `Task.ignore_requirements`
- Deprecate `Logger.tensorboard_single_series_per_graph()` as it is now controlled from the UI 🙂

Bug Fixes

- Fix `default_output_uri` for Dataset creation 371
- Fix `clearml-task` failing without a docker script 378
- Fix Pytorch DDP sub-process spawn multi-process
- Fix `Task.execute_remotely()` on created Task (not initialized Task)
- Fix auto scaler custom bash script should be called last before starting agent
- Fix auto scaler spins too many instances at once then kills the idle ones (spin time is longer than poll time)
- Fix multi-process spawn context using `ProcessFork` kills sub-process before parent process ends