Clearml

Features

- Add continuing of previously executed experiments. Add `Task.init()` argument `continue_last_task` to continue a previously used Task 160
- Allow Task editing/creation from code. `Task.export_task/import_task/update_task()` 128
- Add offline mode. Use `Task.set_offline()` and `Task.import_offline_session()`
- Support setting offline mode via `TRAINS_OFFLINE_MODE=1` environment variable
- Support setting offline API version via `TRAINS_OFFLINE_MODE=2.9` environment variable
- Automatically pickle all objects when uploading as artifacts, `task.upload_artifact()` argument `auto_pickle=True` 153
- Add multiple sections/groups support for Task hyper-parameters using `Task.connect()`
- Add multiple configurations (files) using `Task.connect_configuration`
- Allow enabling OS environment logging using the `sdk.development.log_os_environments` configuration parameter (complements the `TRAINS_LOG_ENVIRONMENT` environment variable)
- Add Optuna support for hyper-parameter optimization controller. `OptimizerOptuna` is now the default optimizer.
- Add initial Keras-Tuner support (keras-team/keras-tuner334)
- Add automatic FastAI logging. It is disabled if tensorboard is loaded (assuming TensorBoradLogger will be used)
- Support Tensorboard text logging (`add_text()`) as debug samples (`.txt` files), instead of as console output
- Allow for more standard confusion matrix reporting. `Logger.report_confusion_matrix()` argument `yaxis_reversed` (flips the confusion matrix if `True`, default `False`) 165
- Add support for Trains Server 0.16.0 (API v2.9 support)
- Allow disabling Trains update message from the log using the `TRAINS_SUPPRESS_UPDATE_MESSAGE` environment variable 157
- Add AWS EC2 Auto-Scaler service wizard and Service
- Improved and updated examples
- Add Keras Tuner CIFAR10 example
- Add FastAI example
- Update PyTorch Jupyter notebook examples 150
- Support global requirements detection using `pip freeze` (set `sdk.development.detect_with_pip_freeze` configuration in `trains.conf`)
- Add `Task.get_projects()` to get all projects in the system, sorted by last update time

Bug Fixes

- Fix UTC to time stamp in comment 152
- Fix and enhance GPU monitoring
- Fix GPU stats on Windows machines 177
- More robust GPU monitoring 170
- Fix filename too long bug https://github.com/allegroai/trains-server/issues/49
- Fix TensorFlow image logging to allow images with no width/height/color metadata 182
- Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush `stdout`
- Fix `plotly` support for `matplotlib` 3.3
- Add Python 2.7 support for `get_current_thread_id()`
- Update examples requirements
- Fix and improve signal handling
- Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms
- Fix auto logging multiple argparse calls before `Task.init()`
- Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named `auxiliary_git_dif`
- Fix requirements detection
- Fix Trains installed from `git+`
- Fix when Trains is not directly imported
- Fix multiple `-e` packages were not detected (only the first one)
- Fix running with Trains in `PYTHONPATH` resulted in double entry of trains
- Fix `Task.set_base_docker()` on main task to do nothing when running remotely

Features

- Add direct plotly figure reporting 136
- Improve matplotlib integration 140
- Add `Task.set_parent()` 144
- Add `Task.wait_for_status()` 145
- Enhance framework `save`/`load` binding callbacks https://github.com/pytorch/ignite/issues/1056
- Add `TrainsJob.is_failed()`
- Add APIClient support using `trains.backend_api.session.APIClient`
- Add URL tracking when using `StorageManager.get_local_copy()` (log remote url instead of local file copy)
- Improve & expand examples

Bug Fixes

- Fix `pytorch_tensorboard.py` example 148
- Fix `ssh://userdomainserver.com/` in Git repository link 149
- Fix do not call `make_deterministic()` on import
- Fix initialization wizard (allow at most two verification retries, then print error) [trains-agent 16](https://github.com/allegroai/trains-agent/issues/16)
- Fix Google CoLab code/package detection
- Improve Pytorch-Ignite integration
- Fix TensorFlow >=2 histogram binding
- Fix `Logger.tensorboard_single_series_per_graph`
- Fix anonymous named models
- Fix incorrect entry point detection when called from Trains wrapper (e.g. `TrainsLogger` in Pytorch Ignite/Lightning)

Features

- Add automation support including hyper-parameters optimization (see example [here](https://github.com/allegroai/trains/blob/master/examples/automation/hyper_parameter_optimizer.py))
- `Task.init()` `auto_connect_arg_parser` argument can accept a dictionary disabling specific keys from the argparser (Trains Slack channel [thread](https://allegroai-trains.slack.com/archives/CTK20V944/p1589355878206700))
- Allow `worker_id` override using `TRAINS_WORKER_NAME` environment variable (Trains Slack channel [thread](https://allegroai-trains.slack.com/archives/CTK20V944/p1589920581251600?thread_ts=1589920220.250100&cid=CTK20V944))
- Support layout configuration for plotly objects using `extra_layout` argument in all `Logger` reporting methods 136
- Add `Task.execute_remotely()` to allow cloning and enqueuing a locally executed task (or stopping and re-enqueuing a remotely executed task) 128
- Add Parquet framework and model type
- Support recursive model folder packaging
- Add `Task.get_reported_console_output()` and `Task.get_reported_scalars()` to allow retrieval of reported reported output and scalar metrics
- Add `Task.add_requirements()` to force requirement package into "installed packages"
- Improve task reuse responsiveness
- Add `raise_on_error` (default `False`) argument to `Model.get_local_copy()` and `Artifact.get_local_copy()` https://github.com/allegroai/trains-agent/issues/17
- Support `Task.get_task()` without project name (i.e. all projects)
- Support using the file calling `Task.init()` as the task's script in case `sys.argv` doesn't point to a git repository
- Support detecting and remotely executing code running from a module (i.e. `-m module`)
- Add callback for framework `save`/`load` binding for better integration with [pytorch/ignite ](https://github.com/pytorch/ignite) https://github.com/pytorch/ignite/issues/1056
- Support new task types provided in [Trains Server v0.15.0](https://github.com/allegroai/trains-server/releases/tag/0.15.0)
- Add automation and distributed examples
- Upgrade default pip version to `<20.2`

Bug Fixes

- Fix `exact_match_regex()` in case of empty pattern 138
- Address deprecation warning and newer `attrs` versions in `MetricsEventAdapter` 134
- Fix issues with plotly support (Trains Slack channel [thread](https://allegroai-trains.slack.com/archives/CTK20V944/p1589871253243600) and [thread](https://allegroai-trains.slack.com/archives/CTK20V944/p1589279340195000))
- Fix default argument behavior to match argparse behavior
- Fix `OutputModel` with `task=None` should use current task, if exists
- Fix `Task.get_task()` to raise proper error on incorrect `task_id`
- Fix `Task.enqueue()` to use an exact queue name match
- Fix `NaN`, `Inf` and `-Inf` values display in reported table (not supported by JSON)
- Limit max requirement size to 0.5MB
- Fix issues with repository analysis
- Fix `StorageManager` should only try to extract .zip files, `Model` should not auto extract package https://github.com/allegroai/trains-agent/issues/17

Features and Bug Fixes

- Add Logger histogram mode (stack/group/relative)
- Add PEP610 support for new pip>=20.1 git reference feature (see https://github.com/pypa/pip/issues/609) #62
- Better support for PyTorch Ignite integration
- Support `` and `?` in uploaded image file names
- Support `ArgumentParser.parse_arg()` used before initializing task when `auto_connect_arg_parser=False`
- Fix upload_object_via_stream() in Azure storage driver
- Fix `Task.get_tasks()` return type

Features

- Add media (audio) support for both Logger and Tensorboard binding using `Logger.report_media()` 120
- Add autokeras example (see [here](https://github.com/allegroai/trains/tree/master/examples/autokeras)) #125
- Add caching of extracted zip artifacts
- Add `Task.get_tasks()` filtering support
- Add `trains.storage.StorageManager`
- Add nicer stdout log flush
- Add text logging to TF V1 example
- Improve SSL retries and error handling
- Improve shutdown behavior in case of a user abort (Ctrl-C)
- Improve repository and module detection
- Improve resource monitor
* Report memory usage for process (and sub-processes)
* Report GPU memory usage for process (and sub-processes)
* Stability
- Improve Jupyter support
* Make sure `trains` is included in Jupyter requirements
* Ignore IPython directives in converted Python script (like `%` and `!` lines)
- Update Pytorch/TensorboardX examples

Bug Fixes

- Fix renaming/deleting model file right after saving will break asynchronous upload 123
- Fix shutdown behavior:
* Logger flush delay and termination
* Repository/package detection termination
* Threads termination
- Fix text encoding `utf-8` and `pr_curve` broken in Tensorboard support
- Fix single log request exceeds max packet size
- Fix upload debug assets with series / title including `,` or `/` did not add iteration index to the filename
- Fix `scikit-learn` module detection
- Use UID in message if user name cannot be retrieved (from the [Trains Slack channel](https://github.com/allegroai/trains#documentation-community--support))
- Fix cast in Tensorflow v2 example

Features and Bug Fixes

- Add OpenMPI/Slurm support, support `TRAINS_FORCE_MASTER_NODE` environment variable to allow forcefully specifying a node is master (not based on the MPI Rank)
- Add `Task.get_models()` and `Task.models` for retrieving stored models for previously executed tasks
- Support multiple `EventWriter` in TensorFlow eager mode (TF 2.0+) 26
- Support reusing models
- Fix TensorFlow `NaN`/`Inf` values support 118
- Fix `joblib` binding
- Fix metrics/files/models upload on shutdown
- Fix support for `Task` init/close multiple times
- Update [manual_reporting.py](https://github.com/allegroai/trains/blob/master/examples/manual_reporting.py) example with `Logger.report_table()` example
- Deprecate `Task.set_model_config()`, `Task.get_model_config_text()` and `Task.get_model_config_dict()`