Mle-toolbox

- Adds tfcuda behavior deterministic flag

Added

- Adds PBT, Successive Halving & Hyperband experiment support
- Adds report support for non-search experiments.
- Adds robust `local -> remote` experiment launching.

Changed

- Restructures experiment wrapper `launch_experiment`
- Moves PBT experiment utilities to `mle-hyperopt`.
- Fix versions of subpackages so that dependencies are static.
- Rename `get_jax_os_ready` to `get_os_env_ready` and include `device_config` in auto-setup of `MLExperiment`.

Fixed

- Updates `mle init` to work with vim editor.
- Fixes all broken links in Readme.

- Small fixes before Sprekeler lab group presentation talk.
- Restructuring imports and making it possible to launch `mle run` from notebook.

Added

- Introduces experimental notebook-friendly `MLELauncher` which allows you to schedule experiments from within a local notebook.
- Adds `mle protocol` subcommand to get a quick view of the last experiments and their status.
- Adds `mle project` to initialize a new project based on cloning the [`mle-project`](https://github.com/mle-infrastructure/mle-project) repository.

Changed

- Refactors out resource monitoring and protocol database to `mle-monitor` sub-package.
- Refactors out job launching and status monitoring to `mle-launcher` sub-package.
- Moves population-based training and hypothesis testing into `experimental` submodule.
- Moves documentation page to `mle-docs` sub-repository.

Added

- 3D animation post-processing helpers (`animate_3D_scatter` and `animate_3D_surface`) and test coverage for visualizations (static/dynamic).
- [`nevergrad`](https://facebookresearch.github.io/nevergrad/index.html) multi-objective hyperparameter optimization. Checkout the toy [example](examples/toy_multiobj).
- Adds `experiment` decorator for easy integration:

python
from mle_toolbox import experiment

experiment("configs/abc.json", model_config={"num_layers": 2})
def run(mle, a):
print(mle.model_config)
print(mle.log)
print(a)

if __name__ == "__main__":
run(a=2)


- Adds `combine_experiments` which loads different `meta_log` and `hyper_log` objects and makes them "dot"-accessible:
python
experiment_dirs = ["../tests/unit/fixtures/experiment_1",
"../tests/unit/fixtures/experiment_2"]
meta, hyper = combine_experiments(experiment_dirs, aggregate_seeds=False)


- Adds option to run grid search for multiple base configurations without having to create individual experiment configuration files.

Changed

- Configuration loading is now more toolbox specific. `load_json_config` and `load_yaml_config` are now part of `mle-logging`. The toolbox now has two "new" function `load_job_config` and `load_experiment_config`, which prepare the raw configs for future usage.
- The `job_config` file now no longer has to be a `.json` file, but can (and probably should) be a `.yaml` file. This makes formatting easier. The hyperoptimization pipeline will generate configuration files that are of the same file type.
- Moves core hyperparameter optimization functionality to [`mle-hyperopt`](https://github.com/RobertTLange/mle-hyperopt). At this point the toolbox wraps around the search strategies and handles the `mle-logging` log loading/data retrieval.
- Reduces test suite since all hyperopt strategy-internal tests are taken care of in `mle-hyperopt`.

Fixed

- Fixed unique file naming of zip files stored in GCS bucket. Now based on the time string.
- Grid engine monitoring now also tracks waiting/pending jobs.
- Fixes a bug in the random seed setting for synchronous batch jobs. Previously a new set of seeds was sampled for each batch. This lead to problems when aggregating different logs by their seed id. Now the first set of seeds is stored and provided as an input to all subsequent `JobQueue` startups.

Added
- Adds general processing job, which generalizes the post-processing job and enables 'shared'/centralized data pre-processing before a (search) experiment and results post-processing/figure generation afterwards. Checkout the [MNIST example](https://github.com/RobertTLange/mle-toolbox/blob/main/examples/torch_mnist/mnist_single.yaml).
- Adds population-based training experiment type (still experimental). Checkout the [MNIST example](https://github.com/RobertTLange/mle-toolbox/tree/main/examples/pbt_mnist) and the [simple quadratic from the paper](https://github.com/RobertTLange/mle-toolbox/tree/main/examples/pbt_quadratic).
- Adds a set of unit/integration tests for more robustness and `flake8` linting.
- Adds code coverage with secrets token.
- Adds `mle.ready_to_log` based on `log_every_k_updates` in `log_config`. No more modulo confusion.
- Adds [slack clusterbot integration](https://github.com/sprekelerlab/slack-clusterbot/) which allows for notifications and report upload.

Changed
- Allow logging of array data in meta log `.hdf5` file by making `tolerant_mean` work for matrices.
- Change configuration .yaml to use `experiment_type` instead of `job_type` for clear distinction:
- *job*: Single submission process on resource (e.g. single seed for single configuration)
- *eval*: Single parameter configuration which can be executed/trained for multiple seeds (individual jobs!)
- *experiment*: Refers to entire sequence of jobs to be executed (e.g. grid search with pre/post-processing)
- Restructure `experiment` subdirectory into `job` for consistent naming.
- Refactor out `MLELogger` into separate `mle-logging` package. It is a core ingredient that should stand alone.

Fixed

- Fixed `mle retrieve` to be actually useful and work robustly.
- Fixed `mle report` to retrieve results if they don't exist (or to use a local directory provided by the user).
- Fixed `mle report` to generate reports via `.html` file and the dependency `xhtml2pdf`.
- Fixed unique hash for experiment results storage. Previously this only used the content of `base_config.json`, which did not result in a unique hash when running different searches via `job_config.yaml`. Now the hash is generated based on a merged dictionary of the time string, `base_config` and `job_config`