Seml

Features
- Added support for zipped parameters (63)
- Zipped parameters behave as one dimension in the cartesian product of `grid`.
- Allow changing the directory where seml loads files to via `TMP_DIRECTORY` in `settings.py` (94)

Bug fixes
- Fix experiment filtering with no-hash and nested configs (89)
- For compatibility with [PyMongo >= 4.0](https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#pymongo4-migration-direct-connection), one can now specify `directConnection` (ca34fbf52b552cab4b7a6811d750f4f1c233175c)
- Better clean-up orphaned files (97)
- Allow float range steps (30b4afa9b741ed0f8e1cdfd72d63f3ee8fd8357f)
- MattermostOberserver bufixes (24910f9059f43e8e962e9e7a36adaa0a8beb4910, 059f055b71614715c2212a0cf212ff20fd64ab21)
- Respect hierarchy for `seed` parameter (93)

Features
- Support for running commands before and after the python script (84)
- Support for parameter overriding via CLI (79)
- Print Slurm error message when `sbatch` / `srun` fails (77).
- Detect when script name shadows a required package, e.g. `numpy.py` and raise an error accordingly (78).
- peak GPU memory usage for `TensorFlow >= 2.5`

Bug fixes
- Support for `pymongo >= 4` (76)
- Wait for slurm to properly cancel jobs when chaining events (87)
- Print proper hostnames for `seml jupyter` (86)

Breaking changes
- Remove the (inoperative) `max_jobs_per_batch` config option.

Features
- `reload-sources` command for updating the source code of specified experiments (57, 64)
- chain multiple commands in one, e.g. `seml coll cancel reset reload-sources start` (61)
- custom threshold settings for confirmation of cancel, reset, delete, and reload-sources (54)
- `-y` option to confirm changes directly (62)
- multi-user collaboration in the same MongoDB
- New `max_simultaneous_jobs` config option that limits the number of simultaneously running jobs per job array

Bug fixes
- Use `repr` and `shlex` for proper shell-escaping and handling of parameter edge cases (51, 53, 69)
- Correctly handle job arrays with a maximum number of simultaneously running jobs (47)
- Fix handling scontrol output without equal signs (49)
- Use UUIDs to fix race conditions in temp dir creation
- Fix crashing when `CONDA_DEFAULT_ENV` is not defined
- Do not discard parameters specifying empty dictionaries (56)
- print SEML usage when called without arguments (59)

- Require Python 3.7 (44)
- Fix filter preventing local workers from running previously started jobs
- Fix launch-worker
- Set PENDING only after submitting Slurm jobs to prevent faulty MongoDB states when killing the command

Bug fixes
- Fix `seml reset`
- Check for login node _before_ setting experiment status
- Allow `--debug-server` only for non-checkpointed code

Breaking changes
- remove backwards compatibility for SEML <0.2 Slurm jobs, and for starting/resetting SEML <0.2.4 MongoDB entries
- use `sbatch_options` of `slurm_default` for `seml jupyter`
- rename `JUPYTER_JOB` sbatch template to `JUPYTER`
- `seml print-command` is now separate from `seml start`

Features
- Improved JUPYTER settings handling and job/output file name
- SEML correctly considers all Slurm states

Bug fixes
- Fix false `seml jupyter` failures caused by rare Slurm states.
- Fix `seml jupyter` with multiple URLs (43)
- Fix print-command
- Fix cancelling jobs with multiple experiments running