Kslurm

Changes


* Update Neuroglia-helpers to latest version
* Fix issues in the work directory in kapp
* Improve handling of program files in kbatch

Changes

* Disable the hash check for the docker container download script until we can make a better solution

- Big improvement to `kjupyter`
- After loading up the jupyter server, the program gives an interactive shell within the job
- When within a `kjupyter` job, `kjupyter` subcommands become available:
- `kjupyter log`: print the jupyter logs
- `kjupyter console`: open up an interactive console attached to an existing active kernel (requires `jupyter-console` to be installed)
- `kjupyter url`: print the browser or server versions of the url
- `kjupyter tunnel`: print bash code to set up an ssh tunnel with the server

- Various bug fixes
- The memory syntax for requesting jobs has been improved
- units are no longer case-sensitive
- Allows requests in mebibytes (MiB) and gibibytes (GiB)
- Fix bug causing gradual degradation of `venv/bin/activate` files managed by `kpy`
- `kpy save` is now safe from data loss in case of process interruption
- Major under-the-hood updates to argument parser
- Improvements to help text and display

New Features
- `--venv` parameter available in `kbatch` and `krun`, allowing preloading of a venv. This is especially useful for scripts.
- `kpy export venv <name> --path <path>` allows the export of kpy venvs to any path.
- `kapp` is a new family of commands to download and manage singularity containers. Full details can be found in the docs
- Containers are pulled by name, but resolved to their content hash. This means that if you, for instance, pull both the `:latest` image and the specific version `:latest` points to, no the image won't be pulled twice. On the other hand, if you pull `:latest` a second time after the container is updated, the newly updated latest image will be pulled.
- Containers are downloaded and built safely, regardless of size. For large containers, a job is scheduled on an interactive node to build the container.
- Containers can be aliased, providing easy access to commonly used apps
- kapp has light wrappers around singularity shell, run, and exec, allowing for easy consumption of kapp managed containers. For complete flexibility, `kapp path <container_uri_or_alias>` returns the resolved path of the container
- All containers are automatically aliased in a snakemake readable format. Calling `kapp snakemake` returns a path you can pass to snakemake via the CLI param `--singularity-prefix`, allowing snakemake to seamlessly consume any kapp managed container.

Fixes and Improvements
- When kjupyter is called with a `--venv` provided, it attempts to install jupyter-lab if not already installed
- Escaped quotes in kbatch and krun commands are properly preserved
- kbatch properly runs sbatch scripts: `SBATCH directives` will be properly parsed
- The venv prompt name properly updates when calling `kpy load`. An asterisk (*) appears beside the venv name when modified (e.g. a new package is installed)
- Better error messages

Changes

- Print scheduled time for batch pvandyken (33)
- kapp pvandyken (35)

🐛 Bug Fixes

- Fix kbatch treatment of scripts and in-job krun pvandyken (34)
- Improve quoting of commands pvandyken (32)
- Jupyter autoinstall pvandyken (31)

Changes

kjupyter:
* Integrates with kpy, so you can directly fire up a jupyter notebook using a kpy saved venv
* When started, it outputs connection links for browser and/or VScode, as well as a copy/paste bash line to open a ssh tunnel, should you need one.

kpy:
* The api is changed so that `kpy list` lists all the saved venvs. `kpy rm` is added to delete a saved venv.
* `kpy activate`, when called without arguments, lists all the created and loaded venvs available to activate (formerly performed by `kpy list`
* `kpy create` will try to detect what version of python is currently activated and use that to initialize a new venv, assuming the user didn't specify a version (previously, it would always install using the python version used to install kslurm)

config:
* `account` can now be set interactively using `kpy config account -i`. This will start a program checking what accounts are available, along with their 'LevelFS' and displaying a selection menu.
* Separate accounts can be specified for cpu and gpu jobs.

Neuroglia-helpers:
* kslurm now packages drop-in support for neuroglia-helpers. Running `kslurm neuroglia-helpers >> $HOME/.bash_profile` will add a line sourcing the appropriate file into the `.bash_profile`, and will recreate nearly all the features enabled through a regular neuroglia-helper install. The profile is set through `kpy config neuroglia_profile <profile_name>`. Full documentation can be found [here](https://github.com/pvandyken/kslurm/blob/master/docs/neuroglia-helpers.md)
🚀 Features

- Interactive setting of account variable pvandyken (29)
- Neuroglia helpers pvandyken (28)
- Jupyter kpy integration pvandyken (27)
- Update kpy api pvandyken (25)
- Typer interface and automatic python version detection pvandyken (26)

🐛 Bug Fixes

- Kpy env switch pvandyken (24)

🧰 Maintenance

- Updated arg model pvandyken (23)
- Add automatic version bump, deployment, and logs pvandyken (22)

📝 Documentation

- More complete installation instructions pvandyken (30)

What's Changed
* Include job templates in distribution by pvandyken in https://github.com/pvandyken/kslurm/pull/21