Viash

The first (major) release this year! The biggest changes are:

* Nextflow VDSL3 is now the default Nextflow platform, whereas the legacy Nextflow platform has been deprecated.
* Support for tracking memory and cpu requirements more elegantly
* Grouping arguments in groups more concisely.
* The addition of a `viash ns exec` command, to be able to execute commands on Viash components more easily.

BREAKING CHANGES

* `NextflowPlatform`: `variant: vdsl3` is now the default NextflowPlatform. `variant: legacy` has been deprecated.

* `Functionality`: Fields `.inputs` and `.outputs` has been deprecated. Please use `.argument_groups` instead (186).
Before:
yaml
functionality:
inputs:
- name: "--foo"
outputs:
- name: "--bar"

Now:
yaml
functionality:
argument_groups:
- name: Inputs
arguments:
- name: "--foo"
type: file
- name: Outputs
arguments:
- name: "--bar"
type: file
direction: output


* Passing strings to an argument group's arguments has been deprecated. Please simply copy the argument itself into the argument group (186).
Before:
yaml
functionality:
arguments:
- name: "--foo"
type: file
- name: "--bar"
type: file
direction: output
argument_groups:
- name: Inputs
arguments: [ foo ]
- name: Outputs
arguments: [ bar ]

Now:
yaml
functionality:
argument_groups:
- name: Inputs
arguments:
- name: "--foo"
type: file
- name: Outputs
arguments:
- name: "--bar"
type: file
direction: output


NEW FUNCTIONALITY

* Allow setting the number of processes and memory limit from within the Viash config,
as well as a list of required commands. Example:

yaml
functionality:
name: foo
requirements:
n_proc: 10
memory: 10G
commands: [ bash, r, perl ]


You can override the default requirements at runtime:

- `./foo ---n_proc 4 ---memory 100pb` (for NativePlatform or DockerPlatform)
- By adding `process.cpus = 4` and `process.memory "100 PB"` to a nextflow.config (for NextflowPlatform)

This results the following meta variables to be injected into a script:

- `meta_n_proc` (in Bash) or `meta["n_proc"]` (in any other language): Number of processes the script is allowed to spawn.
- `meta_memory_b` (in Bash) or `meta["memory_b"]` (in any other language): Amount of memory the script is allowed to allocate, in bytes.
- `meta_memory_kb` (in Bash) or `meta["memory_kb"]` (in any other language): Same but in kilobytes, rounded up.
- `meta_memory_mb` (in Bash) or `meta["memory_mb"]` (in any other language): Same but in megabytes, rounded up.
- `meta_memory_gb` (in Bash) or `meta["memory_gb"]` (in any other language): Same but in gigabytes, rounded up.
- `meta_memory_tb` (in Bash) or `meta["memory_tb"]` (in any other language): Same but in terabytes, rounded up.
- `meta_memory_pb` (in Bash) or `meta["memory_pb"]` (in any other language): Same but in petabytes, rounded up.

* `viash ns exec`: Added a command for executing arbitrary commands for all found Viash components.
The syntax of this command is inspired by `find . -exec echo {} \;`.

The following fields are automatically replaced:
* `{}` | `{path}`: path to the config file
* `{abs-path}`: absolute path to the config file
* `{dir}`: path to the parent directory of the config file
* `{abs-dir}`: absolute path to the directory of the config file
* `{main-script}`: path to the main script (if any)
* `{abs-main-script}`: absolute path to the main script (if any)
* `{functionality-name}`: name of the component

A command suffixed by `\;` (or nothing) will execute one command for each
of the Viash components, whereas a command suffixed by `+` will execute one
command for all Viash components.

* `ConfigMod`: Added a `del(...)` config mod to be able to delete a value from the yaml. Example: `del(.functionality.version)`.

MAJOR CHANGES

* `Folder structure`: Adjusted the folder structure to correctly reflect the the namespace change of viash from `com.dataintuitive.viash` to `io.viash`.

* `Functionality`: Reworked the `enabled` field from boolean to a `status` field which can have the following statusses: `enabled`, `disabled` and `deprecated`.
When parsing a config file which has the `status` field set to `deprecated` a warning message is displayed on stderr.
Backwards for `enabled` is provided where `enabled: true` => `status: enabled` and `enabled: false` => `status: false`. The `enabled` field is marked deprecated.

MINOR CHANGES

* `Resources`: Handle edge case when no resources are specified in the `vsh.yaml` config file and display a warning message.

* `BashWrapper`: Add a warning when an argument containing flags (e.g. `--foo`) is not recognized and will be handled as a positional argument as this is likely a mistake.

* `Functionality`: Add check to verify there are no double argument names or short names in the config `vsh.yaml` declarations.

* `BashWrapper`: Add check to verify a parameter isn't declared twice on the CLI, except in the case `multiple: true` is declared as then it's a valid use case.

* `BashWrapper`: For int min/max checking: use native bash functionality so there is no dependency to `bc`.
For double min/max checking: add fallback code to use `awk` in case `bc` is not present on the system (most likely to happen when running tests in a docker container).

* `viash ns list/viash config view`: Allow viewing the post-processed argument groups by passing the `--parse_argument_groups` parameter.

TESTING

* `ConfigMod`: Added unit tests for condition config mods.

* `MainTestDockerSuite`: Derive config alternatives from the base `vsh.yaml` instead of adding the changes in separate files.
This both reduces file clutter and prevents having to change several files when there are updates in the config format.

* `GitTest`: Added unit tests for Git helper (216).

BUG FIXES

* `csharp_script`, `javascript_script`, `python_script`, `r_script`, `scala_script`: Make meta fields for `memory` and `n_proc` optional.

* `NextflowVdsl3Platform`: Don't generate an error when `--publish_dir` is not defined and `-profile no_publish` is used.

* `Viash run`: Viash now properly returns the exit code from the executed script.

* `Git`: Fix incorrect metadata when git repository is empty (216).

BREAKING CHANGES

* `WorkflowHelper::helpMessage`: Now only takes one argument, namely the config.

MAJOR CHANGES

* `Namespace`: Changed the namespace of viash from `com.dataintuitive.viash` to `io.viash`.

MINOR CHANGES

* `Testbenches`: Add a testbench framework to test lots of character sequences, single or repeating to be tested in the yaml config. This can be used to later extend to other tests.

* `Testbenches::vdsl3`: Add testbenches to verify functionality:
- Vdsl3's `param_list` (`yamlblob`, `yaml`, `json`, `csv`).
- NextFlow's own `params-file`.
- Vdsl3's recalculating resource file paths to be relative to the `param_list` file instead of the workflow file (only available for `yaml`, `json`, `csv`).
- Vdsl3's wrapping of modules to run these as a separate workflow automagically out of the box.

* `Main`: Added `viash --schema_export` which outputs a schema of the Viash config file
to console. This is to be used to automate populating the documentation website.

* `Helper`: Split help message by argument group.

* `Helper`: Remove unneeded arguments.

* `Functionality`: Add default groups `Inputs`, `Outputs` and `Arguments` for all arguments missing from user-defined `argument_groups`.

* `WorkflowHelper::helpMessage`: Rewrite to bring on par with Viash's help message.

* `BooleanArguments`: Renamed internal class names for BooleanArguments to be better in line with how they are named in the config yaml.
`BooleanArgumentRegular` -> `BooleanArgument` (in line with `boolean`)
`BooleanArgumentTrue` -> `BooleanTrueArgument` (in line with `boolean_true`)
`BooleanArgumentFalse` -> `BooleanFalseArgument` (in line with `boolean_false`)

BUG FIXES

* `NextflowVdsl3Platform`: Change how `--id` is processed when a VDSL3 module is called from the CLI.

* `NextflowVdsl3Platform`: Fix error when param_list is `null`.

* `NextflowVdsl3Platform`: Fix error when optional, multiple arguments are set to `null`.

* `Testbenches`: Better capture expected error messages while running testbenches again. Code changes right before previous release re-introduced some of the messages.

* `NextflowVdsl3Platform`: Fix issue where optional parameters aren't removed when `.run(args: [optarg: null])`.

* `WorkflowHelper::readCsv`: Treat empty values as undefined instead of throwing an error.

* `NextflowVdsl3Platform`: Use `$NXF_TEMP` or `$VIASH_TEMP` as temporary directory if the container engine is not set to `docker`, `podman` or `charlieengine`, else set to `/tmp`.

* `Resources`: When adding a resource folder, allow a trailing `/` at the end of the path.
Previously this caused the target folder to be erased and the content of the resource folder to be written directly into the target folder.

NEW FUNCTIONALITY

* `Functionality`: Allow specifying argument groups. Example:
yaml
functionality:
...
argument_groups:
- name: First group
arguments: [foo, bar]
description: Description



* Addition of the `viash_nxf_schema` component for converting a Viash config (for a workflow) into a nextflow schema file.

* `NextflowVdsl3Platform`: Use `--param_list` to initialise a Nextflow channel with multiple parameter sets.
Possible formats are csv, json, yaml, or simply a yaml_blob.
A csv should have column names which correspond to the different arguments of this pipeline.
A json or a yaml file should be a list of maps, each of which has keys corresponding to the arguments of the pipeline.
A yaml blob can also be passed directly as a parameter.
Inside the Nextflow pipeline code, params.params_list can also be used to directly a list of parameter sets.
When passing a csv, json or yaml, relative path names are relativized to the location of the parameter file.

Examples:
sh
nextflow run "target/foo/bar/main.nf" --param_list '[{"id": "foo", "input": "/path/to/bar"}]'
nextflow run "target/foo/bar/main.nf" --param_list "params.csv" --reference "/path/to/ref"


MAJOR CHANGES

* `NextflowVdsl3Platform`: The functionality is now slurped from a json instead of manually
taking care of the formatting in Groovy.

* `NextflowVdsl3Platform`: The `--help` is auto-generated from the config.


MINOR CHANGES

* `NextflowVdsl3Platform`: Allow both `--publish_dir` and `--publishDir` when `auto.publish = true`.

* `NextflowVdsl3Platform`: Allow passing parameters with multiplicity > 1 from Nextflow CLI.

* `Main`: Added `viash --cli_export` which outputs the internal cli construction information
to console. This is to be used to automate populating the documentation website.

* `viash ns`: Display success and failure summary statistics, printed to stderr.

* `DataObject`: `.alternatives` is now a `OneOrMore[String]` instead of `List[String]`, meaning
you can now specify `{ type: string, name: "--foo", alternatives: "-f" }` instead of
`{ type: string, name: "--foo", alternatives: [ "-f" ] }`

* `BashWrapper`: Added metadata field `meta_executable`, which is a shorthand notation for
`meta_executable="$meta_resources_dir/$meta_functionality_name"`

INTERNAL CHANGES

* `Arguments`: Internal naming of functionality.arguments is changed from DataObject to Arguments. Change is also applied to child classes, e.g. StringObject -> StringArgument.

* `Script`: Allow more control over where injected code ends up.

* Restructure type system to allow type-specific arguments.

BUG FIXES

* `DockerPlatform`: Change `org.opencontainers.image.version` annotation to `functionality.version` when set.
Additionally fixed retrieving the git tag possibly returning `fatal: No names found, cannot describe anything.` or similar.

* `viash config inject`: Fix config inject when `.functionality.inputs` or `.functionality.outputs` is used.

* `BashWrapper`: Don't add `bc` as dependency. Only perform integer/float min/max checks when bc is available, otherwise ignore.

* `DockerPlatform`: Fix inputs & outputs arguments being present twice.

* `viash ns test`: Silently skip Nextflow platforms as these don't support tests and will always fail.

* `Testbenches`: Better capture expected error messages while running testbenches. Having these show on the console could be confusing.

* `NextflowVdsl3Platform`: Fix issue when running multiple VDSL3 modules concurrently on the same channel.

NEW FUNCTIONALITY

* `NextflowVdsl3Platform`: Allow overriding the container registry of all Viash components by
setting the `params.override_container_registry` value. Only works for auto-derived image names.

MAJOR CHANGES

* `Functionality`: renamed `tests` to `test_resources`.
Backwards compatibility provided but a notification message is displayed on the console.

MINOR CHANGES

* `Functionality` and `viash ns`: Added `.enabled` in functionality, set to `true` by default.
Filter for disabled components in namespace commands.

* `DockerPlatform`: Add org.opencontainers.image annotations to built docker images.

* `Functionality`: when defining text resources, permit defining `path` instead of `dest`.
If both `dest` and `path` are unset, use a default file name depending on the resource type, such as `script.sh` or `text.txt`.

* `viash build`: Errors are printed in red.

BUG FIXES

* `NextflowVdsl3Platform`: Undefined input files should not inject a `VIASH_PAR_*` variable when `multiple: true`.

* `NextflowVdsl3Platform`: Make injected resources dir absolute.

* `NextflowVdsl3Platform`: Fix escaping of triple single quotes.

* `NextflowVdsl3Platform`: Also apply auto.simplifyInput to Lists.

* `DockerPlatform`: added a `test_setup` that allows adding apt/apk/... setup requirements.
These are only executed when running tests.

MINOR CHANGES

* `--help`: Don't print "my_component <not versioned>" when no version is specified,
but instead simply "my_component".

* `NextflowVdsl3Platform`: Set `mode=copy` for `auto.publish` and `auto.transcript`.

* `NextflowVdsl3Platform`: When a module is used multiple times in the same workflow,
don't throw an error anymore, instead simply generate a warning.

* `NextflowVdsl3Platform`: Throw an error when an input file was not found.

* `viash build`: Indent auto-generated code according the indentation of `VIASH START` when found.

* `Main`: Handle not finding the config file or resources in a config file better.
Display a more helpful message instead of a stack trace.

* `BashWrapper`: Add checks on parameters for valid integer, double and boolean values.

* `BashWrapper`: Add option to limit string and integer values to specific choice values.

* `BashWrapper`: Add option to set min and max values for integer and double values.

* Dependencies:
- Scala was upgraded from 2.12.10 to 2.12.15
- sbt was upgraded from 1.3.4 to 1.6.1
- sbt-scoverage was upgraded from 1.5.1 to 1.9.3

BUG FIXES

* `viash_test`: Add back `--no_cache` parameter to `viash_test`.

* `viash_test`: Fix `--append` parameter for `viash_test`, was not getting passed through.

* `viash ns test`: Fix `--append` parameter, actually start from a clean file if append is false.

* `viash_push`: Fix component not being built during a release of Viash.

* `PythonRequirements`: Fix packages being mentioned twice in a Dockerfile.

* `Main`: Added support spaces in filenames of config files and resources

* `BashWrapper`: Display a message when the last parsed argument would require more values than are still available.
Now display a message that values are missing, used to silently crash the wrapper.

* `viash config inject`: Fix error when file argument is `must_exist: true`.

MAJOR CHANGES

* `Functionality`: Now also accepts 'inputs' and 'outputs' in addition to 'arguments'. For inputs and outputs,
any specified arguments will have default `type: file` and `direction: input` or `direction: output` respectively.

MINOR CHANGES

* `DockerPlatform`: Move description labels to the end of the Dockerfile to improve cross-component caching.

* `Functionality`: Arguments where `.multiple` is `true` can now have lists as `default` and `example`.

* `viash_build`: Added unit test for this component.

* `viash_test`: Added unit test for this component.

* `PythonRequirements`: Allow upgrading dependencies. Example: `[ type: python. pypi: anndata, upgrade: true ]`.

* `NextflowLegacyPlatform`: Remove annoying messages when building Nxf modules.

* `ConfigMods`: Expanded the DSL to allow specifying at which point to apply a config mod.
This functionality was necessary to allow for setting fields which alter the way configs are parsed.
Example of when this is useful: `<preparse> .platforms[.type == "nextflow"].variant := "vdsl3"`.
Updating workflow of parsing a config file is:
- read Yaml from file
- apply preparse config mods
- parse resulting Json as Config, thereby instantiating default values etc.
- convert Config back to Json
- apply postparse config mods (original config mods)
- convert final Json back to Config

BETA FUNCTIONALITY

* `NextflowVdsl3Platform`: A beta implementation of the next-generation Viash+Nextflow platform.
See https://github.com/viash-io/viash/issues/82 for more information. You can access the previous Nextflow
platform by using the `variant` parameter:
yaml
- type: nextflow
variant: legacy
separate_multiple_outputs: false


BUG FIXES

* `viash_build` and `viash_test`: The `query_name` and `query_namespace` arguments were switched around. These arguments are now passed correctly.

* `BashScript`, `JavaScriptScript`, `PythonScript`, `RScript`: Correctly escape `'` (113). Update unit tests accordingly.

* `CSharpScript`, `ScalaScript`: Correctly escape `"` (113). Update unit tests accordingly.

* `viash_build`, `viash_test`, `viash_push`: Don't try to remove log files if they don't exist.

INTERNAL CHANGES

* `DataObject`:
- Renamed `otype` to `flags`.
- Renamed `oType` to `type`
- Deprecated `tag` (unused feature).

* All abstract / inherited classes: Renamed `oType` to `type`.

DEPRECATION

* `Functionality`: Deprecated `function_type` and `add_resources_to_path`. These should be
unused features, by now.