Nf-core

Template

- Fix the syntax of github_output in GitHub actions ([2114](https://github.com/nf-core/tools/pull/2114))
- Fix a bug introduced in 2.7 that made pipelines hang ([2132](https://github.com/nf-core/tools/issues/2132))

Linting

- Allow specifying containers in less than three lines ([2121](https://github.com/nf-core/tools/pull/2121))
- Run prettier after dumping a json schema file ([2124](https://github.com/nf-core/tools/pull/2124))

General

- Only check that a pipeline name doesn't contain dashes if the name is provided by prompt of `--name`. Don't check if a template file is used. ([2123](https://github.com/nf-core/tools/pull/2123))
- Deprecate `--enable_conda` parameter. Use `conda.enable` instead ([2131](https://github.com/nf-core/tools/pull/2131))
- Handle `json.load()` exceptions ([2134](https://github.com/nf-core/tools/pull/2134))

- Patch release to fix pipeline sync ([2110](https://github.com/nf-core/tools/pull/2110))

Another big release with lots of new features and bug fixes. Thanks to all contributors!

**Highlights**

- New `nf-core subworkflows` subcommand for creating, removing, testing, updating and finding subworkflows, see the [documentation](https://nf-co.re/tools/#subworkflows) for more information.
- Every pipeline has now it's own GitHub codespace template, which can be used to develop the pipeline directly in the browser.
- Improved handling of modules and subworkflows from other repos than nf-core/modules.
- Pre-commit is now installed as a dependency, which allows us, besides other things, to run prettier on the fly even if it is not manually installed.
- Shell completion for nf-core commands, more information [here](https://nf-co.re/tools#shell-completion).

Template

Features

- Ignore files in `bin/` directory when running prettier ([2080](https://github.com/nf-core/tools/pull/1957)).
- Add GitHub codespaces template ([1957](https://github.com/nf-core/tools/pull/1957))
- `nextflow run <pipeline> --version` will now print the workflow version from the manifest and exit ([1951](https://github.com/nf-core/tools/pull/1951)).
- Add profile for running `docker` with the ARM chips (including Apple silicon) ([1942](https://github.com/nf-core/tools/pull/1942) and [#2034](https://github.com/nf-core/tools/pull/2034)).
- Flip execution order of parameter summary printing and parameter validation to prevent 'hiding' of parameter errors ([2033](https://github.com/nf-core/tools/pull/2033)).
- Change colour of 'pipeline completed successfully, but some processes failed' from red to yellow ([2096](https://github.com/nf-core/tools/pull/2096)).

Bug fixes

- Fix lint warnings for `samplesheet_check.nf` module ([1875](https://github.com/nf-core/tools/pull/1875)).
- Check that the workflow name provided with a template doesn't contain dashes ([1822](https://github.com/nf-core/tools/pull/1822))

Linting

Features

- Add `--sort-by` option to linting which allows ordering module lint warnings/errors by either test name or module name ([2077](https://github.com/nf-core/tools/pull/2077)).

Bug fixes

- Don't lint pipeline name if `manifest.name` in `.nf-core.yml` ([2035](https://github.com/nf-core/tools/pull/2035))
- Don't check for `docker pull` commands in `actions_ci` lint test (leftover from DSL1) ([2055](https://github.com/nf-core/tools/pull/2055)).

General

Features

- Use pre-commit run prettier if prettier is not available ([1983](https://github.com/nf-core/tools/pull/1983)) and initialize pre-commit in gitpod and codespaces ([#1957](https://github.com/nf-core/tools/pull/1957)).
- Refactor CLI flag `--hide-progress` to be at the top-level group, like `--verbose` ([2016](https://github.com/nf-core/tools/pull/2016))
- `nf-core sync` now supports the template YAML file using `-t/--template-yaml` ([1880](https://github.com/nf-core/tools/pull/1880)).
- The default branch can now be specified when creating a new pipeline repo [1959](https://github.com/nf-core/tools/pull/1959).
- Only warn when checking that the pipeline directory contains a `main.nf` and a `nextflow.config` file if the pipeline is not an nf-core pipeline [1964](https://github.com/nf-core/tools/pull/1964)
- Bump promoted Python version from 3.7 to 3.8 ([1971](https://github.com/nf-core/tools/pull/1971)).
- Extended the chat notifications to Slack ([1829](https://github.com/nf-core/tools/pull/1829)).
- Don't print source file + line number on logging messages (except when verbose) ([2015](https://github.com/nf-core/tools/pull/2015))
- Automatically format `test.yml` content with Prettier ([2078](https://github.com/nf-core/tools/pull/2078))
- Automatically format `modules.json` content with Prettier ([2074](https://github.com/nf-core/tools/pull/2074))
- Add shell completion for nf-core tools commands([2070](https://github.com/nf-core/tools/pull/2070))

Bug fixes, maintenance and tests

- Fix error in tagging GitPod docker images during releases ([1874](https://github.com/nf-core/tools/pull/1874)).
- Fix bug when updating modules from old version in old folder structure ([1908](https://github.com/nf-core/tools/pull/1908)).
- Don't remove local copy of modules repo, only update it with fetch ([1881](https://github.com/nf-core/tools/pull/1881)).
- Improve test coverage of `sync.py` and `__main__.py` ([1936](https://github.com/nf-core/tools/pull/1936), [#1965](https://github.com/nf-core/tools/pull/1965)).
- Add file `versions.yml` when generating `test.yml` with `nf-core modules create-test-yml` but don't check for md5sum [1963](https://github.com/nf-core/tools/pull/1963).
- Mock biocontainers and anaconda api calls in modules and subworkflows tests [1967](https://github.com/nf-core/tools/pull/1967)
- Run tests with Python 3.11 ([1970](https://github.com/nf-core/tools/pull/1970)).
- Run test with a realistic version of git ([2043](https://github.com/nf-core/tools/pull/2043)).
- Fix incorrect file deletion in `nf-core launch` when `--params_in` has the same name as `--params_out` ([1986](https://github.com/nf-core/tools/pull/1986)).
- Updated GitHub actions ([1998](https://github.com/nf-core/tools/pull/1998), [#2001](https://github.com/nf-core/tools/pull/2001))
- Code maintenance ([1818](https://github.com/nf-core/tools/pull/1818), [#2032](https://github.com/nf-core/tools/pull/2032), [#2073](https://github.com/nf-core/tools/pull/2073)).
- Track from where modules and subworkflows are installed ([1999](https://github.com/nf-core/tools/pull/1999)).
- Substitute ModulesCommand and SubworkflowsCommand by ComponentsCommand ([2000](https://github.com/nf-core/tools/pull/2000)).
- Prevent installation with unsupported Python versions ([2075](https://github.com/nf-core/tools/pull/2075)).
- Allow other remote URLs not starting with `http` ([2061](https://github.com/nf-core/tools/pull/2061))

Modules

- Update patch file paths if the modules directory has the old structure ([1878](https://github.com/nf-core/tools/pull/1878)).
- Don't write to `modules.json` file when applying a patch file during `nf-core modules update` ([2017](https://github.com/nf-core/tools/pull/2017)).

Subworkflows

- Add subworkflow commands `create-test-yml`, `create` and `install` ([1897](https://github.com/nf-core/tools/pull/1897)).
- Update subworkflows install so it installs also imported modules and subworkflows ([1904](https://github.com/nf-core/tools/pull/1904)).
- `check_up_to_date()` function from `modules_json.py` also checks for subworkflows ([1934](https://github.com/nf-core/tools/pull/1934)).
- Add tests for `nf-core subworkflows install` command ([1996](https://github.com/nf-core/tools/pull/1996)).
- Function `create()` from `modules_json.py` adds also subworkflows to `modules.json` file ([2005](https://github.com/nf-core/tools/pull/2005)).
- Add `nf-core subworkflows update` command ([2019](https://github.com/nf-core/tools/pull/2019)).

Template

- Add template for subworkflows
- Add `actions/upload-artifact` step to the awstest workflows, to expose the debug log file
- Add `prettier` as a requirement to Gitpod Dockerimage
- Bioconda incompatible conda channel setups now result in more informative error messages ([1812](https://github.com/nf-core/tools/pull/1812))
- Improve template customisation documentation ([1821](https://github.com/nf-core/tools/pull/1821))
- Update MultiQC module, update supplying MultiQC default and custom config and logo files to module
- Add a 'recommend' methods description text to MultiQC to help pipeline users report pipeline usage in publications ([1749](https://github.com/nf-core/tools/pull/1749))
- Fix template spacing modified by JINJA ([1830](https://github.com/nf-core/tools/pull/1830))
- Fix MultiQC execution on template ([1855](https://github.com/nf-core/tools/pull/1855))
- Don't skip including `base.config` when skipping nf-core/configs

Linting

- Pipelines: Check that the old renamed `lib` files are not still present:
- `Checks.groovy` -> `Utils.groovy`
- `Completion.groovy` -> `NfcoreTemplate.groovy`
- `Workflow.groovy` -> `WorkflowMain.groovy`

General

- Add function to enable chat notifications on MS Teams, accompanied by `hook_url` param to enable it.
- Schema: Remove `allOf` if no definition groups are left.
- Use contextlib to temporarily change working directories ([1819](https://github.com/nf-core/tools/pull/1819))
- More helpful error messages if `nf-core download` can't parse a singularity image download
- Add `nf-core subworkflows create` command

Modules

- If something is wrong with the local repo cache, offer to delete it and try again ([1850](https://github.com/nf-core/tools/issues/1850))
- Restructure code to work with the directory restructuring in [modules](https://github.com/nf-core/modules/pull/2141) ([#1859](https://github.com/nf-core/tools/pull/1859))
- Make `label: process_single` default when creating a new module

- Patch release to fix black linting in pipelines ([1789](https://github.com/nf-core/tools/pull/1789))
- Add isort options to pyproject.toml ([1792](https://github.com/nf-core/tools/pull/1792))
- Lint pyproject.toml file exists and content ([1795](https://github.com/nf-core/tools/pull/1795))
- Update GitHub PyPI package release action to v1 ([1785](https://github.com/nf-core/tools/pull/1785))

Template

- Update GitHub actions to use nodejs16 ([1944](https://github.com/nf-core/tools/pull/1944))

Template

- Bumped Python version to 3.7 in the GitHub linting in the workflow template ([1680](https://github.com/nf-core/tools/pull/1680))
- Fix bug in pipeline readme logo URL ([1590](https://github.com/nf-core/tools/pull/1590))
- Switch CI to use [setup-nextflow](https://github.com/nf-core/setup-nextflow) action to install Nextflow ([#1650](https://github.com/nf-core/tools/pull/1650))
- Add `CITATION.cff` [361](https://github.com/nf-core/tools/issues/361)
- Add Gitpod and Mamba profiles to the pipeline template ([1673](https://github.com/nf-core/tools/pull/1673))
- Remove call to `getGenomeAttribute` in `main.nf` when running `nf-core create` without iGenomes ([1670](https://github.com/nf-core/tools/issues/1670))
- Make `nf-core create` fail if Git default branch name is dev or TEMPLATE ([1705](https://github.com/nf-core/tools/pull/1705))
- Convert `console` snippets to `bash` snippets in the template where applicable ([1729](https://github.com/nf-core/tools/pull/1729))
- Add `branch` field to module entries in `modules.json` to record what branch a module was installed from ([1728](https://github.com/nf-core/tools/issues/1728))
- Add customisation option to remove all GitHub support with `nf-core create` ([1766](https://github.com/nf-core/tools/pull/1766))

Linting

- Check that the `.prettierignore` file exists and that starts with the same content.
- Update `readme.py` nf version badge validation regexp to accept any signs before version number ([1613](https://github.com/nf-core/tools/issues/1613))
- Add isort configuration and GitHub workflow ([1538](https://github.com/nf-core/tools/pull/1538))
- Use black also to format python files in workflows ([1563](https://github.com/nf-core/tools/pull/1563))
- Add check for mimetype in the `input` parameter. ([1647](https://github.com/nf-core/tools/issues/1647))
- Check that the singularity and docker tags are parsable. Add `--fail-warned` flag to `nf-core modules lint` ([1654](https://github.com/nf-core/tools/issues/1654))
- Handle exception in `nf-core modules lint` when process name doesn't start with process ([1733](https://github.com/nf-core/tools/issues/1733))

General

- Remove support for Python 3.6 ([1680](https://github.com/nf-core/tools/pull/1680))
- Add support for Python 3.9 and 3.10 ([1680](https://github.com/nf-core/tools/pull/1680))
- Invoking Python with optimizations no longer affects the program control flow ([1685](https://github.com/nf-core/tools/pull/1685))
- Update `readme` to drop `--key` option from `nf-core modules list` and add the new pattern syntax
- Add `--fail-warned` flag to `nf-core lint` to make warnings fail ([1593](https://github.com/nf-core/tools/pull/1593))
- Add `--fail-warned` flag to pipeline linting workflow ([1593](https://github.com/nf-core/tools/pull/1593))
- Updated the nf-core package requirements ([1620](https://github.com/nf-core/tools/pull/1620), [#1757](https://github.com/nf-core/tools/pull/1757), [#1756](https://github.com/nf-core/tools/pull/1756))
- Remove dependency of the mock package and use unittest.mock instead ([1696](https://github.com/nf-core/tools/pull/1696))
- Fix and improve broken test for Singularity container download ([1622](https://github.com/nf-core/tools/pull/1622))
- Use [`$XDG_CACHE_HOME`](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) or `~/.cache` instead of `$XDG_CONFIG_HOME` or `~/config/` as base directory for API cache
- Switch CI to use [setup-nextflow](https://github.com/nf-core/setup-nextflow) action to install Nextflow ([#1650](https://github.com/nf-core/tools/pull/1650))
- Add tests for `nf-core modules update` and `ModulesJson`.
- Add CI for GitLab remote [1646](https://github.com/nf-core/tools/issues/1646)
- Add `CITATION.cff` [361](https://github.com/nf-core/tools/issues/361)
- Allow customization of the `nf-core` pipeline template when using `nf-core create` ([1548](https://github.com/nf-core/tools/issues/1548))
- Add Refgenie integration: updating of nextflow config files with a refgenie database ([1090](https://github.com/nf-core/tools/pull/1090))
- Fix `--key` option in `nf-core lint` when supplying a module lint test name ([1681](https://github.com/nf-core/tools/issues/1681))
- Add `no_git=True` when creating a new pipeline and initialising a git repository is not needed in `nf-core lint` and `nf-core bump-version` ([1709](https://github.com/nf-core/tools/pull/1709))
- Move `strip_ansi_code` function in lint to `utils.py`
- Simplify control flow and don't use equality comparison for `None` and booleans
- Replace use of the deprecated `distutils` Version object with that from `packaging` ([1735](https://github.com/nf-core/tools/pull/1735))
- Add code to cancel CI run if a new run starts ([1760](https://github.com/nf-core/tools/pull/1760))
- CI for the API docs generation now uses the ubuntu-latest base image ([1762](https://github.com/nf-core/tools/pull/1762))
- Add option to hide progress bars in `nf-core lint` and `nf-core modules lint` with `--hide-progress`.

Modules

- Add `--fix-version` flag to `nf-core modules lint` command to update modules to the latest version ([1588](https://github.com/nf-core/tools/pull/1588))
- Fix a bug in the regex extracting the version from biocontainers URLs ([1598](https://github.com/nf-core/tools/pull/1598))
- Update how we interface with git remotes. ([1626](https://github.com/nf-core/tools/issues/1626))
- Add prompt for module name to `nf-core modules info` ([1644](https://github.com/nf-core/tools/issues/1644))
- Update docs with example of custom git remote ([1645](https://github.com/nf-core/tools/issues/1645))
- Command `nf-core modules test` obtains module name suggestions from installed modules ([1624](https://github.com/nf-core/tools/pull/1624))
- Add `--base-path` flag to `nf-core modules` to specify the base path for the modules in a remote. Also refactored `modules.json` code. ([1643](https://github.com/nf-core/tools/issues/1643)) Removed after ([#1754](https://github.com/nf-core/tools/pull/1754))
- Rename methods in `ModulesJson` to remove explicit reference to `modules.json`
- Fix inconsistencies in the `--save-diff` flag `nf-core modules update`. Refactor `nf-core modules update` ([1536](https://github.com/nf-core/tools/pull/1536))
- Fix bug in `ModulesJson.check_up_to_date` causing it to ask for the remote of local modules
- Handle errors when updating module version with `nf-core modules update --fix-version` ([1671](https://github.com/nf-core/tools/pull/1671))
- Make `nf-core modules update --save-diff` work when files were created or removed ([1694](https://github.com/nf-core/tools/issues/1694))
- Get the latest common build for Docker and Singularity containers of a module ([1702](https://github.com/nf-core/tools/pull/1702))
- Add short option for `--no-pull` option in `nf-core modules`
- Add `nf-core modules patch` command ([1312](https://github.com/nf-core/tools/issues/1312))
- Add support for patch in `nf-core modules update` command ([1312](https://github.com/nf-core/tools/issues/1312))
- Add support for patch in `nf-core modules lint` command ([1312](https://github.com/nf-core/tools/issues/1312))
- Add support for custom remotes in `nf-core modules lint` ([1715](https://github.com/nf-core/tools/issues/1715))
- Make `nf-core modules` commands work with arbitrary git remotes ([1721](https://github.com/nf-core/tools/issues/1721))
- Add links in `README.md` for `info` and `patch` commands ([1722](https://github.com/nf-core/tools/issues/1722)])
- Fix misc. issues with `--branch` and `--base-path` ([1726](https://github.com/nf-core/tools/issues/1726))
- Add `branch` field to module entries in `modules.json` to record what branch a module was installed from ([1728](https://github.com/nf-core/tools/issues/1728))
- Fix broken link in `nf-core modules info`([1745](https://github.com/nf-core/tools/pull/1745))
- Fix unbound variable issues and minor refactoring [1742](https://github.com/nf-core/tools/pull/1742/)
- Recreate modules.json file instead of complaining about incorrectly formatted file. ([1741](https://github.com/nf-core/tools/pull/1741)
- Add support for patch when creating `modules.json` file ([1752](https://github.com/nf-core/tools/pull/1752))