Nextstrain-cli

Improvements

* Commands which utilize a runtime—`nextstrain build`, `nextstrain shell`, and
`nextstrain view`—now support specifying envdirs to forward into the runtime
environment by setting `NEXTSTRAIN_RUNTIME_ENVDIRS` to a `:`-separated (`;`
on Windows) list of paths. This is in addition to the existing support for
specifying one or more `--envdir` options.
([365](https://github.com/nextstrain/cli/pull/365))

Features

* `nextstrain build` and `nextstrain shell` now better support pathogen
repositories which place workflows in subdirectories. The top-level of the
repo must contain a `nextstrain-pathogen.yaml` file for this support to
activate. The file may be empty for now, though we anticipate using it for
pathogen-level metadata in the future to aid indexing, listing, and
attribution of pathogen repos.

As an example of the new support, consider the following repo layout

mpox/
├── nextstrain-pathogen.yaml
├── ingest/
│ ├── Snakefile
│ └── …
├── phylogenetic/
│ ├── Snakefile
│ └── …
├── shared/
│ ├── reference.fasta
│ └── …
└── …

where `ingest/` and `phylogenetic/` contain workflows that use
`shared/reference.fasta` via a relative path (i.e.
`../shared/reference.fasta`).

It's now possible to invoke those workflows with any of the following:

nextstrain build mpox/ingest/
nextstrain build mpox/phylogenetic/

cd mpox
nextstrain build ingest/
nextstrain build phylogenetic/

cd phylogenetic
nextstrain build .
nextstrain build ../ingest/

regardless of runtime.

Previously, such workflows required careful invocation, e.g.

nextstrain build mpox/ -d phylogenetic/ -s phylogenetic/Snakefile

when using runtimes with filesystem isolation (i.e. the [containerized][]
ones; Docker, Singularity, and AWS Batch) but not when using runtimes without
it.

When active, this feature makes the top-level of the pathogen repo (e.g.
`mpox/`) available in the container at `/nextstrain/build` while the
initial working directory is set to the workflow subdirectory in the
container (e.g. `/nextstrain/build/phylogenetic`). That is, the filesystem
isolation boundary is drawn at the top-level of the pathogen repo instead of
at the workflow directory (i.e. what's given to `nextstrain build`).
([355](https://github.com/nextstrain/cli/pull/355))

[containerized]: https://docs.nextstrain.org/projects/cli/en/8.2.0/runtimes/#comparison

Improvements

* We now produce standalone installation archives for macOS running on aarch64
hardware (aka arm64, Apple Silicon, M1/M2). The standalone installer will
use these archives starting with this release.
([357](https://github.com/nextstrain/cli/pull/357),
[358](https://github.com/nextstrain/cli/pull/358))

* The Conda and Docker runtime checks performed by `nextstrain setup` and
`nextstrain check-setup` now test if Rosetta 2 is enabled for macOS on
aarch64 (aka arm64, Apple Silicon, M1/M2) hardware. Rosetta is required for
the Conda runtime and optional, but recommended, for the Docker runtime.
Previously only the standalone installer checked for Rosetta, but starting
with this release it will not.
([361](https://github.com/nextstrain/cli/pull/361),
[358](https://github.com/nextstrain/cli/pull/358))

* `nextstrain build` now errors if a [development overlay option][] such as
`--augur` or `--auspice` is given when using a runtime without support for
those (anything but Docker or Singularity). Previously, it would silently
ignore those options when unsupported by the runtime. The new behaviour
matches the behaviour of `nextstrain shell` since 5.0.0.
([354](https://github.com/nextstrain/cli/pull/354))

[development overlay option]: https://docs.nextstrain.org/projects/cli/en/8.2.0/commands/build/#development-options-for-docker

Improvements

* Several kinds of errors from `nextstrain login` and `nextstrain whoami`
related to their interactions with a remote server are now clearer.
([347](https://github.com/nextstrain/cli/pull/347))

This release drops support for Python versions 3.6 and 3.7 and adds support for
3.11. ([325](https://github.com/nextstrain/cli/pull/325),
[327](https://github.com/nextstrain/cli/pull/327))

Note that this Python version support only matters if you're installing
Nextstrain CLI from PyPI or Bioconda
([c.f.](https://docs.nextstrain.org/projects/cli/en/8.0.0/installation/)). It
does not apply if you're installing Nextstrain CLI using the standalone
installation method we recommend in the [Nextstrain installation
documentation](https://docs.nextstrain.org/page/install.html). In that case, a
supported Python version is always bundled with `nextstrain`.

Improvements

* The `nextstrain remote` family of commands now support alternative
nextstrain.org-like remotes such as internal Nextstrain Groups Server
instances and development instances of nextstrain.org. Authentication with
these remotes is supported via `nextstrain login` and related commands.
Remotes maintain their authentication alongside each other, e.g. you can be
logged into nextstrain.org as well as an alternative nextstrain.org-like
instance.

As an example, a Nextstrain Groups Server instance accessible at
nextstrain.example.com could now be logged into and interacted with like so:

nextstrain login nextstrain.example.com
nextstrain whoami nextstrain.example.com
nextstrain remote ls nextstrain.example.com/groups/bedford-lab/

The default remote is still nextstrain.org.
([333](https://github.com/nextstrain/cli/pull/333))

* `nextstrain login` now performs authentication via a web browser by default
(using OpenID Connect 1.0 and OAuth 2.0). The previously method of direct
password entry is still used when a username is provided (e.g. with
`--username` or `-u`). See `nextstrain login --help` for more information.
([333](https://github.com/nextstrain/cli/pull/333))

* With the new support for being logged into multiple remotes, `nextstrain
logout` now also supports an `--all` flag to remove all locally-saved
credentials without having to logout of each remote separately.
([333](https://github.com/nextstrain/cli/pull/333))

* `nextstrain remote upload` now skips gzip compression when uploading
zstandard-compressed (`.zst`) files, matching its behaviour for other types
of compressed files.
([330](https://github.com/nextstrain/cli/pull/330))

* Commands that may automatically open a web browser, e.g. `nextstrain view` or
`nextstrain login`, now respect the semi-conventional `NOBROWSER` environment
variable to disable opening a browser. Set `NOBROWSER` to a non-empty value
to disable automatic opening. When disabled, the URL to manually open will
be shown.
([332](https://github.com/nextstrain/cli/pull/332))

* The error message emitted by the `nextstrain remote` family of commands when
they're unable to access an S3 bucket now lists a few possible reasons for
the failure.
([341](https://github.com/nextstrain/cli/pull/341))

* `nextstrain remote download` now supports downloading core datasets which are
only visible on the web by direct access via their URL. For example, the
following now work (where in previous versions they did not):

nextstrain remote download https://nextstrain.org/nextclade/sars-cov-2/21L
nextstrain remote download https://nextstrain.org/enterovirus/d68/vp1/2020-01-23

This support also covers [past snapshots of
datasets](https://docs.nextstrain.org/en/latest/guides/versions.html), which
is a recently-added feature to nextstrain.org. For example:

nextstrain remote download https://nextstrain.org/flu/seasonal/h3n2/ha/6y2023-07-01
nextstrain remote download https://nextstrain.org/ncov/gisaid/21L/global/6m2024-01-09

([345](https://github.com/nextstrain/cli/pull/345))

Bug fixes

* Commands making use of user authentication (e.g. `nextstrain login`,
`nextstrain whoami`, some invocations of `nextstrain remote`, etc.) no longer
fail if the logged in user is a member of no groups.
([323](https://github.com/nextstrain/cli/pull/323))

* On Linux and macOS, the standalone installer's final instructions now
correctly suggest modifying `~/.profile` instead of, incorrectly, `~/.shrc`
when the default shell is sh.
([321](https://github.com/nextstrain/cli/pull/321))

* The Singularity runtime once again supports Apptainer's `singularity`
executable. The minimum version checking added in 7.0.0 broke usage of the
Singularity runtime with Apptainer (compared with SingularityCE). Our intent
is to support both lineages of Singularity. Thanks to osageorange for
raising this issue and testing the fix!
([343](https://github.com/nextstrain/cli/pull/343))

Improvements

* When attached to an AWS Batch job, `nextstrain build` now only requires
confirmation to stop the job (e.g. a double Control-C/`SIGINT`) when stdin is
a terminal (TTY). When stdin is not a terminal, as is frequently the case in
automated or programmatic contexts, then `SIGINT` requests job cancellation
immediately.
([308][])

* When attached to an AWS Batch job, `nextstrain build` now also detaches upon
receiving `SIGHUP` (in addition to detaching upon receiving
Control-Z/`SIGTSTP`). `SIGHUP` is sent, for example, when a user closes (or
loses) the terminal in which `build` is still running.
([308][])

* When launching or attaching to an AWS Batch job, `nextstrain build` now
supports the `--detach-on-interrupt` option to change Control-C/`SIGINT` to a
request to detach from the job rather than to cancel it. This is useful in
automated contexts where `SIGINT` may be sent outside of a user's control,
and it's also handy as a molly-guard to avoid accidental cancellation when
the intent is to just observe a job.
([308][])

* When attaching to an AWS Batch job, `nextstrain build` now supports the
`--cancel` option to immediately request cancellation.
([308][])

[308]: https://github.com/nextstrain/cli/pull/308

Bug fixes

* We've plugged some isolation leaks in the Conda runtime where the
[Python user site directory](https://docs.python.org/3/library/site.html),
e.g. `~/.local/lib/pythonX.Y/site-packages`, as well as the
[`PYTHONPATH` and `PYTHONHOME` environment variables](https://docs.python.org/3/using/cmdline.html#environment-variables)
could influence and break the runtime.
([311](https://github.com/nextstrain/cli/pull/311))

_See also changes in 7.3.0 which was an unreleased version._

Development

* Update CI to test against the SingularityCE 3.x series only ([314](https://github.com/nextstrain/cli/pull/314))