Nextstrain-cli

This release contains a **potentially-breaking change** for existing usages of
`nextstrain view`, though we expect the change to impact very few usages. The
change is described below.

Improvements

* `nextstrain view` now supports viewing narratives, as was always intended.
Previously the launched Auspice would either show baked in test narratives or
no narratives at all, depending on the Auspice version in the runtime.
([240][])

* `nextstrain view` now supports being given more kinds of paths, including
paths to a specific dataset or narrative file and paths to directories
containing _auspice/_ and/or _narratives/_ subdirectories.

This is a **potentially-breaking change**, as `nextstrain view <dir>` will
now prefer to show datasets from _`<dir>`/auspice/_ if that subdirectory
exists. Previously it would only ever look for datasets in the given
_`<dir>`_. We expect this to change behaviour for very few usages as it only
affects situations where _`<dir>`_ contains both datasets and an _auspice/_
directory.

See `nextstrain view --help` for more details on the kinds of paths
supported.
([240][])

* `nextstrain view` now automatically opens Auspice in a new browser tab (or
window) by default when possible.

If a specific dataset or narrative file was given as the path to `nextstrain
view`, then that dataset or narrative is opened. Otherwise, if there's only
a single dataset or narrative available in the directory path given to
`nextstrain view`, then it is opened. Otherwise, Auspice's listing of
available datasets and narratives is opened.
([240][])

[240]: https://github.com/nextstrain/cli/pull/240

* Local images used in a narrative are now automatically embedded into it when
uploading the narrative to nextstrain.org via `nextstrain remote upload`.
In local text editors which can render Markdown, this permits previewing of
narratives that reference images on the local filesystem without requiring
manual conversion to remote images or embedded images before upload.
([235](https://github.com/nextstrain/cli/pull/235))

* The `nextstrain remote upload` command now outputs a nicer error message
that's more interpretable and actionable when nextstrain.org returns a "bad
request" error. The error message also includes the error details returned
by nextstrain.org.
([238](https://github.com/nextstrain/cli/pull/238))

Development

* The Conda runtime now uses Micromamba 1.0.0 (an upgrade from 0.27.0).
([233](https://github.com/nextstrain/cli/pull/233))

Bug fixes

* `nextstrain shell` no longer errors when its history file, e.g.
_~/.nextstrain/shell-history_, doesn't exist. This primarily affected the
Docker runtime and was a regression from 4.2.0 introduced in 5.0.0.
([232](https://github.com/nextstrain/cli/pull/232))

prior to final release. For convenience, the changes from those pre-releases
are also re-described here._

The major improvement in this release is the introduction of a new Conda
runtime, filling a gap between the Docker runtime and the ambient runtime
(formerly "native" runtime). See more details below.

This release also contains **potentially-breaking changes** for existing
usages of `nextstrain remote download` and `nextstrain update`. The changes
are described below.

Improvements

* A new Conda runtime (aka runner or build environment) now complements the
existing Docker and ambient runtimes and fills a gap between them. This
runtime is more isolated and reproducible than your ambient environment, but
is less isolated and robust than the Docker runtime. Like the Docker
runtime, the Conda runtime is fully-managed by Nextstrain CLI and receives
updates via `nextstrain update`.

The new runtime uses the [Conda](https://docs.conda.io) ecosystem with
packages from [our own channel](https://anaconda.org/Nextstrain/nextstrain-base),
[Bioconda](https://bioconda.github.io/) and
[Conda-Forge](https://conda-forge.org/), installed by
[Micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
into an isolated location, typically `~/.nextstrain/runtimes/conda/env`. It
does not interact with or impact other usage of Conda/Mamba environments and
will not, for example, appear in the output of `conda env list`.

Set up of the runtime is automated and can be performed by running:

nextstrain setup conda

When complete, you'll be able to use the `--conda` runtime option supported
by Nextstrain CLI commands such as `nextstrain build`, `nextstrain view`,
`nextstrain shell`, etc.
([218][])

* The "native" runtime (aka runner or build environment) is now the "ambient"
runtime. This name better reflects what it is and further distinguishes it
from the new Conda runtime, which is also "native" in the binary executable
sense.

Existing usages of "native" should be unaffected. The `--native` option
continues to work anywhere it used to previously, though it is hidden from
`--help` output to discourage new use. The string "native" is also accepted
anywhere runner names are accepted, e.g. in config as the `core.runner`
setting or in command-line arguments to `check-setup` or `setup`.
([224](https://github.com/nextstrain/cli/pull/224))

* `nextstrain setup docker` now downloads the Docker runtime image if it's not
already available locally. This can be a useful initial step after
installation to avoid the automatic download on first use.
([222](https://github.com/nextstrain/cli/pull/222))

* The local filenames produced by `nextstrain remote download` now include
more of the remote dataset/narrative path. This reduces the potential for
ambiguous filenames and makes it easier to copy datasets/narratives between
destinations (e.g. from one group to another) while retaining the same path.
It is, however, a **potentially-breaking change** if you're relying on the
filenames of the downloaded datasets/narratives (e.g. for automation).

For example, downloading `nextstrain.org/flu/seasonal/h3n2/ha/2y` previously
produced the local files:


2y.json
2y_root-sequence.json
2y_tip-frequencies.json


which could easily conflict with the similarly-named
`nextstrain.org/flu/seasonal/h3n2/na/2y`,
`nextstrain.org/flu/seasonal/h1n1pdm/ha/2y`, etc. The downloaded files are
now named:


flu_seasonal_h3n2_ha_2y.json
flu_seasonal_h3n2_ha_2y_root-sequence.json
flu_seasonal_h3n2_ha_2y_tip-frequencies.json


Within groups, filenames are similarly longer but the group name is not
included. For example, downloading `groups/blab/ncov/cross-species/cat`
previously produced:


cat.json
cat_root-sequence.json
cat_tip-frequencies.json


and now produces:


ncov_cross-species_cat.json
ncov_cross-species_cat_root-sequence.json
ncov_cross-species_cat_tip-frequencies.json

([213](https://github.com/nextstrain/cli/pull/213))

* Advanced globbing features are now supported in patterns for the `--download`
option of `nextstrain build`, including multi-part wildcards (`**`), extended
globbing (`(…)`, `+(…)`, etc.), and negation (`!…`). Basic globbing
features like single-part wildcards (`*`), character classes (`[…]`), and
brace expansion (`{…, …}`) are still supported. Note that the `--download`
option continues to be applicable only to the AWS Batch runtime (e.g. the
`--aws-batch` option).
([215](https://github.com/nextstrain/cli/pull/215))

* `check-setup` now accepts one or more runtime names as arguments.

The default behaviour doesn't change, but specifying runtimes now lets you
restrict checks to a single runtime or, with multiple runtimes, re-order them
by your preference for use with --set-default.
([218][])

* `update` now only updates a specific runtime instead of all of them at once.

With no arguments, the default runtime is updated. The name of another
runtime to update instead may be provided as an argument.

In practice this isn't much of a behaviour change because only one runtime
currently supports updating (Docker); the others (ambient, AWS Batch) just
pass. Existing users are unlikely to notice the change unless they use
multiple runtimes and Docker is not their default. In that case, `update`
may stop updating Docker for them when it would have done so previously,
which is a **potentially-breaking change**.
([218][])

* A new command, `setup`, now exists to perform automatic set up of runtimes
that support it (currently only Conda). For all runtimes, even those that
don't support automatic set up, the `setup` command will also run the same
checks as `check-setup` and optionally set the default runtime.
([218][])

* The shell launched by the `shell` command now remembers its own command
history and differentiates its command prompt from other shells with a
stylized variant of the Nextstrain wordmark.
([218][])

* The output of commands in dry run mode (e.g. with the `--dry-run` option) is
now uniformly indicated to be a dry run by the prefix `DRY RUN │ `. This
includes the `remote` family of commands and the new `setup` command.
([218][])

* Runtime checks in `check-setup` and `setup` now test for not just the
presence of Snakemake, Augur, and Auspice, but also that they can be
executed.
([218][])

Development

* We now provide standalone installers (i.e. shell programs) to download and
unpack the standalone installation archives into standard locations,
potentially upgrading/overwriting a prior standalone install. These
installers will be served from GitHub directly out of this project's
repository via convenience redirects on nextstrain.org.

These will eventually form the basis for Nextstrain install instructions that
don't suffer from Python bootstrapping issues. As a preview for now, you can
play around with the following platform-specific commands:

curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/linux | bash
curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/mac | bash
Invoke-RestMethod https://nextstrain.org/cli/installer/windows | Invoke-Expression

A new companion command, `init-shell`, exists to simplify shell configuration
(i.e. `PATH` modification) for such installations.

* The `NEXTSTRAIN_HOME` environment variable can now be used to specify the
desired location for per-user settings, files, etc., overriding the default
of _~/.nextstrain/_.
([218][])

* A new `nextstrain authorization` command makes it easier to generate direct
requests to nextstrain.org's web API using the same credentials as the CLI.
([229](https://github.com/nextstrain/cli/pull/229))

* The development documentation now documents how to build the documentation
locally, and sphinx-autobuild is used to make a very nice edit-preview cycle
with quick turnaround.
([218][])

* Development dependency issues with `flake8` and `sphinx-markdown-tables`,
caused by upstream changes, are now resolved.
([218][])

_This is the second development pre-release made prior to the final release of 5.0.0._

Improvements

* The "native" runtime (aka runner or build environment) is now the "ambient"
runtime. This name better reflects what it is and further distinguishes it
from the new Conda runtime, which is also "native" in the binary executable
sense.

Existing usages of "native" should be unaffected. The `--native` option
continues to work anywhere it used to previously, though it is hidden from
`--help` output to discourage new use. The string "native" is also accepted
anywhere runner names are accepted, e.g. in config as the `core.runner`
setting or in command-line arguments to `check-setup` or `setup`.
([224](https://github.com/nextstrain/cli/pull/224))

* `nextstrain setup docker` now downloads the Docker runtime image if it's not
already available locally. This can be a useful initial step after
installation to avoid the automatic download on first use.
([222](https://github.com/nextstrain/cli/pull/222))

* `nextstrain build`'s check for use of the `--image` option with unsupported
runtimes now includes the Conda runtime.
([224](https://github.com/nextstrain/cli/pull/224))

* The Conda runtime now uses the new [`nextstrain-base` Conda
meta-package](https://anaconda.org/Nextstrain/nextstrain-base) instead of
using a hardcoded list of packages.

This decouples Conda runtime updates from Nextstrain CLI updates, as we can
make new releases of `nextstrain-base` and users can update to those without
upgrading Nextstrain CLI itself. This brings the update story for the Conda
runtime into much better parity with the Docker runtime.

Using the meta-package also brings increased reproducibility to the runtime,
as the package completely locks its full transitive dependency tree. This
means that if version _X_ of `nextstrain-base` worked in the past, it'll
still work the same way in the future.

The `NEXTSTRAIN_CONDA_BASE_PACKAGE` environment variable may be used with
`nextstrain setup conda` to install a specific version. The value is a
[Conda package specification][], e.g. `nextstrain-base ==X`.
([228](https://github.com/nextstrain/cli/pull/228))

* The Conda runtime now uses a pinned version of Micromamba (currently 0.27.0)
so that new releases of the latter can't break `nextstrain setup conda` or
`nextstrain update conda` between one day and the next. The pinned version
will be bumped up over time as needed with subsequent releases of Nextstrain
CLI.

The `NEXTSTRAIN_CONDA_MICROMAMBA_VERSION` environment variable may be used
with `nextstrain setup conda` to override the built-in pin, either with
another specific version or `latest`.

[Conda package specification]: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications

Bug fixes

* The Conda runtime now runs Micromamba in greater isolation to avoid undesired
interactions when a) Nextstrain CLI itself is running inside an
externally-activated Conda environment and/or b) user-specific Mamba
configuration exists. This applies to usages of `nextstrain setup` and
`nextstrain update` with the Conda runtime.
([223](https://github.com/nextstrain/cli/pull/223))

* The Conda runtime now configures the appropriate channels during `update` too,
not just during `setup`, ensuring package updates are found.
([228](https://github.com/nextstrain/cli/pull/228))

* The Conda runtime now avoids pinning Python in the isolated environment to
allow it to be upgraded by `update`.
([228](https://github.com/nextstrain/cli/pull/228))

Development

* The Conda runtime is now tested in CI, joining the Docker and ambient
runtimes.
([223](https://github.com/nextstrain/cli/pull/223))

_This is a development pre-release made prior to the final release of 5.0.0._

The major improvement in this release is the introduction of a new Conda
runtime, filling a gap between the Docker runtime and the "native" (soon to be
"ambient") runtime. See more details below.

This release also contains **a potentially-breaking change** for existing
usages of `nextstrain remote download` and `nextstrain update`. The changes
are described below.

Improvements

* A new Conda runtime (aka runner or build environment) now complements the
existing Docker and "native" runtimes and fills a gap between them. This
runtime is more isolated and reproducible than your native ambient
environment, but is less isolated and robust than the Docker runtime. Like
the Docker runtime, the Conda runtime is fully-managed by Nextstrain CLI and
receives updates via `nextstrain update`.

The new runtime uses the [Conda](https://docs.conda.io) ecosystem with
packages from [Bioconda](https://bioconda.github.io/) and
[Conda-Forge](https://conda-forge.org/), installed by
[Micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
into an isolated location, typically `~/.nextstrain/runtimes/conda/env`. It
does not interact with or impact other usage of Conda/Mamba environments and
will not, for example, appear in the output of `conda env list`.

Set up of the runtime is automated and can be performed by running:

nextstrain setup conda

When complete, you'll be able to use the `--conda` runtime option supported
by Nextstrain CLI commands such as `nextstrain build`, `nextstrain view`,
`nextstrain shell`, etc.
([218][])

* The local filenames produced by `nextstrain remote download` now include
more of the remote dataset/narrative path. This reduces the potential for
ambiguous filenames and makes it easier to copy datasets/narratives between
destinations (e.g. from one group to another) while retaining the same path.
It is, however, a **potentially-breaking change** if you're relying on the
filenames of the downloaded datasets/narratives (e.g. for automation).

For example, downloading `nextstrain.org/flu/seasonal/h3n2/ha/2y` previously
produced the local files:


2y.json
2y_root-sequence.json
2y_tip-frequencies.json


which could easily conflict with the similarly-named
`nextstrain.org/flu/seasonal/h3n2/na/2y`,
`nextstrain.org/flu/seasonal/h1n1pdm/ha/2y`, etc. The downloaded files are
now named:


flu_seasonal_h3n2_ha_2y.json
flu_seasonal_h3n2_ha_2y_root-sequence.json
flu_seasonal_h3n2_ha_2y_tip-frequencies.json


Within groups, filenames are similarly longer but the group name is not
included. For example, downloading `groups/blab/ncov/cross-species/cat`
previously produced:


cat.json
cat_root-sequence.json
cat_tip-frequencies.json


and now produces:


ncov_cross-species_cat.json
ncov_cross-species_cat_root-sequence.json
ncov_cross-species_cat_tip-frequencies.json

([213](https://github.com/nextstrain/cli/pull/213))

* Advanced globbing features are now supported in patterns for the `--download`
option of `nextstrain build`, including multi-part wildcards (`**`), extended
globbing (`(…)`, `+(…)`, etc.), and negation (`!…`). Basic globbing
features like single-part wildcards (`*`), character classes (`[…]`), and
brace expansion (`{…, …}`) are still supported. Note that the `--download`
option continues to be applicable only to the AWS Batch runtime (e.g. the
`--aws-batch` option).
([215](https://github.com/nextstrain/cli/pull/215))

* `check-setup` now accepts one or more runtime names as arguments.

The default behaviour doesn't change, but specifying runtimes now lets you
restrict checks to a single runtime or, with multiple runtimes, re-order them
by your preference for use with --set-default.
([218][])

* `update` now only updates a specific runtime instead of all of them at once.

With no arguments, the default runtime is updated. The name of another
runtime to update instead may be provided as an argument.

In practice this isn't much of a behaviour change because only one runtime
currently supports updating (Docker); the others (native, AWS Batch) just
pass. Existing users are unlikely to notice the change unless they use
multiple runtimes and Docker is not their default. In that case, `update`
may stop updating Docker for them when it would have done so previously,
which is a **potentially-breaking change**.
([218][])

* A new command, `setup`, now exists to perform automatic set up of runtimes
that support it (currently only Conda). For all runtimes, even those that
don't support automatic set up, the `setup` command will also run the same
checks as `check-setup` and optionally set the default runtime.
([218][])

* The shell launched by the `shell` command now remembers its own command
history and differentiates its command prompt from other shells with a
stylized variant of the Nextstrain wordmark.
([218][])

* The output of commands in dry run mode (e.g. with the `--dry-run` option) is
now uniformly indicated to be a dry run by the prefix `DRY RUN │ `. This
includes the `remote` family of commands and the new `setup` command.
([218][])

* Runtime checks in `check-setup` and `setup` now test for not just the
presence of Snakemake, Augur, and Auspice, but also that they can be
executed.
([218][])

Development

* We now provide standalone installers (i.e. shell programs) to download and
unpack the standalone installation archives into standard locations,
potentially upgrading/overwriting a prior standalone install. These
installers will be served from GitHub directly out of this project's
repository via convenience redirects on nextstrain.org.

These will eventually form the basis for Nextstrain install instructions that
don't suffer from Python bootstrapping issues. As a preview for now, you can
play around with the following platform-specific commands:

curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/linux | bash
curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/mac | bash
Invoke-RestMethod https://nextstrain.org/cli/installer/windows | Invoke-Expression

A new companion command, `init-shell`, exists to simplify shell configuration
(i.e. `PATH` modification) for such installations.

* The `NEXTSTRAIN_HOME` environment variable can now be used to specify the
desired location for per-user settings, files, etc., overriding the default
of _~/.nextstrain/_.
([218][])

* The development documentation now documents how to build the documentation
locally, and sphinx-autobuild is used to make a very nice edit-preview cycle
with quick turnaround.
([218][])

* Development dependency issues with `flake8` and `sphinx-markdown-tables`,
caused by upstream changes, are now resolved.
([218][])


[218]: https://github.com/nextstrain/cli/pull/218

Bug fixes

* Using `remote delete` on nextstrain.org now correctly outputs the "Deleting…"
message *before* performing each delete, as intended (and as S3 remotes do).
Previously, the message was misleadingly output *after* each delete was
already performed.
([209](https://github.com/nextstrain/cli/pull/209))

* Detection of the installation method during self-upgrade checks in
`nextstrain update` and `nextstrain check-setup` now looks for explicit
installer metadata files and no longer assumes Pip as the final fallback. If
the installation method is not explicitly detected, then no upgrade
instructions are shown. Better to suggest nothing than to suggest the wrong
thing.
([207][])

* An uncaught `StopIteration` error that could have occurred in very specific
and limited circumstances during self-upgrade checks in `nextstrain update`
and `nextstrain check-setup` can no longer occur.
([207][])

Improvements

* The `nextstrain remote download`, `upload`, and `delete` commands now support
a `--dry-run` mode.

This mode, as is broader convention, goes through the motions of doing the
thing, as much as possible, but doesn't _actually_ do the thing. That is, no
changes should occur when `--dry-run` is active.

This is particularly useful for seeing what will happen if you're unsure of
how a path or argument is handled. Dry runs of the `list` (`ls`) command
don't make sense and aren't included.
([210](https://github.com/nextstrain/cli/pull/210))

* Installations via a Conda package are now detected during self-upgrade checks
and the suggested upgrade command uses `mamba install` or `conda install`.
([207][])

* Suggested upgrade commands now specify the expected new version so they fail
if that version is not found rather than succeed but do nothing (or do
something different).
([207][])

Development

* A new debugging mode can be enabled by setting the `NEXTSTRAIN_DEBUG`
environment variable to `1` (or another truthy value). Currently the only
effect is to print more information about handled (i.e. anticipated) errors.
For example, stack traces and parent exceptions in an exception chain are
normally omitted for handled errors, but setting this env var includes them.
Future debugging and troubleshooting features, like verbose operation
logging, will likely also condition on this new debugging mode.

* We now avoid a runtime dep on setuptools by switching from
`pkg_resources.parse_version` to `packaging.version.parse`. The latter was
already transitively in our dep tree.
([207][])


[207]: https://github.com/nextstrain/cli/pull/207