Nextstrain-augur

Features

* refine: Upgrade TreeTime from 0.8.6 to >= 0.9.2 which enables a speedup of timetree inference in marginal mode due to the use of Fast Fourier Transforms [1018][]. (rneher and anna-parker). Use the `refine` flag `--use-fft` to use this feature.

Bug Fixes

* refine, export v1: Use pandas.DataFrame.at instead of .loc for single values [979][]. (victorlin)
* refine: Gracefully handle all exceptions from TreeTime [1023][]. (anna-parker)
* refine: Document branch length units `treetime` expects [1024][]. (anna-parker)
* dates: Raise an error when metadata to `get_numerical_dates()` is not a pandas DataFrame [1026][]. (victorlin)

[979]: https://github.com/nextstrain/augur/pull/979
[1018]: https://github.com/nextstrain/augur/pull/1018
[1023]: https://github.com/nextstrain/augur/pull/1023
[1024]: https://github.com/nextstrain/augur/pull/1024
[1026]: https://github.com/nextstrain/augur/pull/1026

Major Changes

* Moved the following modules to subpackages [1002][]. (joverlee521)
These are technically breaking changes for the API, but they do not change the Augur CLI commands.
* `import.py` -> `import_/__init__.py`
* `import_beast.py` -> `import_/beast.py`
* `measurements.py` -> `measurements/__init__.py` + `measurements/concat.py` + `measurements/export.py`
* Move the following internal functions/classes [1002][]. (joverlee521)
* `augur.add_default_command` -> `argparse_.add_default_command`
* `utils.HideAsFalseAction` -> `argparse_.HideAsFalseAction`
* Subcommands must include a `register_parser` function to add their own parser instead of a `register_arguments` function [1002][]. (joverlee521)
* utils: Remove internal function `utils.read_metadata()` [978][]. (victorlin)
* Use `io.read_metadata()` going forwards.
* To switch to using metadata as a pandas DataFrame (recommended):
* Iterate through strains: `metadata.items()` -> `metadata.iterrows()`
* Check strain presence: `strain in metadata` -> `strain in metadata.index`
* Check field presence: `field in metadata[strain]` -> `field in metadata.columns`
* Get metadata for a strain: `metadata[strain]` -> `metadata.loc[strain]`
* Get field for a strain: `metadata[strain][field]` -> `metadata.at[strain, field]`
* To keep using metadata in a dictionary:
py
metadata = read_metadata(args.metadata)
metadata.insert(0, "strain", metadata.index.values)
columns = metadata.columns
metadata = metadata.to_dict(orient="index")


Features

* export: `--skip-validation` now also skips version compatibility checks [902][]. (corneliusroemer)
* filter: Report names of duplicate strains found during metadata parsing [1008][] (huddlej)
* translate: Add support for Nextclade gene map GFFs [1017][] (huddlej)

Bug Fixes

* filter: Rename internal force inclusion filtering functions [1006][] (victorlin)

[902]: https://github.com/nextstrain/augur/pull/902
[978]: https://github.com/nextstrain/augur/pull/978
[1002]: https://github.com/nextstrain/augur/pull/1002
[1006]: https://github.com/nextstrain/augur/pull/1006
[1008]: https://github.com/nextstrain/augur/pull/1008
[1017]: https://github.com/nextstrain/augur/pull/1017

Bug Fixes

* filter: Move `register_arguments` to the top of the module for better readability [995][]. (victorlin)
* filter: Fix a regression [introduced in 16.0.2](https://github.com/nextstrain/augur/commit/4859b5d70e77cc9a0bb99e741fefb29952058b71) that caused grouping with subsampled max sequences and force-included strains to fail in a data-specific way [#1000][]. (huddlej)

[995]: https://github.com/nextstrain/augur/pull/995
[1000]: https://github.com/nextstrain/augur/pull/1000

Bug Fixes

* The entropy panel was unavailable if mutations were not translated [881][]. This has been fixed by creating an additional `annotations` block in `augur ancestral` containing (nucleotide) genome annotations in the node-data [961][] (jameshadfield)
* ancestral: WARNINGs to stdout have been updated to print to stderr [961][] (jameshadfield)
* filter: Explicitly drop date/year/month columns from metadata during grouping. [967][] (victorlin)
* This fixes a bug [871][] where `augur filter` would crash with a cryptic `ValueError` if `year` and/or `month` is a custom column in the input metadata and also included in `--group-by`.
* filter: Fix duplicates that may appear in metadata when using `--include`/`--include-where` with subsampling [986][] (victorlin)

[881]: https://github.com/nextstrain/augur/issues/881
[961]: https://github.com/nextstrain/augur/pull/961
[967]: https://github.com/nextstrain/augur/pull/967
[871]: https://github.com/nextstrain/augur/issues/871
[986]: https://github.com/nextstrain/augur/pull/986

Bug Fixes

* filter: Handle errors from `filter_by_query` [942][] (victorlin)
* translate: output nuc annotation when reading from gff3 gene map [976][] (corneliusroemer)
* CI: Remove step for selecting PyPI instance [974][] (victorlin)
* CI: Add token to use GitHub CLI [958][] (victorlin)

[942]: https://github.com/nextstrain/augur/pull/942
[976]: https://github.com/nextstrain/augur/pull/976
[974]: https://github.com/nextstrain/augur/pull/974
[958]: https://github.com/nextstrain/augur/pull/958

Major Changes

* filter: Error when any group-by column is not found [933][] (victorlin)
* Check your workflows for any new errors that may arise from this.
* parse: Error on duplicates instead of silently passing [918][] (victorlin)
* Check your workflows for any new errors that may arise from this.
* utils: Remove `utils.myopen()` [926][] (victorlin)
* Use `io.open_file()` going forwards.
* Moved the following internal functions [929][], [923][] (victorlin):
* `utils.read_vcf` -> `io.read_vcf`
* `utils.run_shell_command` -> `io.run_shell_command`
* `utils.shquote` -> `io.shquote`
* `utils.ambiguous_date_to_date_range` -> `dates.ambiguous_date_to_date_range`
* `utils.is_date_ambiguous` -> `dates.is_date_ambiguous`
* `utils.get_numerical_date_from_value` -> `dates.get_numerical_date_from_value`
* `utils.get_numerical_dates` -> `dates.get_numerical_dates`
* Drop support for dict type as the first parameter [934][]
* `filter.write_vcf` -> `io.write_vcf`

Features

* Add the measurements subcommand with two sub-subcommands, export and concat [879][] (joverlee521)
* filter: Report min and max date separately [930][] (victorlin)
* export v2: Allow the color scale type to be temporal [969][] (jameshadfield)
* Handle `FileNotFoundError` and unexpected exceptions gracefully [914][] (victorlin)

Bug Fixes

* filter: Properly handle error on duplicates [918][] (victorlin)
* filter: Reorganize Cram test files [943][] (victorlin)
* filter: Reword comment on vcftools [924][] (victorlin)
* io: Split io.py into smaller files under new io/ [949][] (victorlin)
* io: Add tests for `io.open_file()` [926][] (victorlin)
* Move AugurError to new errors.py, replace RuntimeError [921][] (victorlin)
* Remove internal usage of `utils.read_metadata()` [934][], [972][] (victorlin)
* schemas: Add missing display_default properties for Auspice config v2 [916][] (tsibley)
* CI: Split codecov into separate job, combine coverage of all matrix jobs [968][] (tsibley)
* CI: Temporarily disable failing test [962][] (victorlin)
* CI: pip install without editable mode [956][] (victorlin)
* CI: Include functional tests in code coverage [899][] (huddlej)
* CI: Move --quiet flag to accommodate snakemake=7.7.0 behavior [927][] (victorlin)
* CI: Move docker rebuild step to release workflow [912][] (victorlin)
* Update release process [913][] (victorlin)

[968]: https://github.com/nextstrain/augur/pull/968
[949]: https://github.com/nextstrain/augur/pull/949
[879]: https://github.com/nextstrain/augur/pull/879
[962]: https://github.com/nextstrain/augur/pull/962
[956]: https://github.com/nextstrain/augur/pull/956
[921]: https://github.com/nextstrain/augur/pull/921
[943]: https://github.com/nextstrain/augur/pull/943
[929]: https://github.com/nextstrain/augur/pull/929
[934]: https://github.com/nextstrain/augur/pull/934
[930]: https://github.com/nextstrain/augur/pull/930
[926]: https://github.com/nextstrain/augur/pull/926
[933]: https://github.com/nextstrain/augur/pull/933
[899]: https://github.com/nextstrain/augur/pull/899
[923]: https://github.com/nextstrain/augur/pull/923
[914]: https://github.com/nextstrain/augur/pull/914
[913]: https://github.com/nextstrain/augur/pull/913
[927]: https://github.com/nextstrain/augur/pull/927
[924]: https://github.com/nextstrain/augur/pull/924
[916]: https://github.com/nextstrain/augur/pull/916
[918]: https://github.com/nextstrain/augur/pull/918
[912]: https://github.com/nextstrain/augur/pull/912
[969]: https://github.com/nextstrain/augur/pull/969
[972]: https://github.com/nextstrain/augur/pull/972