Nextstrain-augur

Features

* export, frequencies, refine, traits: Add a new flag `--metadata-id-columns` to customize the possible metadata ID columns. Previously, this was only available in `augur filter`. [1240][] (victorlin)
* Add new sub-subcommand augur curate format-dates. The format-dates command is intended to be used to format date fields to ISO 8601 date format (YYYY-MM-DD), where incomplete dates are masked with `XX` (e.g. 2023 -> 2023-XX-XX). [1146][] (joverlee521)

Bug fixes

* parse: Fix a bug where `--fix-dates` was always applied, with a default of `--fix-dates=monthfirst`. Now, running without `--fix-dates` will leave dates as-is. [1247][] (victorlin)
* `augur.io.open_file`: Previously, the docs described a type restriction on `path_or_buffer` but it was not enforced. It has been updated to allow all I/O classes, and is enforced at run-time. [1250][] (victorlin)
* filter: Fix a bug where data files consisting of only numerical strain names would not work when both `--metadata` and `--sequences` are passed. [1256][] (victorlin)

[1146]: https://github.com/nextstrain/augur/pull/1146
[1240]: https://github.com/nextstrain/augur/pull/1240
[1247]: https://github.com/nextstrain/augur/issues/1247
[1250]: https://github.com/nextstrain/augur/pull/1250
[1256]: https://github.com/nextstrain/augur/pull/1256

Bug fixes

* utils: Serialize pandas Series in `write_json`. [1213][] (victorlin)

[1213]: https://github.com/nextstrain/augur/pull/1213

Bug fixes

* CI: Add a Github action to test augur on 8 Nextstrain pathogen workflows using example data. [1217][] (corneliusroemer)
* parse: Denote required arguments including `--fields`, `--output-sequences`, and `--output-metadata`. [1228][] (huddlej)
* Fix export of the `strand` attribute of gene annotations. Previously, features on the negative strand were not annotated as such since the code assumed that the `strand` attribute was boolean instead of `[-1, +1]`. [1211] rneher and j23414.
* augur.io.read_metadata: explicitly set `date` column as `string` type to prevent year only dates from being inferred as integers. [1235][] (joverlee521)

[1211]: https://github.com/nextstrain/augur/pull/1211
[1217]: https://github.com/nextstrain/augur/pull/1217
[1228]: https://github.com/nextstrain/augur/pull/1228
[1235]: https://github.com/nextstrain/augur/pull/1235

Bug fixes

* export: No longer export duplicate entries in the colorings array, a bug which has been present in Augur since at least v12 [719][]. [1218][] (jameshadfield)
* export: In version 22.0.0, some configurations of export may have resulted in the clade coloring appearing last in the Auspice dropdown rather than first. This is now fixed. [1218] (jameshadfield)
* export: In version 22.0.0, validation of `augur.utils.read_node_data` was changed to error when a node data JSON did not contain any actual data. This causes export to error when an empty node data JSON is passed, as for example in ncov's pathogen-ci. This is now fixed by warning instead. The bug was originally introduced in PR [728][]. [1214][] (corneliusroemer)

[719]: https://github.com/nextstrain/augur/issues/719
[1214]: https://github.com/nextstrain/augur/pull/1214
[1218]: https://github.com/nextstrain/augur/pull/1218

Major Changes

* export, filter, frequencies, refine, traits: From versions 10.0.0 through 21.1.0, arbitrary delimiters for `--metadata` were supported due to internal implementation differences from the advertised CSV and TSV support. Starting with this version, non-CSV/TSV files will no longer be supported by default. To adjust for this breaking change, specify custom delimiters with the new `--metadata-delimiters` flag. [1196][] (victorlin)
* `augur.io.read_metadata`: Previously, this supported any arbitrary delimiters for the metadata. Now, it only supports a list of possible delimiters represented by the new `delimiters` keyword argument, which defaults to `,` and `\t`. [812][] (victorlin)
* refine: The seeding method for `--seed` has been updated. This affects usages that rely on the reproducibility of outputs with the same `--seed` value prior to this version. Outputs from this version onwards should be reproducible until the next implementation change, which we don't expect to happen any time soon. [1207][] (rneher)

Features

* Constrain `bcbio-gff` to >=0.7.0 and allow `Biopython` >=1.81 again. We had to introduce the `Biopython` constraint in v21.0.1 (see [1152][]) due to `bcbio-gff` <0.7.0 relying on the removed `Biopython` feature `UnknownSeq`. [1178][] (corneliusroemer)
* `augur.io.read_metadata` (used by export, filter, frequencies, refine, and traits): Previously, this used the Python parser engine for [`pandas.read_csv()`][]. Updated to use the C engine for faster reading of metadata. [812][] (victorlin)
* curate: Allow custom metadata delimiters with the new `--metadata-delimiters` flag. [1196][] (victorlin)
* Bump the default recursion limit to 10,000. Users can continue to override this limit with the environment variable `AUGUR_RECURSION_LIMIT`. [1200][] (joverlee521)
* clades, export v2: Clade labels + coloring keys are now definable via arguments to augur clades allowing pipelines to use multiple invocations of augur clades resulting in multiple sets of colors and branch labels. How labels are stored in the (intermediate) node-data JSON files has changed. This should be fully backwards compatible for pipelines using augur commands, however custom scripts may need updating. PR [728][] (jameshadfield)
* refine: add flag `--max-iter` to control the maximal number of iterations TreeTime uses to infer time trees. This was previously hard-coded to 2, which is now the default. [1203][] (rneher)
* refine: add flags `--greedy-resolve` and `--stochastic-resolve` to customize polytomy resolution. [1203][], [1207][] (rneher)
* `--greedy-resolve`: resolve polytomies by greedily minimizing tree length (default behavior, unchanged).
* `--stochastic-resolve`: resolve polytomies as random coalescent trees.
* These are mutually exclusive with the pre-existing `--keep-polytomies` flag.

Bug fixes

* filter, frequencies, refine, parse: Previously, ambiguous dates in the future had a limit of today's date imposed on the upper value but not the lower value. It is now imposed on the lower value as well. [1171][] (victorlin)
* refine: `--year-bounds` was ignored in versions 9.0.0 through 20.0.0. It now works. [1136][] (victorlin)
* tree: Input alignment filenames which do not end in `.fasta` are now properly handled when using IQ-TREE. Previously their contents were overwritten first by `augur tree` itself (resulting in truncation) and then by the log output of IQ-TREE (resulting in an error). Thanks to Jon Bråte for reporting this bug. [1206][] (tsibley)
* clades: A number of small bug fixes, improvements to documentation, tests and improved error detection. [1199][] (jameshadfield)

[728]: https://github.com/nextstrain/augur/pull/728
[812]: https://github.com/nextstrain/augur/pull/812
[1136]: https://github.com/nextstrain/augur/issues/1136
[1152]: https://github.com/nextstrain/augur/pull/1152
[1171]: https://github.com/nextstrain/augur/issues/1171
[1178]: https://github.com/nextstrain/augur/pull/1178
[1196]: https://github.com/nextstrain/augur/pull/1196
[1199]: https://github.com/nextstrain/augur/pull/1199
[1200]: https://github.com/nextstrain/augur/pull/1200
[1203]: https://github.com/nextstrain/augur/pull/1203
[1206]: https://github.com/nextstrain/augur/pull/1206
[1207]: https://github.com/nextstrain/augur/pull/1207
[`pandas.read_csv()`]: https://pandas.pydata.org/pandas-docs/version/1.5/reference/api/pandas.read_csv.html

Features

* filter: Add `--empty-output-reporting={error,warn,silent}` option to allow filter to produce empty outputs without raising an error. The default behavior is still to raise an error when filter produces an empty output, so users will have to explicitly pass the "warn" or "silent" value to bypass the error. [1175][] (joverlee521)

Bug fixes

* translate: Fix error handling when features cannot be read from reference sequence file. [1168][] (victorlin)
* translate: Remove an unnecessary check which allowed for inaccurate error messages to be shown. [1169][] (victorlin)
* frequencies: Previously, monthly pivot points calculated from the end of a month may have been shifted by 1-3 days. This is now fixed. [1150][] (victorlin)
* docs: Fix minor formatting issues. [1095][] (victorlin)
* Update development status on PyPI from "3 - Alpha" to "5 - Production/Stable". This should have been done since the beginning of this changelog, but now it is official. [1160][] (corneliusroemer)

[1095]: https://github.com/nextstrain/augur/pull/1095
[1150]: https://github.com/nextstrain/augur/pull/1150
[1160]: https://github.com/nextstrain/augur/pull/1160
[1168]: https://github.com/nextstrain/augur/pull/1168
[1169]: https://github.com/nextstrain/augur/pull/1169
[1175]: https://github.com/nextstrain/augur/pull/1175