Nextstrain-augur

Major Changes

- Drop support for Python 3.8. [1693] (victorlin)
- Drop support for older versions of jsonschema (<4.18.0). [1691] (victorlin)
- Drop support for xopen <2.0.0. [1692] (victorlin)

Bug fixes

- export: validation will no longer crash with `KeyError: 'tree'` when newer versions of jsonschema (≥4.18.0) are installed. [1358] (victorlin)

[1358]: https://github.com/nextstrain/augur/issues/1358
[1691]: https://github.com/nextstrain/augur/pull/1691
[1692]: https://github.com/nextstrain/augur/pull/1692
[1693]: https://github.com/nextstrain/augur/pull/1693

Features

- This is the first version to officially support Python 3.12 and Pandas v2. [1671] [1678] (corneliusroemer, victorlin)
- curate: change output metadata to [RFC 4180 CSV-like TSVs][] to match the TSV format output by other Augur subcommands and the Nextstrain ecosystem as discussed in [1566][]. [1565][] (joverlee521)

[1565]: https://github.com/nextstrain/augur/pull/1565
[1566]: https://github.com/nextstrain/augur/issues/1566
[RFC 4180 CSV-like TSVs]: https://datatracker.ietf.org/doc/html/rfc4180
[1671]: https://github.com/nextstrain/augur/pull/1671
[1678]: https://github.com/nextstrain/augur/pull/1678

Features

* ancestral, translate: Add `--skip-validation` as an alias to `--validation-mode=skip`. [1656][] (victorlin)
* clades: Allow customizing the validation of input node data JSON files with `--validation-mode` and `--skip-validation`. [1656][] (victorlin)
* tree: When using iqtree, check for all synonyms of default args when detecting potential conflicts, e.g. `--threads-max` is equivalent to `-ntmax`. Previously, we were only checking for the latter. Also use new, preferred IQtree2 option names (e.g. `--polytomy` instead of `-czb` etc.). [1547][] (corneliusroemer)

Bug Fixes

* index: Previously specifying a directory that does not exist in the path to `--output` would result in an incorrect error stating that the input file does not exist. It now shows the correct path responsible for the error. [1644][] (victorlin)
* curate format-dates: Update help docs and improve failure messages to show use of `--expected-date-formats`. [1653][] (joverlee521)
* parse: fix test failure with pandas 2.2. [1471] (emollier)

[1471]: https://github.com/nextstrain/augur/pull/1471
[1644]: https://github.com/nextstrain/augur/issues/1644
[1547]: https://github.com/nextstrain/augur/pull/1547
[1653]: https://github.com/nextstrain/augur/pull/1653
[1656]: https://github.com/nextstrain/augur/pull/1656

Major Changes

* filter: Duplicate header names in the FASTA file (`--sequences`) will now result in an error. [1613] (victorlin)
* parse: When both `strain` and `name` fields are present, the `strain` field will now be used as the sequence ID field. [1629][] (victorlin)
* merge: Generated source columns (e.g. `__source_metadata_{NAME}`) are now omitted by default. They may be explicitly included with `--source-columns=TEMPLATE` or explicitly omitted with `--no-source-columns`. This may be a breaking change for any existing uses of `augur merge` relying on the generated columns, though as `augur merge` is relatively new we believe usage to be scant if extant at all. [1625][] [1632][] (tsibley)

Bug Fixes

* filter: Previously, when `--subsample-max-sequences` was slightly lower than the number of groups, it was possible to fail with an uncaught `AssertionError`. Internal calculations have been adjusted to prevent this from happening. [1588][] [1598][] (victorlin)

[1588]: https://github.com/nextstrain/augur/issues/1588
[1598]: https://github.com/nextstrain/augur/issues/1598
[1613]: https://github.com/nextstrain/augur/pull/1613
[1625]: https://github.com/nextstrain/augur/issues/1625
[1629]: https://github.com/nextstrain/augur/pull/1629
[1632]: https://github.com/nextstrain/augur/issues/1632

Features

* merge: Table-specific id columns and delimiters may now be specified, e.g. `--metadata-id-columns X=id Y=strain` and `--metadata-delimiters X=, Y=';'`, to allow more precise behaviour and avoid ordering issues. [1594][] (tsibley)

Bug Fixes

* filter: Improved warning and error messages in the case of missing columns. [1604] (victorlin)
* merge: Any user-customized `~/.sqliterc` file is now ignored so it doesn't break `augur merge`'s internal use of SQLite. [1608][] (tsibley)
* merge: Non-id columns in metadata inputs that would conflict with the output id column are now forbidden and will cause an error if present. Previously they would overwrite values in the output id column, causing incorrect output. [1593][] (tsibley)
* import: Spaces in BEAST MCC tree annotations (for example, from a discrete state reconstruction) no longer break `augur import beast`'s parsing. [1610][] (watronfire)

[1593]: https://github.com/nextstrain/augur/pull/1593
[1594]: https://github.com/nextstrain/augur/pull/1594
[1604]: https://github.com/nextstrain/augur/pull/1604
[1608]: https://github.com/nextstrain/augur/pull/1608
[1610]: https://github.com/nextstrain/augur/pull/1610

Features

* A new command, `augur merge`, now allows for generalized merging of two or more metadata tables. [1563][] (tsibley)
* Two new commands, `augur read-file` and `augur write-file`, now allow external programs to do i/o like Augur by piping from/to these new commands. They provide handling of compression formats and newlines consistent with the rest of Augur. [1562][] (tsibley)
* A new debugging mode can be enabled by setting the `AUGUR_DEBUG` environment variable to `1` (or any non-empty value). Currently the only effect is to print more information about handled (i.e. anticipated) errors. For example, stack traces and parent exceptions in an exception chain are normally omitted for handled errors, but setting this env var includes them. Future debugging and troubleshooting features, like verbose operation logging, will likely also condition on this new debugging mode. [1577][] (tsibley)
* filter: Added the ability to use weights in subsampling. See help text of `--group-by-weights` and the updated [Filtering and Subsampling guide][] for more information. [1454][] (victorlin)

Bug Fixes

* Embedded newlines in quoted field values of metadata files read/written by many commands, annotation files read by `augur curate apply-record-annotations`, and index files written by `augur index` are now properly handled. [1561][] [1564][] (tsibley)
* Output written to stderr (e.g. informational messages, warnings, errors, etc.) is now always line-buffered regardless of the Python version in use. This helps with interleaved stderr and stdout. Previously, stderr was block-buffered on Python 3.8 and line-buffered on 3.9 and higher. [1563][] (tsibley)

[1454]: https://github.com/nextstrain/augur/pull/1454
[1561]: https://github.com/nextstrain/augur/pull/1561
[1562]: https://github.com/nextstrain/augur/pull/1562
[1563]: https://github.com/nextstrain/augur/pull/1563
[1564]: https://github.com/nextstrain/augur/pull/1564
[1577]: https://github.com/nextstrain/augur/pull/1577
[Filtering and Subsampling guide]: https://docs.nextstrain.org/en/latest/guides/bioinformatics/filtering-and-subsampling.html