Biom-format

----------

New features and support for Pandas >= 1.0, released on 5 November 2020.

Important:

* Cython and numpy are no more required to be present before building, see [PR 840](https://github.com/biocore/biom-format/pull/840)

New Features:

* Added support for the AnnData format, see [PR 845](https://github.com/biocore/biom-format/pull/845)
* Performance boost to `Table.remove_empty`. For large tables this cuts the running time from 20 seconds to ~1.1 seconds, see [PR 847](https://github.com/biocore/biom-format/pull/847)
* A much faster way to merge tables (without metadata) has been added. For large tables, this was a few minutes rather than a few hours. This method is implicitly invoked when calling `Table.merge` if unioning both axes, and the tables lack metadata. `Table.concat` is still much faster, but assumes one axis is disjoint. See [PR 848](https://github.com/biocore/biom-format/pull/848).

* Simplify interaction with the concatenation method, allowing for passing in an individual table and support for a general `biom.concat(tables)` wrapper. See [PR 851](https://github.com/biocore/biom-format/pull/851).
* Added support for parsing adjacency table structures, see [issue 823](https://github.com/biocore/biom-format/issues/823).

Bug fixes:

* Support for pandas >= 1.0, see the comment and commits [here](https://github.com/biocore/biom-format/issues/837#issuecomment-721241751)

----------

New features and bug fixes, released on 28 January 2020.

Important:

* Python 2.7 and 3.5 support has been dropped.
* Python 3.8 support has been added into Travis CI.
* A change to the defaults for `Table.nonzero_counts` was performed such that the default now is to count the number of nonzero features. See [issue 685](https://github.com/biocore/biom-format/issues/685)
* We now require a SciPy >= 1.3.1. See [issue 816](https://github.com/biocore/biom-format/issues/816)

New Features:

* The detailed report is no longer part of the table validator. See [issue 378](https://github.com/biocore/biom-format/issues/378).
* `load_table` now accepts open file handles. See [issue 481](https://github.com/biocore/biom-format/issues/481).
* `biom export-metadata` has been added to export metadata as TSV. See [issue 820](https://github.com/biocore/biom-format/issues/820).
* `Table.to_tsv` has been modified to allow for `direct_io`. See [issue 836](https://github.com/biocore/biom-format/pull/836).

Bug fixes:

* `Table.to_dataframe(dense=False)` does now correctly produce sparse data frames (and not accidentally dense ones as before). See [issue 808](https://github.com/biocore/biom-format/issues/808).
* Order of error evaluations was unstable in Python versions without implicit `OrderedDict`. See [issue 813](https://github.com/biocore/biom-format/issues/813). Thanks gwarmstrong for identifying this bug.
* `Table._extract_data_from_tsv` would fail if taxonomy was provided, and if the first row had the empty string for taxonomy. See [issue 827](https://github.com/biocore/biom-format/issues/827). Thanks KasperSkytte for identifying this bug.

----------

New features and bug fixes, released on 28 September 2018.

Important:

* Python 3.4 support has been dropped. We now only support Python 2.7, 3.5, 3.6 and 3.7.
* We will be dropping Python 2.7 support on the next release.
* Pandas >= 0.20.0 is now the minimum required version.
* pytest is now used instead of nose.

New Features:

* Massive performance boost to `Table.collapse` with the default collapse function. The difference was 10s of milliseconds vs. minutes stemming from prior use of `operator.add`. See [issue 761](https://github.com/biocore/biom-format/issues/761).
* `Table.align_to` for aligning one table to another. This is useful in multi-omic analyses where multiple preparations have been performed on the sample physical samples. This is essentially a helper method around `Table.sort_order`. See [issue 747](https://github.com/biocore/biom-format/issues/747).
* Added additional sanity checks when calling `Table.to_hdf5`, see [PR 769](https://github.com/biocore/biom-format/pull/769).
* `Table.subsample()` can optionally perform subsampling with replacement. See [issue 774](https://github.com/biocore/biom-format/issues/774).
* `Table.to_dataframe()` now supports a `dense` argument to return `pd.DataFrame`. See [issue 762](https://github.com/biocore/biom-format/issues/762).
* Parsing methods for BIOM-Format 1.0.0 tables now preserve dict ordering. See [issue 781](https://github.com/biocore/biom-format/issues/781).

Bug fixes:

* `Table.subsample(by_id=True, axis='observation')` did not subsample over the 'observations'. Because of the nature of the bug, an empty table was returned, so the scope of the issue is such that it should not have produced misleading results but instead triggered empty table errors, with the exception of the pathological case of the ID namespaces between features and samples not being disjoint. See [PR 759](https://github.com/biocore/biom-format/pull/759) for more information.
* Tables of shape `(0, n)` or `(n, 0)` were raising exceptions when being written out. See [issue 619](https://github.com/biocore/biom-format/issues/619).
* Tables loaded with a `list` of empty `dict`s will have their metadata attributes set to None. See [issue 594](https://github.com/biocore/biom-format/issues/594).

----------

New features and bug fixes, released on 27 April 2017.

New Features:

* `Table.from_hdf5` now supports a rapid subset in the event that metadata is
not needed. In benchmarking against the Earth Microbiome Project BIOM table,
the reduction in runtime was multiple orders of magnitude while additionally
preserving substantial memory.
* `Table.rankdata` has been added to convert values to ranked abundances on
either axis. See [issue 645](https://github.com/biocore/biom-format/issues/639).
* Format of numbers in ``biom summarize-table`` output is now more readable and localized. See [issue 679](https://github.com/biocore/biom-format/issues/679).
* `Table.concat` has been added to the API and allows for concatenating multiple tables in which the IDs of one of the axes are known to be disjoint. This has substantial performance benefits over `Table.merge`.
* `Table.sort_order` was performing an implicit cast to dense, and not leveraging fancy indexing. A substantial performance gain was acheived. See [PR 720](https://github.com/biocore/biom-format/pull/720).
* `biom subset-table` now accepts a QIIME-like mapping file when subsetting by IDs [Issue 587](https://github.com/biocore/biom-format/issues/587)
* `Table.del_metadata` was added to support the removal of metadata entries from the table [Issue 708](https://github.com/biocore/biom-format/issues/708).
* `Table.to_dataframe` was added to cast the internal matrix data to a Pandas `SparseDataFrame` [Issue 622](https://github.com/biocore/biom-format/issues/622).
* `Table.metadata_to_dataframe` was added to cast axis metadata to a Pandas `DataFrame` [Issue 622](https://github.com/biocore/biom-format/issues/622).
* `test_table.py` and `test_util.py` now use a stable random seed. See issue [728](https://github.com/biocore/biom-format/issues/728)
* Failure to cast a value when parsing a TSV will now print the associated line number which had the bad value. See [284](https://github.com/biocore/biom-format/issues/284).
* `Table.remove_empty` has been added to remove zero'd samples, observations or both. See [721](https://github.com/biocore/biom-format/issues/721).
* A subcommand of the command line interface was added to obtain a table's IDs: `table-ids`.

Bug fixes:

* ``-o`` is now a required parameter of ``biom from-uc``. This was not the case previously, which resulted in a cryptic error message if ``-o`` was not provided. See [issue 683](https://github.com/biocore/biom-format/issues/683).
* Matrices are now cast to csr on `Table` construction if the data evaluate as `isspmatrix`. This fixes [717](https://github.com/biocore/biom-format/issues/717) where some API methods assumed the data were csc or csr.
* `Table.concat` was not handling tables without metadata, resulting in an exception due to mismatches metadata shape. See [724](https://github.com/biocore/biom-format/issues/724).
* When validating a BIOM-Format 1.0.0 table, specifying the version string would trigger an error. See [664](https://github.com/biocore/biom-format/issues/664). An explicit regression test was not added as this stemmed from an integration, and there currently is not support for script usage tests; see [#656](https://github.com/biocore/biom-format/issues/656).
* `Table.nnz` was not calling `eliminate_zeros()` on the underlying sparse matrix, resulting in the possibility of counting explicitly set zero values. See [727](https://github.com/biocore/biom-format/issues/727).
* `Table.from_hdf5` was not properly turning `bytes` into `str` for the `table_id` and the `type` HDF5 attributes. See [731](https://github.com/biocore/biom-format/issues/731).
* `Table.__init__` now always performs an `astype(float)` on the contained `spmatrix`. This type normalization is beneficial for underlying Cython code on the filtering and transform operations. It is possible this will introduce some performance overhead, however in _most_ cases the data should already be float. See [718](https://github.com/biocore/biom-format/issues/718).
* `Table.to_hdf5` was not handling lists of str appropriately in the general case. Ssee [638](https://github.com/biocore/biom-format/issues/638).
* `Table.to_hdf5` was not handling taxonomy as flat strings, which was a common mistake that was outside of expectations for the formatter. The formatter now attempts to split on semicolon if this scenario is encountered, and errors with a more informative error if a problem occurs. See [530](https://github.com/biocore/biom-format/issues/530).

----------

New features and bug fixes, released on 21 October 2015.

Changes:

* Codebase is now Python 2/3 compatible. It is currently tested with Python
versions 2.7, 3.4 and 3.5.
* `biom-serve` and the accompanying html interface has been removed.

New Features:

* `Table.head` has been added to retrieve the first few rows and or columns
from a table. This can be accessed through the new ``biom head`` command.
See [issue 639](https://github.com/biocore/biom-format/issues/639).
* ``biom.parse.from_uc`` has been added to support creation of ``biom.Table``
objects from vsearch/uclust/usearch ``.uc`` files. This can be accessed
through the new ``biom from-uc`` command. See
[issue 648](https://github.com/biocore/biom-format/issues/648).
* Codebase now uses [click](http://click.pocoo.org) instead of
[pyqi](https://github.com/biocore/pyqi) for its command line interface.
See [issue 631](https://github.com/biocore/biom-format/issues/631).

Bug fixes:

* `Table.update_ids` strict check was too aggressive. See
[issue 633](https://github.com/biocore/biom-format/issues/633).
* `biom --version` now prints the software version (previously the individual
commands did this, but not the base command).
* `Table.vlen_list_of_str_formatter` was considering a `str` to be valid for
formatting resulting in an obscure error when a `str`, as opposed to a
`list` of `str`, was used for taxonomy. See
[issue 709](https://github.com/biocore/biom-format/issues/709).

----------

Bug fixes, released on April 22nd 2015

Changes:

* Codebase updated to reflect pep8 1.6.x

New features:

* `Table.to_hdf5` and `Table.from_hdf5` now support custom parsers and
formatters, see issue 608

Bug fixes:

* `Table.update_ids` was not updating the internal ID lookup caches, issue 599
* `--is-json` has been removed from the table validator as it was being ignored
anyway, issue 591
* `biom summarize-table` can now properly interact with pipes. This previously
worked on OSX but did not on Linux. Issue 598
* `biom convert` was recording the wrong version information from HDF5 -> JSON,
issue 595
* `Table.collapse`, under `one_to_many` was not constructing the resulting
matrix properly, issue 606
* Improve error message when trying to load an empty file, issue 614.
* Improve error handling when filtering tables, and return tables of shape
`(0, n)` instead of `(0, 0)` when fully filtering out a table along an
axis, issue 620
* Fix `Table.nonzero` to work on data that is not already in csr, issue 625.