Cellxgene-census

Changes

Minor updates in preparation for releasing December LTS Census and contributed embeddings.

Deprecations

* [python] cellxgene_census.experimental.util.X_sparse_iter is deprecated and will be removed in a future release. Please use the new SOMA blockwise iterators available on the SparseNDMatrix class.

What's Changed
* [builder] Update release cleanup code to support relative URIs by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/888
* [python] Geneformer WDL: pin TileDB and Census versions by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/889
* [misc] contrib embedding curation fixes by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/890
* [r] Update R docs in preparation for API release 1.9.1 by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/893


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.9.0...v1.9.1

Changes
* R and Python package dependencies updated to use TileDB-SOMA version 1.6.0

Deprecations
* [python] `cellxgene_census.experimental.util.X_sparse_iter` is deprecated and will be removed in a future release. Please use the new SOMA `blockwise` iterators available on the SparseNDMatrix class.

What's Changed
* [misc] Bump aiohttp from 3.8.6 to 3.9.0 in /tools/cellxgene_census_builder by dependabot in https://github.com/chanzuckerberg/cellxgene-census/pull/868
* [builder] Absolute uris should use the default mirror by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/863
* [python] GeneformerTokenizer maintenance by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/861
* [python] migrate GeneformerTokenizer to blockwise iterator by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/866
* [python] [r] Fix tests that had the mirror name hardcoded by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/869
* [builder] Fix builder release manifest validation to not check for absolute uri by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/882
* [python] upgrade to tiledbsoma and add obsm to get_anndata by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/885
* [misc] Contrib embeddings by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/853
* [r] add obsm_layers to get_seurat() and get_single_cell_experiment() by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/887
* [python] Geneformer fine-tuning scripts by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/805
* [python] scVI pipeline by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/787


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.8.0...v1.9.0

Changes

* R and Python package dependencies updated to use TileDB-SOMA version 1.5.1

Deprecations

* [python] `cellxgene_census.experimental.util.X_sparse_iter` is deprecated as of this release and will be removed in a future release. Please use the new SOMA `blockwise` iterators available on the SparseNDMatrix class.

What's Changed

* [misc] update GHA action and pre-commit versions by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/829
* [misc] fix installation of geneformer's transitive dependency in py-dependency-check.yml by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/833
* [misc] update actions/stale by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/834
* [misc] prep for 1.8 release by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/852
* [python] update hvg acceptance tests to account for numerical instability by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/838
* [builder] Sync whole bucket by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/819
* [builder] Acceptance/Builder tests misc fixes by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/855
* [docs] Update bucket name in AWS Open Data docpage by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/851
* [docs] DOC: Update cellxgene_census_aws_open_data.md typo by kamilkrukowski in https://github.com/chanzuckerberg/cellxgene-census/pull/858
* [r] add r-dependency-check.yml nightly test of current release package by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/769
* [r] Fix acceptance tests by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/859

New Contributors
* kamilkrukowski made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/858

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.7.0...v1.8.0

Additions

* [python] add support for distributed PyTorch by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/699
* [python] Add `X_layers` argument to `cellxgene_census.get_anndata()` by pablo-
* [python] Geneformer Dataset builder by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/762
* [python][misc] GHA for Python dependency checks by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/763
* [python] GeneformerTokenizer speed optimizations by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/784
* [python] add multi-batch-key support to highly-variable gene finder by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/788
* [python] Add exclude_zeros to pp.mean_variance by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/731
* [python] Add pytorch global shuffling by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/775
* [docs] Add "what's new?" article for out-of-core methods by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/765
* [docs] Add "what's new?" article for out-of-core methods by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/765
* [docs] Create newsletter_september2023.md by MaximilianLombardo in https://github.com/chanzuckerberg/cellxgene-census/pull/732
* [docs] Adds normalized layer banner to landing page by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/808
* [docs] editorial changes to october news by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/810
* [docs] Add documentation for AWS by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/823
* [builder] Add additional debug logging by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/770
* [misc] Add authentication to py-dependency-check workflow by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/768
* [misc] LTS / package compatibility check by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/797


Fixes

* [python] Fix import in example for geneformer tokenizer by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/780
* [python] remove unnecessary dependency pin by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/802
* [python] temporarily skip geneformer tests by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/818
* [python] add PEP 561 py.typed marker by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/822
* [python] Fix HVG crasher by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/821
* [docs] Fix minor typo in blog post about out-of-core functionalities by aaronwolen in https://github.com/chanzuckerberg/cellxgene-census/pull/774
* [docs] Fix minor typo in blog post about out-of-core functionalities by
* [docs] fix index docs for AWS page by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/825
* [docs] fix typos AWS page by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/826
* [builder] thread usage tuning by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/759
* [builder] Fix additional builder logging by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/771
gar in https://github.com/chanzuckerberg/cellxgene-census/pull/761
* [builder] perform all consolidation modes by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/773
* [builder] Restore h5ad replication by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/783
* [misc] shorten Census LTS compatibility workflow name by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/803

What's Changed

* [python] limit runtime of --expensive unit tests by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/779
* [python][builder] Explicit LTS releases in releases.json by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/737
* [r] Bump r version to 1.7.0 by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/828
* [builder] use TileDB core 2.16 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/800
* [builder] update tiledbsoma pin to 1.4.4 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/812

New Contributors
* aaronwolen made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/774
* MaximilianLombardo made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/732

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.6.0...v1.7.0

Additions

Census APIs now support access to data mirrors, geographical data mirrors are coming soon!

* [python] Mirrors support by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/552
* [r] support mirrors & relative URI by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/719

Other:

* [r] test framework for first-time R package installation by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/687
* [r] more vignettes by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/680
* [r][docs] minor docs updates by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/696
* [builder] Explicit log message tests by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/720
* [builder] add builder dmesg logging on error by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/745
* [docs] Add normalized layer information to user-facing schema overview by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/753
* [docs] Add markdown linting to pre-commit by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/721

Fixes

* [python] numerical precision improvements in Census builder by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/716
* [python] revert Mac M1/M2 cert fix on open_soma() by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/746
* [docs] Add `SingleCellExperiment` mention to landing page. by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/725
* [docs] pin Sphinx version by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/729
* [docs] add explicit filtering for None git tags by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/712
* [docs] fix markdown lint by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/735
* [builder] Resolve CI failure caused by X count matrix with zero-sum rows by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/742

What's Changed
* [python] get_census_version_directory returns sorted elements by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/728
* [python] Bump tiledbsoma pinned version to 1.4.3 by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/738
* [python] Pin `tiledbsoma~=1.4.3` in Python API by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/756
* [r][docs] Update version to 1.5.2 and doc-site by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/743
* [r][docs] Bump R API version to 1.6.0 by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/757
* [docs][python] Reinforce knowledge of `is_primary_data` in all notebooks by pablo-gar in
* [docs] Update API release process to support both R and Python releases by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/739
* [builder][docs] update to CxG schema version 3.1.0 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/740
https://github.com/chanzuckerberg/cellxgene-census/pull/754
* Bump gitpython from 3.1.31 to 3.1.32 in /tools/cellxgene_census_builder by dependabot in https://github.com/chanzuckerberg/cellxgene-census/pull/713
* feat: Add a Github Action that automatically closes stale PRs. by maniarathi in https://github.com/chanzuckerberg/cellxgene-census/pull/724
* fix: override default so that issues are not marked as stale by maniarathi in https://github.com/chanzuckerberg/cellxgene-census/pull/730

New Contributors
* maniarathi made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/724

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.5.1...v1.6.0

What's Changed
* [docs] Add "What's new page" to doc-site and R V1 news article (671) by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/671
* [docs] Update publish date on new article 20230808-r_api_release.md by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/701
* [r] Fixes source links in R doc site by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/702
* [python] update tiledbsoma version to 1.4.0 by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/709
* [docs] Add R tutorials link to landing page by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/704
* [r] Bump R package version to 1.5.1 for release by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/710


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.5.0...v1.5.1