Cellxgene-census

The Census builder is now using [Census schema 2.0.0](https://github.com/chanzuckerberg/cellxgene-census/blob/main/docs/cellxgene_census_schema.md) which in turn uses [CELLxGENE dataset schema version 5.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/5.0.0/schema.md). All [Census data releases](https://github.com/chanzuckerberg/cellxgene-census/blob/main/docs/cellxgene_census_docsite_data_release_info.md) starting on 2024-03-26 will adhere to this schema.

* Update to require [CELLxGENE schema version 5.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/5.0.0/schema.md). **Includes breaking changes**.
* Expanded list of assays included in the Census.
* Expanded the list of assays defined as full-gene sequencing assays, which have special normalized layer handling.
* Clarified handling of datasets which are multi-species on the obs or var axis.

Additions
* [builder] CxG schema 5 / Census schema 2 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1024

What's Changed
* [misc] 'native cell' -> 'unknown' on geneformer label_blocklist by danieljhegeman in https://github.com/chanzuckerberg/cellxgene-census/pull/999
* [misc] Prepare for 1.12.0 release by prathapsridharan in https://github.com/chanzuckerberg/cellxgene-census/pull/1061

New Contributors
* danieljhegeman made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/999

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.11.1...v1.12.0

The Census is now using [CELLxGENE schema version 4.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/4.0.0/schema.md).

Additions

* [docs] Adds biological demonstration notebooks by atarashansky in https://github.com/chanzuckerberg/cellxgene-census/pull/875
* [docs] Add citation notebooks by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/916
* [builder] schema 4.0 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/872

Breaking changes

* [python] remove deprecated X_sparse_iter API by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/962

What’s Changed

* [python] [R] upgrade tiledbsoma to 1.7.0 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/969
* [docs] add `no-sign-request` flag to model download in tutorials by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/956
* [builder] remove unused H5AD files from build by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/946
* [python] Syntactic sugar for specifying tiledb_config options by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/902
* [python] Make default soma context available for customization by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/895

New Contributors
* atarashansky made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/875
* seve made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/846

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.9.1...v1.10.0

Changes
This release pins `pytorch` to 2.2.0 to work around an incompatibility with `torchdata` introduced by the latest `pytorch` release. Therefore, if you're using the Census pytorch loaders and you're seeing dependency issues, you should upgrade to this version.

What's Changed
* [docs] add news article for categoricals by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/1086
* [docs] Add 2024 articles to docsite by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/1088
* [misc] Remove MacOS support claim for census builder by prathapsridharan in https://github.com/chanzuckerberg/cellxgene-census/pull/1084
* [builder] cellxgene_ontology_guide integration by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1094
* [builder] Remove datasets from "HTAN VUMC" Collection from the blocklist by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1099
* [docs] Update embedding notebook by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/1124
* [ci] Fix torch dependency issue and pause tests on Mac/Python3.8 by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1118
* [python] Fix pytorch imports by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1129


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.13.0...v1.13.1

New embeddings API
Census embeddings can now accessed using a new, simplified API. Check the notebooks for [collaboration](https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_access_maintained_embeddings.ipynb) and [hosted](https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_embedding.ipynb) models for more information.

obs columns are now categorical instead of strings
Starting from the `2024-04-01` Census build, a subset of the columns in the `obs` dataframe are now categorical instead of strings.

For **Python users**, note that Pandas will encode these columns as `pandas.Categorical` for which some downstream operations may need to be adapted. See [this link](https://pandas.pydata.org/docs/user_guide/categorical.html#operations) for more details. In particular:

> Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data

and

> DataFrame methods like sum, groupby, pivot, value_counts also show “unused” categories when observed=False, which is the default.

For **R users**, note that these columns will be encoded as `factor` and similarly downstream operations may need to be adapted. See [this link](https://r4ds.had.co.nz/factors.html) for more details.

For **Python and R users** interfacing with `arrow`, these columns will be encoded as `dictionary`, see more details for R in [this link](https://arrow.apache.org/docs/r/reference/dictionary.html) and Python in [this link](https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html).

Additions
* [builder] enable Arrow Dictionary feature flag by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1064
* [python] New embeddings API by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1023
* [docs] New embeddings API notebooks by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1070


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.12.0...v1.13.0

Changes
* Fix r-universe build

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.11.0...v1.11.1