Cellxgene-census

The Census builder is now using [Census schema 2.0.0](https://github.com/chanzuckerberg/cellxgene-census/blob/main/docs/cellxgene_census_schema.md) which in turn uses [CELLxGENE dataset schema version 5.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/5.0.0/schema.md). All [Census data releases](https://github.com/chanzuckerberg/cellxgene-census/blob/main/docs/cellxgene_census_docsite_data_release_info.md) starting on 2024-03-26 will adhere to this schema.

* Update to require [CELLxGENE schema version 5.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/5.0.0/schema.md). **Includes breaking changes**.
* Expanded list of assays included in the Census.
* Expanded the list of assays defined as full-gene sequencing assays, which have special normalized layer handling.
* Clarified handling of datasets which are multi-species on the obs or var axis.

Additions
* [builder] CxG schema 5 / Census schema 2 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1024

What's Changed
* [misc] 'native cell' -> 'unknown' on geneformer label_blocklist by danieljhegeman in https://github.com/chanzuckerberg/cellxgene-census/pull/999
* [misc] Prepare for 1.12.0 release by prathapsridharan in https://github.com/chanzuckerberg/cellxgene-census/pull/1061

New Contributors
* danieljhegeman made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/999

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.11.1...v1.12.0

The Census is now using [CELLxGENE schema version 4.0.0](https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/4.0.0/schema.md).

Additions

* [docs] Adds biological demonstration notebooks by atarashansky in https://github.com/chanzuckerberg/cellxgene-census/pull/875
* [docs] Add citation notebooks by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/916
* [builder] schema 4.0 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/872

Breaking changes

* [python] remove deprecated X_sparse_iter API by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/962

What’s Changed

* [python] [R] upgrade tiledbsoma to 1.7.0 by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/969
* [docs] add `no-sign-request` flag to model download in tutorials by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/956
* [builder] remove unused H5AD files from build by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/946
* [python] Syntactic sugar for specifying tiledb_config options by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/902
* [python] Make default soma context available for customization by atolopko-czi in https://github.com/chanzuckerberg/cellxgene-census/pull/895

New Contributors
* atarashansky made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/875
* seve made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/846

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.9.1...v1.10.0

New embeddings API
Census embeddings can now accessed using a new, simplified API. Check the notebooks for [collaboration](https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_access_maintained_embeddings.ipynb) and [hosted](https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_embedding.ipynb) models for more information.

obs columns are now categorical instead of strings
Starting from the `2024-04-01` Census build, a subset of the columns in the `obs` dataframe are now categorical instead of strings.

For **Python users**, note that Pandas will encode these columns as `pandas.Categorical` for which some downstream operations may need to be adapted. See [this link](https://pandas.pydata.org/docs/user_guide/categorical.html#operations) for more details. In particular:

> Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data

and

> DataFrame methods like sum, groupby, pivot, value_counts also show “unused” categories when observed=False, which is the default.

For **R users**, note that these columns will be encoded as `factor` and similarly downstream operations may need to be adapted. See [this link](https://r4ds.had.co.nz/factors.html) for more details.

For **Python and R users** interfacing with `arrow`, these columns will be encoded as `dictionary`, see more details for R in [this link](https://arrow.apache.org/docs/r/reference/dictionary.html) and Python in [this link](https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html).

Additions
* [builder] enable Arrow Dictionary feature flag by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1064
* [python] New embeddings API by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1023
* [docs] New embeddings API notebooks by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1070


**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.12.0...v1.13.0

Changes
* Fix r-universe build

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.11.0...v1.11.1

Changes
R and Python package dependencies updated to use TileDB-SOMA version 1.8.0

What's Changed
* [misc] Fix CL_0002494 in geneformer pipeline by pablo-gar in https://github.com/chanzuckerberg/cellxgene-census/pull/1000
* [docs] Add embeddings API to docs by martinkim0 in https://github.com/chanzuckerberg/cellxgene-census/pull/1005
* [docs] Fix docstrings for various functions by martinkim0 in https://github.com/chanzuckerberg/cellxgene-census/pull/1002
* [builder] fix incorrect filtering of tissue type by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1010
* [misc] add support for var embeddings to contrib curation tool by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/995
* [builder] add explicit validation check for primary tissue cells by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1015
* [builder] update uberon and tissue_general mapping to schema 4 standard by bkmartinjr in https://github.com/chanzuckerberg/cellxgene-census/pull/1018
* [docs] Update copyright notice by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/978
* [docs] misc docsite adjustments by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1020
* [profiler] Early version of profiler harness by beroy in https://github.com/chanzuckerberg/cellxgene-census/pull/959
* [docs] Geneformer docs errata by mlin in https://github.com/chanzuckerberg/cellxgene-census/pull/1003
* [python] [r] Upgrade to tiledbsoma 1.8.0 by ebezzi in https://github.com/chanzuckerberg/cellxgene-census/pull/1046

New Contributors
* beroy made their first contribution in https://github.com/chanzuckerberg/cellxgene-census/pull/959

**Full Changelog**: https://github.com/chanzuckerberg/cellxgene-census/compare/v1.10.2...v1.11.0