Tiledbsoma

New features added

* [r] Additional matrix readers for NDArrays by eddelbuettel in https://github.com/single-cell-data/TileDB-SOMA/pull/720
* [python] Make add-X-layer logic friendlier by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/793

Breaking changes

* [python] Stateful (open/close) objects by mlin in https://github.com/single-cell-data/TileDB-SOMA/pull/730

Minor changes

* [python] Replace flake8 & isort with ruff by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/751
* [r] Prevent one test script from failing under `rcmdcheck` by eddelbuettel in https://github.com/single-cell-data/TileDB-SOMA/pull/796
* [python] Create an isort (now ruff) config at the root of the repository by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/782
* [r] Allow use pbmc3k data package in test by eddelbuettel in https://github.com/single-cell-data/TileDB-SOMA/pull/798
* More unit tests, plus partial fix for 801 by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/802
* [python] Propagate context rather than parent.context by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/803
* [python] Do more is-Arrow checks on create and write by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/797
* Import somacore v0.0.0a7, without making any functional changes. by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/808
* [python] Pin to specific TileDB core major+minor by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/807

Bug fixes

* [python] Fix pip-install instructions by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/795

**Full Changelog**: https://github.com/single-cell-data/TileDB-SOMA/compare/0.5.0a6...0.5.0a7

New features added

* [python] Enable Unicode non-indexed columns by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/777

Minor changes

* [r/python] Rename sidf to sdf [trivial] by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/776
* [python] Change _CachedElement._TdbInfo.type from str to type by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/775
* [python] A bit more deadstrip by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/781
* [python] Enhance value filter unit tests by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/785
* [python] Additional experiment-query unit tests by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/789
* [python] refactor SOMAReader construction into TileDBArray base helper method by mlin in https://github.com/single-cell-data/TileDB-SOMA/pull/787
* Simplify to_tiledb_supported_array_type by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/790

Bug fixes

* [python] Undo AWS_DEFAULT_REGION workaround by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/771

New features added

* [r/python/c++] Depend on core 2.14, tiledb-r 0.18, tiledb-py 0.20 by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/770

Breaking changes

* [python] Depend on somacore 0.0.0a5 by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/765

Minor changes

* [python] Add pre-commit hook for mypy and remove explicit mypy step from python-ci by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/743
* Remove pre-main spec file by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/758
* [python] Set up caching for packages downloaded with Pip by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/757
* [r/python/c++] Expand tabs that are immediately following spaces by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/754
* [python] Do not export QueryCondition by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/759
* [python] Remove ndims/ndim temp mod, given somacore 0.0.0a4 by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/762
* [python] Cache builds of libtiledbsoma in Python CI by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/760
* [python] Cache pre-commit environments by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/756
* [python] Replace bare `assert` with typed exceptions by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/746
* [python] Add test for to_anndata column_names by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/763
* [python] Add GHA workflow for minimal CI jobs by atolopko-czi in https://github.com/single-cell-data/TileDB-SOMA/pull/750
* [python] Add pandas-stubs for mypy and fix uncovered typing issues by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/772
* [python] Remove unnecessary steps from lint job by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/767
* [python] Add test case for issue 768 -- existence of `ExperimentAxisQuery.get_indexer()` by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/769
* [r] Simple R UTF-8 test by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/773

**Full Changelog**: https://github.com/single-cell-data/TileDB-SOMA/compare/0.5.0a4...0.5.0a5

Breaking changes

* [python] Import somacore by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/678
* [python] Update to version 0.0.0a3 of somacore. by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/732
* [r] `ids` to `coords` as read args by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/748
* [python] Add SomaSessionContext to all SOMA objects by atolopko-czi in https://github.com/single-cell-data/TileDB-SOMA/pull/681
* [python] Prep for ExperimentAxisQuery perf work in `somacore` by bkmartinjr in https://github.com/single-cell-data/TileDB-SOMA/pull/745

New features added

* [r] Add SOMAMeasurement and SOMAExperiment classes by aaronwolen in https://github.com/single-cell-data/TileDB-SOMA/pull/729

Minor changes

* [ci/python] Don't build wheels on each PR push by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/697
* [ci/python] Include missing doc link in wheels YAML by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/703
* [python] build wheels only for CPython (not PyPy) to save CI time by mlin in https://github.com/single-cell-data/TileDB-SOMA/pull/702
* [ci/python] Dedupe PyPI stubs by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/704
* Apply black/isort/flake8/mypy to all repo .py files by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/711
* Add pull_request_template.md by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/712
* Add quote marks around a few strings in C++ errors. by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/721
* [python] Update pyarrow dependency by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/714
* [python] Separate pip installation of prereqs and tiledbsoma itself. by thetorpedodog in https://github.com/single-cell-data/TileDB-SOMA/pull/717
* [python] Stronger type checking by gsakkis in https://github.com/single-cell-data/TileDB-SOMA/pull/726
* Add CZI to the license and modify to include 2023 by maniarathi in https://github.com/single-cell-data/TileDB-SOMA/pull/727
* [python] Audit for bare Exception by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/742
* [python] Take ndim accessor from somacore by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/747

Bug fixes

* [python] Correct incorrect object-type metadata on storage by johnkerl in https://github.com/single-cell-data/TileDB-SOMA/pull/709

Upcoming deprecation notices

New Contributors

* maniarathi made their first contribution in https://github.com/single-cell-data/TileDB-SOMA/pull/727
* atolopko-czi made their first contribution in https://github.com/single-cell-data/TileDB-SOMA/pull/681

**Full Changelog**: https://github.com/single-cell-data/TileDB-SOMA/compare/0.1.20...0.5.0a4

Breaking changes

Major query improvement:

* [python] ExperimentAxisQuery by bkmartinjr in 632

Improved names for `read` methods:

* [python] Fluent API changes for read methods by bkmartinjr in 663

New features added

Minor query improvements:

* [c++] Support empty range query by gspowley in 618
* [c++] Support empty range query and Arrow nullable fields by gspowley in 622
* [python] Support indexing by [] -> empty results in Python by johnkerl in 623
* [python] DataFrame count accessor by johnkerl in 637
* [python] Add len(dataframe) by johnkerl in 649

Ingestion-UX improvements:

* [python] Less-in-core ingest from H5AD [main] by johnkerl in 640
* [python] Resume mode [main] by johnkerl in 664
* [python] More memory-reduction for ingest [main] by johnkerl in 682

Minor changes

* [r] Add SOMACollection class by aaronwolen in 606
* [r] Rename NdArray -> NDArray by johnkerl in 609
* [python] Shorten class constants using `Final` typing feature by thetorpedodog in 616
* [c++] Add clang-format check to CI by gspowley in 617
* Add wiki link to README.md by johnkerl in 619
* [c++] Add SOMAReader reset for stateful API changes by gspowley in 624
* [r/ci] Fix and re-enable CI macOS builds by aaronwolen in 629
* [python] Update to setuptools 65.5.1 [main] by johnkerl in 630
* [r] Adjust SparseNDArray test to renamed soma_data method by eddelbuettel in 634
* Python 3.7 CI by johnkerl in 620
* [r] Regularize test-file naming [main] by johnkerl in 553
* [python] Add some docstrings by johnkerl in 635
* [python] Deadstrip some old code by johnkerl in 647
* [r] Correct handling of soma_joinid and soma_rowid by aaronwolen in 642
* [r] Enhancements to SparseNDArray by eddelbuettel in 645
* [python] Temp pin to TileDB-Py 0.19.0 by johnkerl in 651
* [python] Typofix by johnkerl in 648
* [python] Fixing typing_extensions dependency by johnkerl in 650
* [python] setup.* `typing-extensions` vs `import typing_extensions` by johnkerl in 655
* [python] Remove _indent by johnkerl in 658
* [python] Trivial io-module function-reorder by johnkerl in 662
* [python] Complete basic anndata I/O tests by johnkerl in 667
* [python] Use RNA not mRNA in examples/tests by johnkerl in 665
* [python] Simplify setup by gsakkis in 674
* [python/ci] Fix setup.py to play well with cibuildwheel by gsakkis in 677
* [python] Trivial comments for wheel-build by johnkerl in 676
* [python] Prep for import somacore by johnkerl in 673
* [python] Remove duplicate `test_general_utilities.py` file. by thetorpedodog in 680
* [python] Add a few hints so mypy knows what type things are. by thetorpedodog in 683
* Remove bdist_egg command & unused names from setup.py by gsakkis in 691
* [python] Bump versions of precommit hooks by gsakkis in 692
* [r] Additional sparse test code by eddelbuettel in 690
* [ci] Use paths-ignore to avoid R CI for Python commits and vice versa by johnkerl in 694

Bug fixes

* [python] Honor AWS_DEFAULT_REGION if no overriding ctx is specified by johnkerl in 625
* [r] Fix storage of int64 values by SOMADataFrame by aaronwolen in 639
* [python] Do not use extent 2048 on 8-bit index types by johnkerl in 653
* [r] Fix creation of SOMADataFrames with non-int32 dimensions by aaronwolen in 657
* [python] Unbreak editable install from clean by johnkerl in 689

Upcoming deprecation notices

* Experiment-level `open` and `close`, including open-at-timestamp support
* `ids` in some methods vs `coords` in others to be aligned (686)
* Add explicit `context` to the API (644)

**Full Changelog**: https://github.com/single-cell-data/TileDB-SOMA/compare/0.5.0a2...0.5.0a3

This is for development on the `main` branch, in contrast to 0.1.x releases which track development on the `main-old` branch.

Ongoing development

* NdArray refinement - tranche 4 by bkmartinjr in 367
* [python] Conform to spec by reading as pyarrow.Table not pyarrow.RecordBatch by johnkerl in 355
* Correctly handle empty query results in SparseNdArray read operations by bkmartinjr in 371
* Initial SOMADataFrame class by aaronwolen in 377
* Fix for SOMAIndexedDataFrames with float dims by aaronwolen in 379
* Initial SOMAIndexedDataFrame class by aaronwolen in 378
* Prevent word splitting path in `bld` by aaronwolen in 387
* [r] Enhance support for (partial) table extraction by eddelbuettel in 389
* Python collections refinement by bkmartinjr in 381
* [python] Use true ASCII attributes in dataframes by johnkerl in 359
* [r] Support Query Condition objects, small other polishes by eddelbuettel in 398
* Refine soma_ column handling in Python API by bkmartinjr in 397
* [r] Add basic unit testing for SOMAReader interface by eddelbuettel in 401
* [c++] Add fast nnz by gspowley in 404
* fix index in pandas read all by bkmartinjr in 408
* [python] Disambiguate bytes/ascii on tiledb schema -> arrow schema by johnkerl in 406
* rename type to soma_type by bkmartinjr in 409
* [python] Match SparseNdArray dim/attr names to spec by johnkerl in 411
* [c++] Expose two functions from managed_query in soma_reader by eddelbuettel in 412
* add unit tests for issues 415, 418, 419 by bkmartinjr in 420
* [r] Provide nnz() accessor by eddelbuettel in 422
* [c++,r] Simplify spdlog setup for use from R by eddelbuettel in 421
* Refactor from_anndata support by bkmartinjr in 416
* Use C++17 to ease build requirements by eddelbuettel in 417
* [python] Disallow duplicates [main branch] by johnkerl in 437
* [r] Add support for row selection by eddelbuettel in 402
* [r] Add Continuous Integration for the R package by eddelbuettel in 403
* [python/c++] Connect C++ nnz to Python by johnkerl in 439
* [c++] Revert 439 by johnkerl in 443
* [c++] Internal context-handling updates by johnkerl in 444
* [python/ci] Run Python CI on 3.8 and 3.9 by johnkerl in 445
* Correct configure by ensuring R_HOME is set early by eddelbuettel in 449
* [python] Depend on TileDB Core 2.12.0 by johnkerl in 448
* [c++] Do not require query condition columns in column_names by gspowley in 393
* [python/c++] Connect C++ nnz to Python by johnkerl in 446
* [c++] Add column name to the arrow schema by gspowley in 452
* [c++] Support passing column_names as None by gspowley in 451
* [r] Enhanced soma reader by eddelbuettel in 454
* [c++] Add obs slice X test by gspowley in 456
* [python/c++] Connect C++ reader for dataframes by johnkerl in 400
* [python] Correctly handle zero-length dataframes with SOMAReader by johnkerl in 459
* [python] More unit-test cases for 459 by johnkerl in 460
* [c++] Expose is_complete predicate from managed_query in soma reader by eddelbuettel in 461
* [python/c++] Connect C++ reader for SOMASparseNdArray by johnkerl in 447
* [r] Initial support for iterated reads from SOMAReader by eddelbuettel in 462
* [python] Unprefix SOMA names by johnkerl in 464
* [r] Minor rearrangement demoting initial test function and renaming by eddelbuettel in 468
* [python] Fix 471 by johnkerl in 472
* [python] Update to TileDB-Py 0.18.0 [main] by johnkerl in 476
* [python] Replace Exception with SOMA-specific exceptions by johnkerl in 486
* [r] Expand iterated reader via refactored row/col selectors by eddelbuettel in 480
* [r] Initial SOMADenseNDArray class by aaronwolen in 488
* [c++] Accept dim_points as pyarrow.Array by gspowley in 498
* [r] Add coverage reports in script and yaml by eddelbuettel in 504
* [python] Depend on TileDB-Py 0.18.1 [main] by johnkerl in 503
* [r] Arrow conversion utilities by aaronwolen in 505
* [r] Use libtiledbsoma.a in build by eddelbuettel in 510
* [c++] Fix macos build error by gspowley in 513
* [c++] Enable internal TileDB stats collection by gspowley in 515
* [python] Update slicing logic for SparseNdArray by johnkerl in 485
* [r] Enable code coverage token use by eddelbuettel in 521
* [c++] Improve build experience by gspowley in 522
* [python] Update SOMAIndexedDataFrame indexing logic by johnkerl in 483
* add temporary configuration to SOMAReader instances by bkmartinjr in 527
* improving IndexedDataFrame test coverage by mlin in 529
* [c++] Cmake bug fix and improvements by gspowley in 525
* [python] Remove DataFrame in favor of IndexedDataFrame by johnkerl in 538
* [r] Support logging via spdlog by eddelbuettel in 530
* [python] Remove DataFrame::is_indexed by johnkerl in 545
* [r] Remove DataFrame in favor of IndexedDataFrame by johnkerl in 537
* [c++] Release Python GIL on reads by gspowley in 550
* [r] Ignore directory with TileDB Core headers in coverage analysis by eddelbuettel in 552
* [r] Update logging code utilizing templated interface by eddelbuettel in 555
* [python] unit tests and error checking for 300 by bkmartinjr in 559
* Add the `platform_config` parameter to create methods. by thetorpedodog in 542
* [r] Extend iterated reader interface with platform_config by eddelbuettel in 556
* DataFrame read result_order by mlin in 528
* [c++] Map 8-bit TILEDB_BOOL to 1-bit Arrow boolean by bekadavis9 in 561
* [r] Simplify logging via 'spdl' package by eddelbuettel in 563
* Add documentation for `platform_config` to the Python README. by thetorpedodog in 568
* [r] Make source build more robust by eddelbuettel in 572
* [r] Tighten package by removing some initial exploratory code by eddelbuettel in 577
* Fix 578 (tile-order/cell-order typo) by johnkerl in 580
* [python] Support half-slices in indexed reads by bekadavis9 in 582
* Fix DenseNdArray slicing by johnkerl in 581
* Use C++ reader for SOMA.DenseNdArray by johnkerl in 574
* Fix flaky unit test by johnkerl in 585
* Update to TileDB 2.13 [main] by johnkerl in 587
* Support value-filter on boolean columns [pending core 2.13 / tiledb-py 0.19] by johnkerl in 579
* Un-xfail half-slice tests in DenseNdArray by johnkerl in 589
* Extend indexing-related unit-test cases for DenseNdArray by johnkerl in 588
* Use soma_reader in SOMADataFrame by eddelbuettel in 590
* [r] Convert arrow::large_utf8() to ascii by aaronwolen in 591
* Support string dims in soma.DataFrame by johnkerl in 592
* Include Python 3.10 in CI matrix by johnkerl in 597
* Use soma_reader in SOMADenseNdArray read method by eddelbuettel in 598
* [python] Add tests for point indexing with ndarray by bkmartinjr in 604
* [r] Tweak logging default value enabling easier use by eddelbuettel in 607
* [python] Rename NdArray -> NDArray by johnkerl in 608
* Support reading data that uses old NdArray name by bkmartinjr in 610
* [r] Making package R CMD check clean by eddelbuettel in 614

Continuous integration and build infrastructure

* [python] Pause PyPI publish on main by johnkerl in 369
* [ci] Work around recent Mac GHA issue by johnkerl in 370
* [ci] Fix scripts/test for MacOS by johnkerl in 358
* [python] Depend on TileDB-Py 0.17.5 by johnkerl in 394
* [r] Support build against external artifact and system library by eddelbuettel in 388
* [ci] Use MacOS 12 for CI by johnkerl in 413
* [ci] Update to checkout/actionsv3 to suppress nag about node.js 12 by eddelbuettel in 455
* Build flow improvements by gspowley in 496
* report pytest coverage to codecov by mlin in 495
* Build python sdist & wheels by mlin in 605
* py packaging: strike 'v' prefix from expected version numbers by mlin in 613

New Contributors

* mlin made their first contribution in 495
* bekadavis9 made their first contribution in 561

**Full Changelog**: https://github.com/single-cell-data/TileDB-SOMA/compare/0.5.0a1...0.5.0a2