Virtualizarr

This release brings a stricter internal model for manifest paths, support for appending to existing icechunk stores, an experimental non-kerchunk-based HDF5 reader, handling of nested groups in DMR++ files, as well as many other bugfixes and documentation improvements.

What's Changed
* FAQ updates by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/266
* Import top-level version of xarray classes by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/267
* Update index.md by thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/275
* Remove unused ManifestBackendArray class by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/282
* Fix bug in RT of parquet detection by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/278
* Search for coord_names in separate_coords by ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/191
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/283
* Bump minimum Xarray dependency to 2024.10.0 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/284
* Dont write _ARRAY_DIMENSIONS to icechunk by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/286
* Fix release notes for v1.1.0 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/288
* dmrpp root and nested group parsing fix by ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/265
* Update README.md by joshmoore in https://github.com/zarr-developers/VirtualiZarr/pull/294
* Remove numcodecs specific install by mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/301
* Update contributors guide by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/298
* Clarify which features are currently available in FAQ by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/296
* Fix sphinx warnings by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/300
* Update pkg install in docs contribution guide by douglatornell in https://github.com/zarr-developers/VirtualiZarr/pull/304
* Support downstream type checking by including py.typed by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/306
* Non-kerchunk backend for HDF5/netcdf4 files. by sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/87
* Release note for 87 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/307
* Add status badges to README by douglatornell in https://github.com/zarr-developers/VirtualiZarr/pull/303
* Expand xarray openable type hint with ReadBuffer by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/316
* Consolidate hdf reader tests into their own tests module by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/314
* Refactor kerchunk reader tests to call open_virtual_dataset by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/317
* Add virtual_backend_kwargs argument to open_virtual_dataset by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/315
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/322
* Refactor dmrpp tests to expose data file path by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/323
* Paths as URIs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/243
* Add list of previous talks to readme by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/313
* Correct some documentation links to function the public API by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/325
* Ignore spontaneous mypy errors for h5py classes. ref 324 by sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/328
* Append to icechunk stores by abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/272
* Add new page with links to example notebooks by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/331
* Release summary for v1.2.0 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/332

New Contributors
* joshmoore made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/294
* douglatornell made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/304
* sharkinsspatial made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/87

**Full Changelog**: https://github.com/zarr-developers/VirtualiZarr/compare/v1.1.0...v1.2.0

This release adds [Icechunk](https://icechunk.io/) support!!

It also brings a complete refactoring of the system of readers and writers internally, which allowed us to make Kerchunk an optional dependency. There are also many other bugfixes and smaller improvements.

What's Changed
* xr.testing with ManifestArray fix (update isnan ufunc) by ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/188
* Clarify that virtualizarr is a user-level replacement for kerchunk by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/192
* Exclude empty `paths` on `ChunkDict` creation by ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/198
* Extend refspec support to [path] entries (without offset/length) by maresb in https://github.com/zarr-developers/VirtualiZarr/pull/187
* Conformant ZarrV3 codecs and fill values by ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/193
* https access fix by ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/196
* Handle scalar dataset variables by ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/205
* Set ZArray default fill_value as NaT for datetime64 by thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/206
* Update .pre-commit-config mypy + bump ruff version by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/211
* Update static typing by TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/213
* Adds concurrency to CI w/ cancel-in-progress=True by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/214
* Implement pydantic models as dataclasses by TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/210
* use the theme options for `pydata_sphinx_theme` by keewis in https://github.com/zarr-developers/VirtualiZarr/pull/223
* Removes default storage options by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/228
* open_virtual_dataset with dmr++ by ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/113
* Internal refactor to separate reading and writing concerns by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/231
* Let Xarray handle `decode_times` by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/232
* Support specifying single HDF Group in open_virtual_dataset by scottyhq in https://github.com/zarr-developers/VirtualiZarr/pull/165
* Adds defaults in `open_virtual_dataset_from_v3_store` by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/234
* Virtualizarr + Coiled Serverless Example Notebook by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/233
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/236
* Add example to create a virtual dataset using lithops by thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/203
* Update backend.py (tiny typo) by mdsumner in https://github.com/zarr-developers/VirtualiZarr/pull/240
* Makes mypy a seperate CI job by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/254
* Fix mypy errors around numpy functions not being strictly type hinted by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/252
* Allow `open_virtual_dataset` to read existing Kerchunk references by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/251
* Skip tests that require kerchunk by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/259
* allow creating references for empty archival datasets by keewis in https://github.com/zarr-developers/VirtualiZarr/pull/260
* Split kerchunk reader up by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/261
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/250
* Add CI job for testing upstream versions of dependencies by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/264
* Add Icechunk Support by mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/256

New Contributors
* ayushnag made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/188
* ghidalgo3 made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/198
* maresb made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/187
* thodson-usgs made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/206
* keewis made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/223
* mdsumner made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/240
* mpiannucci made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/256

**Full Changelog**: https://github.com/zarr-developers/VirtualiZarr/compare/v1.0.0...v1.1.0

New Contributors
* scottyhq made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/143
* moradology made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/177

**Full Changelog**: https://github.com/zarr-developers/VirtualiZarr/compare/v0.1.0...v1.0

This release marks VirtualiZarr as mostly feature-complete, in the sense of achieving feature parity with kerchunk's logic for combining datasets, providing an easier way to manipulate kerchunk references in memory and generate kerchunk reference files on disk.

Future VirtualiZarr development will focus on generalizing and upstreaming useful concepts into the Zarr specification, the Zarr-Python library, Xarray, and possibly some new packages. See the roadmap in the documentation for details.

What's Changed
* Hypothesis test broadcasting by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/139
* Empty release notes for v0.2 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/145
* Mark tests which require network access by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/144
* Install dependencies for tests via mamba by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/148
* Use default version scheme for setuptools_scm by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/149
* Use 3 numpy arrays for manifest internally by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/107
* Rename paths in manifest by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/152
* Ensure _ARRAY_DIMENSIONS get dropped from attrs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/153
* Ensure attributes on coordinate variables are preserved during round-tripping by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/154
* Identify non dimension coords by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/156
* Also test exporting references to in-memory kerchunk reference dict by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/158
* Use magic bytes to identify file formats by scottyhq in https://github.com/zarr-developers/VirtualiZarr/pull/143
* Decoding `cftime_variables` by jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/122
* Fix opening tiff and fits files by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/162
* Clarify that virtual datasets are not normal xarray datasets by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/173
* Warn on index creation by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/170
* Update roadmap for v1.0 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/164
* Add example of using cftime_variables to usage docs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/174
* Future-proof offset and size records in chunkmanifest by moradology in https://github.com/zarr-developers/VirtualiZarr/pull/177
* Use a set to avoid duplicate var names from kerchunk by moradology in https://github.com/zarr-developers/VirtualiZarr/pull/179

The first release of VirtualiZarr!

This release presents the basic MVP of this library, including the ability to inspect netCDF4/HDF5 files, store the byte ranges in an `xarray.Dataset` via `ManifestArray` objects, concatenate those objects, then serialize the result to disk as kerchunk-formatted reference files.

Expect more features and significant optimizations soon.

What's Changed

* Xarray accessor to create kerchunk reference dict by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/28
* Equality checking by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/30
* Support xarray concat, including broadcasting by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/34
* CI for running tests by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/36
* Roughout of Sphinx Docs by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/27
* Updated doc.yml to include pip by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/40
* Adds netCDF3 vs netCDF4 distinction to _automatically_determine_filetype. by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/43
* Test concat of dimension coordinate not backed by an index by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/44
* Rename open_dataset_via_kerchunk to open_virtual_dataset by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/47
* Narrative docs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/48
* More narrative docs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/50
* Update CI checks with ruff by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/54
* open_virtual_dataset with and without indexes by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/52
* API docs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/56
* Updated NetCDF IO path by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/55
* Created conftest.py and moved two fixtures into conftest by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/57
* Ab/filters dtype by abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/66
* Switching netcdf3 & netcdf4 filetype detection to file magic 🧙 by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/64
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/70
* Load selected variables instead of making them virtual by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/69
* pin_kerchunk_0.2.2 by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/75
* Remove python 3.12 from CI matrix by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/76
* Convert user defined filetype to FileType by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/79
* FAQ page by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/81
* Try to remove sidebar in docs by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/82
* `main.yml` CI installs from pyproject.toml by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/90
* Update installation instructions by jbusecke in https://github.com/zarr-developers/VirtualiZarr/pull/91
* Write manifests to zarr store by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/45
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/101
* Bump Ruff version and add formatting by jbusecke in https://github.com/zarr-developers/VirtualiZarr/pull/98
* Opening 0D scalars by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/102
* Fix bug with expand dims of a scalar array by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/103
* Install xarray from main by jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/106
* Unpin kerchunk (set floor) and enable Python 3.12 by jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/108
* Depend on latest version of xarray by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/109
* Remove python 3.9 by jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/112
* Adding `reader_options` kwargs to open_virtual_dataset. by norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/67
* Write to parquet by jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/110
* Test fsspec roundtrip by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/42
* Inline loaded variables into kerchunk references by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/73
* Release notes page by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/120
* requires-python = ">=3.10" by abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/127
* Pass args and add test by abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/128
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/129
* Allow other fsspec protocols than local and s3 by TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/126
* Add dunder version to top-level __init__.py by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/133
* Add release workflow by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/136
* Only run distribution workflow on releases by maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/140
* change default reader_options to None by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/137
* Replace np.NaN with np.nan in preparation for numpy 2.0 by TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/138

New Contributors
* TomNicholas made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/28
* abarciauskas-bgse made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/66
* pre-commit-ci made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/70
* jbusecke made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/91
* jsignell made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/106
* TomAugspurger made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/126
* maxrjones made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/133

**Full Changelog**: https://github.com/zarr-developers/VirtualiZarr/commits/v0.1