Tiledb

Disk Format

* (documentation) The storage format specification was updated to document format changes of previous versions throughout the main document. [5329](https://github.com/TileDB-Inc/TileDB/pull/5329)

Breaking behavior

* Deleting an array or group will also delete its root directory if it was left empty. [5269](https://github.com/TileDB-Inc/TileDB/pull/5269)

New features

* Dumping fragment info now includes the name of each fragment's array schema. [5257](https://github.com/TileDB-Inc/TileDB/pull/5257)
* Added support for the `ssl.ca_path` config option on Azure. [5286](https://github.com/TileDB-Inc/TileDB/pull/5286)
* Connections to TileDB REST API now use TCP keepalive by default. [5319](https://github.com/TileDB-Inc/TileDB/pull/5319) and curl retries [#5273](https://github.com/TileDB-Inc/TileDB/pull/5273).
* Added support for "current domain" on dense arrays. [5303](https://github.com/TileDB-Inc/TileDB/pull/5303)

Configuration Options

* Add `vfs.log_operations` config option to log in trace level all VFS operations and their duration. [5278](https://github.com/TileDB-Inc/TileDB/pull/5278)
* The default value of `vfs.s3.region` was changed to empty, which will let the AWS pick the default region from sources such as environment variables, profile configuration, and instance metadata. [5317](https://github.com/TileDB-Inc/TileDB/pull/5317)
* Add `rest.curl.tcp_keepalive` config option that controls using TCP keepalive for TileDB REST connections. It is enabled by default. [5319](https://github.com/TileDB-Inc/TileDB/pull/5319)

Improvements

* Fix read queries on sparse arrays where only aggregates are requested and no layout is specified. [5255](https://github.com/TileDB-Inc/TileDB/pull/5255)
* Prevent potentially unsafe consolidation in dense arrays by fragment list. [5251](https://github.com/TileDB-Inc/TileDB/pull/5251)
* Improve error messages for unsupported data types in aggregates. [5289](https://github.com/TileDB-Inc/TileDB/pull/5289)
* Remove StorageManager from stats dumps. [5297](https://github.com/TileDB-Inc/TileDB/pull/5297)
* Precompute memory usage for tile offsets and error if not loadable due to memory limits. [5310](https://github.com/TileDB-Inc/TileDB/pull/5310)
* GCS object composition temporary files now end with `.tiledb.tmp`, which may be used to set a lifecycle rule. [5372](https://github.com/TileDB-Inc/TileDB/pull/5372)
* Updated libmagic to version 5.45. [5332](https://github.com/TileDB-Inc/TileDB/pull/5332)
* Linux aarch64 release [5271](https://github.com/TileDB-Inc/TileDB/pull/5271)

Deprecations

* Deprecate `tiledb_filestore_*` APIs. [5371](https://github.com/TileDB-Inc/TileDB/pull/5371)

Defects removed

* Fix an incompatibility with S3 Express One Zone, by stopping setting `Content-MD5` on all S3 uploads. [5226](https://github.com/TileDB-Inc/TileDB/pull/5226)
* Update range end when merging overlapping subarray ranges. [5268](https://github.com/TileDB-Inc/TileDB/pull/5268)
* Fixed GZip compression of empty data. [5296](https://github.com/TileDB-Inc/TileDB/pull/5296)
* Fix HTTP requests for AWS default credentials provider chain not honoring config options. [5315](https://github.com/TileDB-Inc/TileDB/pull/5315)
* Fix dense reader fill value with `sm.var_offsets.mode = elements` [5331](https://github.com/TileDB-Inc/TileDB/pull/5331)
* Schema evolution bug fix: Reads no longer fail after dropping a fixed attribute and adding it back as var-sized. [5321](https://github.com/TileDB-Inc/TileDB/pull/5321)
* Fixed the `tiledb_ctx_alloc_with_error` function not being exported from the library. [5357](https://github.com/TileDB-Inc/TileDB/pull/5357)
* Set increased traversal limit in enumeration deserialization [5365](https://github.com/TileDB-Inc/TileDB/pull/5365)
* Fix heap corruption when getting context and query stats from C++ under certain circumstances. [5366](https://github.com/TileDB-Inc/TileDB/pull/5366)
* Schema evolution bug fix: Reads no longer fail after dropping a fixed attribute and adding it back as var-sized, part 2. [5362](https://github.com/TileDB-Inc/TileDB/pull/5362)
* Remove defective test case that's failing in nightly builds. [5369](https://github.com/TileDB-Inc/TileDB/pull/5369)
* Follow up on 5359 [5368](https://github.com/TileDB-Inc/TileDB/pull/5368)

API changes

C API

* Add `tiledb_array_schema_get_enumeration` API. [5359](https://github.com/TileDB-Inc/TileDB/pull/5359)
* Add `tiledb_group_add_member_with_type` API to get group member with type in single call. [5336](https://github.com/TileDB-Inc/TileDB/pull/5336)
* Introduce tiledb_array_load_enumerations_all_schemas. [5349](https://github.com/TileDB-Inc/TileDB/pull/5349)
* Added `tiledb_group_dump_str_v2` API that returns a `tiledb_string_t*`. The existing `tiledb_group_dump_str` API is deprecated. [5367](https://github.com/TileDB-Inc/TileDB/pull/5367)
* Add `tiledb_fragment_info_dump_str` C API and replace `FragmentInfo::dump(FILE*)` with `operator<<` overload. [5266](https://github.com/TileDB-Inc/TileDB/pull/5266)

C++ API

* Fix error log messages when using the `Array` class in the C++ API. [5262](https://github.com/TileDB-Inc/TileDB/pull/5262)
* Add overload to the `Array::create` function that explicitly accepts a context. [5325](https://github.com/TileDB-Inc/TileDB/pull/5325)
* Update `Group::add_member` API to accept an optional type. [5336](https://github.com/TileDB-Inc/TileDB/pull/5336)
* Introduce `ArrayExperimental::load_enumerations_all_schemas`. [5349](https://github.com/TileDB-Inc/TileDB/pull/5349)

Build System Changes

* Update vcpkg version baseline to https://github.com/microsoft/vcpkg/commit/101cc9a69a1061969caf4b73579a34873fdd60fe. [#5010](https://github.com/TileDB-Inc/TileDB/pull/5010)
* Fixed cross-compiling support in the libmagic vcpkg port overlay. [5333](https://github.com/TileDB-Inc/TileDB/pull/5333)
* The `TILEDB_CCACHE` option was fixed to have effect, after being accidentally disabled in version 2.26.0. [5342](https://github.com/TileDB-Inc/TileDB/pull/5342)
* Fixed double installation of a header when serialization is enabled. [5354](https://github.com/TileDB-Inc/TileDB/pull/5354)

Defects removed

* Fix HTTP requests for AWS default credentials provider chain not honoring config options. [5318](https://github.com/TileDB-Inc/TileDB/pull/5318)

Improvements

* Fail early on dense reads when tile offsets are too large. [5311](https://github.com/TileDB-Inc/TileDB/pull/5311)

Breaking behavior

* The superbuild architecture of the build system has been removed and TileDB is now a single-level CMake project. Build commands of the form `make && make -C tiledb <targets>` will have to be replaced by `make <targets>`. [5021](https://github.com/TileDB-Inc/TileDB/pull/5021)

Breaking C API changes

* All deprecated C and C++ APIs were removed. [5146](https://github.com/TileDB-Inc/TileDB/pull/5146)

New features

* Add support for getting the datatype of a ndrectangle dimension. [5229](https://github.com/TileDB-Inc/TileDB/pull/5229)
* Add dim num support for ndrectangle. [5230](https://github.com/TileDB-Inc/TileDB/pull/5230)

Improvements

* Added new functions to write schema dump to a string rather than stdout, so that non-console interactive environments such as Jupyter will be able to capture and print the output. [5026](https://github.com/TileDB-Inc/TileDB/pull/5026)
* Improve dense read performance for small reads. [5145](https://github.com/TileDB-Inc/TileDB/pull/5145)
* Skip caching redirect uri on array create. [5224](https://github.com/TileDB-Inc/TileDB/pull/5224)
* Enable curl error retries. [5275](https://github.com/TileDB-Inc/TileDB/pull/5275)

Deprecations

* Warn users using dense arrays with sparse fragments. [5116](https://github.com/TileDB-Inc/TileDB/pull/5116)

Defects removed

* Fix fragment consolidation to allow using absolute URIs. [5135](https://github.com/TileDB-Inc/TileDB/pull/5135)
* Reset offsets in buffer list for retries. [5220](https://github.com/TileDB-Inc/TileDB/pull/5220)
* Fix symbol clashes between `tiledb` and `pyarrow` by building the AWS SDK with its internal symbols hidden. [5223](https://github.com/TileDB-Inc/TileDB/pull/5223)

Configuration changes

* The `sm.use_refactored_readers` config option is no longer recognized. Refactored readers are used by default. To use the legacy reader, set the `sm.query_(dense|sparse_global_order|sparse_unordered_with_dups)_reader` config option (depending on the reader you are using) to `legacy`. [5183](https://github.com/TileDB-Inc/TileDB/pull/5183)

API changes

C API

* Add ctx to CurrentDomain CAPI. [5219](https://github.com/TileDB-Inc/TileDB/pull/5219)
* Add new CAPIs to dump array schema, attribute, dimension, domain, enumeration and group to a string. [5026](https://github.com/TileDB-Inc/TileDB/pull/5026)

Build System Changes

* The version of OpenSSL linked to the release artifacts was updated to 3.1.4, and moving forward will be synced to the vcpkg `builtin-baseline`. [5174](https://github.com/TileDB-Inc/TileDB/pull/5174)

Announcements

* TileDB 2.25, includes the new current domain feature which allows to specify an area of the domain that is considered to be active for sparse arrays.

Deprecation announcements

* The HDFS backend is no longer officially tested by TileDB. As announced before, it is scheduled to be removed in version 2.28, to be released in Q4 2024. [5085](https://github.com/TileDB-Inc/TileDB/pull/5085)
* Support for reading sparse fragments in dense arrays will be removed in version 2.27. Writting sparse fragments in dense arrays was removed in version 2.5. [5116](https://github.com/TileDB-Inc/TileDB/pull/5116)
* Support for returning the same results multiple times in sparse reads when ranges overlap will be removed in version 2.27. This was possible by setting `sm.merge_overlapping_ranges_experimental` to `false`, but the default `true` behavior has been there since version 2.17.

New features

* REST support for current domain. [5136](https://github.com/TileDB-Inc/TileDB/pull/5136)
* Disallow writing outside of the current domain. [5165](https://github.com/TileDB-Inc/TileDB/pull/5165)
* Current domain: disallow reading outside of current domain. [5168](https://github.com/TileDB-Inc/TileDB/pull/5168)

Improvements

* Improve memory consumption for tile structures in dense reader. [5046](https://github.com/TileDB-Inc/TileDB/pull/5046)

Defects removed

* Fail early when trying to add members with relative URIs in remote groups. [5025](https://github.com/TileDB-Inc/TileDB/pull/5025)
* Correct defective return value in `Posix::ls_with_sizes`. [5037](https://github.com/TileDB-Inc/TileDB/pull/5037)
* Prevent constructing attribute with invalid cell_val_num. [4952](https://github.com/TileDB-Inc/TileDB/pull/4952)
* Do not mask failures when listing a directory fails on POSIX. [5043](https://github.com/TileDB-Inc/TileDB/pull/5043)
* Fix write queries using `sm.var_offsets.extra_element=true`. [5033](https://github.com/TileDB-Inc/TileDB/pull/5033)
* Fix segfaults in WebP queries ran in parallel. [5065](https://github.com/TileDB-Inc/TileDB/pull/5065)
* Fix exceptions with message: unknown exception type; no further information. [5080](https://github.com/TileDB-Inc/TileDB/pull/5080)
* Fix check for out of bounds dimension in Dimension::dimension_ptr. [5094](https://github.com/TileDB-Inc/TileDB/pull/5094)
* Fix array latest schema selection for same MS timestamps schemas. [5143](https://github.com/TileDB-Inc/TileDB/pull/5143)
* Fix serialization issue with schema evolution for query v3. [5154](https://github.com/TileDB-Inc/TileDB/pull/5154)

Configuration changes

* Add `vfs.s3.storage_class` config option to set the storage class of newly uploaded S3 objects. [5053](https://github.com/TileDB-Inc/TileDB/pull/5053)
* Add `rest.custom_headers.*` config option to set custom headers on REST requests. [5104](https://github.com/TileDB-Inc/TileDB/pull/5104)
* Add `rest.payer_namespace` config option to set the namespace to be charged for REST requests. [5105](https://github.com/TileDB-Inc/TileDB/pull/5105)

API changes

C API

* Add CurrentDomain API support. [5041](https://github.com/TileDB-Inc/TileDB/pull/5041)

C++ API

* Current Domain CPP API implementation. [5056](https://github.com/TileDB-Inc/TileDB/pull/5056)

Build System Changes

* Backwards compatibility with older CMake versions for libfaketime. [5049](https://github.com/TileDB-Inc/TileDB/pull/5049)
* Automatic downloading of vcpkg can be disabled by enabling the `TILEDB_DISABLE_AUTO_VCPKG` CMake option, in addition to setting the environment variable with trhe same name. [5048](https://github.com/TileDB-Inc/TileDB/pull/5048)
* Improve embedding of `magic.mgc` and allow compiling with any libmagic version. [4989](https://github.com/TileDB-Inc/TileDB/pull/4989)

Internal Improvements

* Implement actualize function that orders data underlying `alt_var_length_view` [5087](https://github.com/TileDB-Inc/TileDB/pull/5087)
* Implement a partitioning function to partition cells to fit into fixed size bins [5092](https://github.com/TileDB-Inc/TileDB/pull/5092)
* Implementation of a `chunk_view` class to provide a subset of C++23 chunk_view, suitable for supporting external sort. [5035](https://github.com/TileDB-Inc/TileDB/pull/5035)
* Tests that the chunks in a `chunk_view` can be separately sorted. [5052](https://github.com/TileDB-Inc/TileDB/pull/5052)

Defects removed

* Fix serialization issue with schema evolution for query v3. [5154](https://github.com/TileDB-Inc/TileDB/pull/5154)