Delta-sharing

We’d like to announce the release of Delta Sharing 1.0.5, which introduces the following improvements.

- Add queryID in userAgent for spark delta sharing queries. (481)
- Add json predicate hints to load_as_pandas. (480)
- [Streaming] Lower the minimum streaming rpc interval to 10s (479)
- Handle ConnectionClosedException in place in DeltaSharingRestClient.getResponse. (475)
- Do not retry on ConnectionClosedException (470)
- Adds more logging in DeltaSharingFileSystem (468)

Credits: Abhijit Chakankar, Lin Zhou

We’d like to announce the release of Delta Sharing 1.0.4, which introduces the following improvements.

- Supports http proxy in accessing DeltaSharing file-system. (450)
- Added more logging for paginated query requests (445) (446)

Credits: Charlene Lyu, Lin Zhou, Moe Derakhshani

We’d like to announce the release of Delta Sharing 1.0.3, which introduces the following improvements.
Improvements:
- Improve error handling and logging in DeltaSharingSource. (436)
- Introduce Timestamp and UUID as suffix to table path in RemoteDeltaLog (443)

Credits: Lin Zhou

We’d like to announce the release of Delta Sharing 1.0.2, which introduces the following improvements.
Improvements:

- DeltaSharingClient able to set responseFormat as a list of format it could cover, and handle the response in any format in the list. (422, 427)
- RemoteDeltaLog to be able to accept table metadata as an input parameter when being initialized. (423)

Credits: Lin Zhou

We’d like to announce the release of Delta Sharing 1.0.1, which introduces the following improvements.
Improvements:

- Improve error message on unrecognized file from server, and add queryTableVersionIntervalSeconds (402)
- Migrate parquet sharing related classes from spark/ to client/ (405)
- Fix Tests with a copy of DeltaMetadata (408)
- Move limit pushdown config to ConfUtils (411)
- Fix type of files used for DeltaTableFiles (418)

Credits: Charlene Lyu, Lin Zhou

We are super excited to announce the release of Delta Sharing 1.0.0, which includes several exciting new features and artifacts.

Delta Format Sharing
In order to support Delta advanced features such as DeletionVectors and ColumnMapping in the Delta Sharing OSS Spark connector, “Delta format sharing” is introduced. The core Delta Sharing functionalities are grouped into the delta-sharing-client library and will be released separately. The response of Delta format sharing will not be processed by delta-sharing-client; instead,advanced Delta Spark library will be leveraged to read the data. We also upgraded the Delta Sharing server to support this. (334, 335, 337, 338, 370, 374, 375, 376, 377, 378, 388, 389, 392, 393, 395)

JsonPredicate Enhancements (JsonPredicatesV2)
We enhanced JsonPredicates to support predicate-based filtering on all columns and not just partition columns. We also added support for the following schema types: Float, Double, Timestamp. (349, 352, 368, 394)
We added JsonPredicate support in the pandas connector (396)

Pagination on Delta Sharing Queries
Pagination on Delta Sharing data access APIs enables control over the number of files returned per Query Table request. This improves the scalability of the Delta Sharing service when working with tables with large file volumes, preventing timeouts and resource limit exceedances. (352, 353, 354, 356, 360, 362, 366)

Other improvements and fixes
Support the release of Maven artifacts for both Scala 2.12 and Scala 2.13. 333
Support refresh token to refresh the pre-signed urls correctly for queries on the latest snapshot. (385, 386, 387)
Fix bug where to_pandas doesn't properly handle the case when column names between Delta and parquet metadata are different in casing (384)
A couple build and test fixes. (330, 364, 369)
A couple fixes on the PROTOCOL description. 340, 365
Fix an issue in DeltaSharingSource: to double check sortedFetchedFiles contain the correct files and set previousOffset correctly. 328, 344

Credits
Abhijit Chakankar, Charlene Lyu, Kr eis, Lin Zhou, William Chau, andyl-db