Daft

Changes

🏆 Highlights

- [FEAT] [Tensor] Add support for `Tensor` and `FixedShapeTensor` types. clarkzinzow (1073)

✨ New Features

- [FEAT] Consolidate to list namespace jaychia (1180)
- [FEAT] Add .image.crop Expression jaychia (1175)
- [FEAT] [Tensor] Add support for `Tensor` and `FixedShapeTensor` types. clarkzinzow (1073)
- [FEAT] Basic support for Arrow 128-bit Decimal. xcharleslin (1129)
- [FEAT] Native Parquet Downloader samster25 (1107)

🚀 Performance Improvements

- [PERF] Simple Read Planner and RangeReader for Native Parquet Reader samster25 (1172)

👾 Bug Fixes

- [BUG] Fix ownership model of IOClient samster25 (1128)
- [BUG] Ownership of Runtime and Clients samster25 (1125)

📖 Documentation

- [DOCS] Fix broken link to Ray Datasets docs jaychia (1186)
- [FEAT] Consolidate to list namespace jaychia (1180)
- [DOCS] Add docs for tensor dtype jaychia (1170)
- [DOCS] Add Flyte example jaychia (1150)
- [CHORE] Update README.rst typo jaychia (1141)

🧰 Maintenance

- [CHORE] Bump cargo version to 0.1.9 jaychia (1187)
- [CHORE] Exclude JSON pre-commit fixer for ipynb files jaychia (1184)
- [CHORE] New daft-plan crate; trait TreeDisplay xcharleslin (1176)
- [CHORE] More Parquet benchmarking jaychia (1160)
- [CHORE] Enable Parquet Integration tests for decimal types samster25 (1161)
- [CHORE] cache all crates samster25 (1158)
- [CHORE] move parquet unit tests under io samster25 (1157)
- [CHORE] [CI] use smarter github rust cache action samster25 (1156)
- [CHORE] bump profiling timeout samster25 (1155)
- [CHORE] Native Parquet Integration Tests samster25 (1154)
- [CHORE] Remove use of `dirs_exist_ok` which was only added in Py3.8 jaychia (1153)
- [CHORE] Add parquet benchmarking jaychia (1151)
- [CHORE] Cleans up IO integration test fixtures for re-use jaychia (1152)
- [CHORE] Update README.rst typo jaychia (1141)
- [CHORE] No-op test for various parquet files jaychia (1130)
- [CHORE] Tidy typing for remaining binary ops: logical, comp xcharleslin (1124)
- [CHORE] Use workspace for cargo check samster25 (1127)

⬆️ Dependencies

<details>
<summary>10 changes</summary>

- Bump orjson from 3.9.1 to 3.9.2 dependabot (1143)
- Bump pandas from 2.0.2 to 2.0.3 dependabot (1142)
- Bump snafu from 0.7.4 to 0.7.5 dependabot (1146)
- Bump serde\_json from 1.0.99 to 1.0.100 dependabot (1147)
- Bump opencv-python from 4.7.0.72 to 4.8.0.74 dependabot (1117)
- Bump ray[data,default] from 2.4.0 to 2.5.1 dependabot (1074)
- Bump chrono-tz from 0.8.2 to 0.8.3 dependabot (1119)
- Bump pyo3 from 0.19.0 to 0.19.1 dependabot (1122)
- Bump async-trait from 0.1.68 to 0.1.71 dependabot (1126)
- Bump tokio from 1.28.2 to 1.29.1 dependabot (1120)
</details>

Changes

✨ New Features

- [FEAT] Ranged Get Native Downloader samster25 (1113)
- [FEAT] Native S3 Downloader Anonymous Mode samster25 (1105)
- [FEAT] Enable reading a list of URLs in read\_\* APIs jaychia (1102)
- [FEAT] Arithmetic with timestamps and durations. xcharleslin (1103)
- [FEAT] Automatic Region Retrying for S3 Native Downloader samster25 (1098)
- [FEAT] Better styling of large dataframe cells in HTML jaychia (1097)

👾 Bug Fixes

- [BUG] S3 Downloader set default region when region not detected samster25 (1100)

📖 Documentation

- [CHORE] Update README.rst for image downloading jaychia (1109)
- [DOCS] Update image tutorials with `.image` namespaced expressions jaychia (1110)

🧰 Maintenance

- [CHORE] Tidy up typing of binary ops [1/2] xcharleslin (1114)
- [CHORE] Pin Pydantic to \< 2 jaychia (1115)
- [CHORE] Remove rogue print statement jaychia (1112)
- [CHORE] Install wheel together with requirements in release build jaychia (1111)
- [CHORE] Update README.rst for image downloading jaychia (1109)
- [CHORE] Adding more test fixtures for different I/O sources jaychia (1083)
- [CHORE] Cache build artifacts in target folder jaychia (1104)
- [CHORE] Fix CI caching to cache integration test builds separately jaychia (1101)
- [CHORE] Use maturin directly instead of multiplatform build step jaychia (1099)

⬆️ Dependencies

- Bump serde\_json from 1.0.97 to 1.0.99 dependabot (1095)
- Bump pytest from 7.3.2 to 7.4.0 dependabot (1089)

Changes

🏆 Highlights

- [FEAT] Add `DataFrame.to_torch_map_dataset` and `.to_torch_iter_dataset`. xcharleslin (1086)
- [PERF] Rust based url downloading with error handling samster25 (1061)

✨ New Features

- [FEAT] Enable Native Downloader IO Config samster25 (1090)
- [FEAT] Add `DataFrame.to_torch_map_dataset` and `.to_torch_iter_dataset`. xcharleslin (1086)
- [FEAT] DataFrame.\_\_iter\_\_() and .iter\_partitions() xcharleslin (1062)
- [FEAT] New DataType: Duration (without arithmetic) xcharleslin (1051)
- [FEAT] [Images] [9/N] Infer `Image` type for PIL images on ingress. clarkzinzow (1067)
- [FEAT] Automatically cast logical types to Python objects on `Series.to_pylist()`. clarkzinzow (1063)
- [FEAT] [Images] [8/N] Add encoding and resizing support for fixed-shape images. clarkzinzow (1052)
- Dataframe Iter 1/n: Physical plan streams results into Runner. xcharleslin (1060)

🚀 Performance Improvements

- [PERF] Rust based url downloading with error handling samster25 (1061)

👾 Bug Fixes

- [BUG] Fix remote mode typo xcharleslin (1092)
- [BUG] Reenable HTML viz hooks for np.ndarray and PIL Images jaychia (1078)
- [BUG] Fix string index bug in table repr xcharleslin (1079)
- [BUG] pin the version of python used in publishing samster25 (1068)
- [BUG] [CI] Fix merge conflict due to out-of-date base. clarkzinzow (1066)

📖 Documentation

- [FEAT] Add `DataFrame.to_torch_map_dataset` and `.to_torch_iter_dataset`. xcharleslin (1086)
- [CHORE] Fix filepath for autogeneration of .list.join docs jaychia (1084)
- In CI, limit tutorial to 500 rows xcharleslin (1076)
- [DOCS] Embeddings tutorial: Temporarily remove full dataset xcharleslin (1039)
- [DOCS] Remove release notes from documentation, link to Github instead jaychia (1049)

🧰 Maintenance

- [CHORE] set dependabot schedule to weekly samster25 (1085)
- [CHORE] Refactor integration test to use wheel built for release jaychia (1087)
- [CHORE] unpin numpy version for py\<3.8 jaychia (1088)
- [CHORE] Fix filepath for autogeneration of .list.join docs jaychia (1084)
- [CHORE] Crate Smash v1 samster25 (1080)
- [CHORE] Scheduler cleanup: merge logical\_op\_runners.py into execution\_step xcharleslin (1020)
- [CHORE] Inline the label enforcer into the release drafter wf jaychia (1057)
- [CHORE] Fix naming of "Release Drafter" workflow in trigger jaychia (1055)
- [CHORE] Add new trigger to run PR label enforcement after Release Drafter jaychia (1054)
- [CHORE][CI] Use pyarrow Table sort API that's compatible with older pyarrow versions clarkzinzow (1053)

⬆️ Dependencies

<details>
<summary>4 changes</summary>

- Bump hypothesis from 6.79.1 to 6.79.2 dependabot (1082)
- [CHORE] Crate Smash v1 samster25 (1080)
- Bump hypothesis from 6.78.2 to 6.79.1 dependabot (1065)
- Bump numpy from 1.24.3 to 1.25.0 dependabot (1064)
</details>

Changes

🏆 Highlights

- [FEAT] Support for Timestamp datatype. xcharleslin (1032)

✨ New Features

- [FEAT] Support for Timestamp datatype. xcharleslin (1032)
- [FEAT] Thread user-provided schema through to DataFrame reads jaychia (1024)
- [FEAT] Daft Image viz support. Remove Tabulate dependency. xcharleslin (1027)
- [FEAT] Dataframe Concats jaychia (1023)
- [FEAT] Add kernels for .list.join on a list[utf8] column jaychia (989)
- [FEAT][Table-Read-Schema 2/3] Add table casting logic jaychia (1012)
- [FEAT][Table-Read-Schema 1/3] Split reading tabular file formats into 2 method calls jaychia (1010)
- [FEAT][Images] [7/N] Add image encoding support. clarkzinzow (1013)
- [FEAT] Visualization cleanup 2/n: Add repr\_html to Series, Table, and PyO3 xcharleslin (1018)
- [FEAT] Visualization cleanup (1/n): Use Table for repr xcharleslin (1011)

📖 Documentation

- [DOCS] Fix links to ray.io latest docs jaychia (1038)
- [DOCS] Add initial docs pass, adding lots of cross-reference links. clarkzinzow (1009)
- [DOCS][Images] [6/N] Fix image dtype docs. clarkzinzow (1008)

The Daft 0.1.5 release features better series exporting, bugfixes and improved documentation.

Enhancements

* Enable Cast from Image to Python via Numpy [990](https://github.com/Eventual-Inc/Daft/pull/990)

Bug Fixes

* Fix Image Resize/Decode Expressions [1001](https://github.com/Eventual-Inc/Daft/pull/1001)

Build Changes

* Python script for subprefixing s3 tpch files [997](https://github.com/Eventual-Inc/Daft/pull/997)
* Update pyo3-log from 0.8.1 to 0.8.2 [996](https://github.com/Eventual-Inc/Daft/pull/996)
* Update hypothesis from 6.75.9 to 6.76.0 [995](https://github.com/Eventual-Inc/Daft/pull/995)

Documentation
* Include Dataframe comparison and related projects in readme [1005](https://github.com/Eventual-Inc/Daft/pull/1005)
* Include Benchmarks in Readme [1003](https://github.com/Eventual-Inc/Daft/pull/1003)
* Add Red Pajamas Tutorial to docs [1002](https://github.com/Eventual-Inc/Daft/pull/1002)
* Include Blog in docs [1000](https://github.com/Eventual-Inc/Daft/pull/1000)
* Update Datatype docs for complex types [999](https://github.com/Eventual-Inc/Daft/pull/999)

The Daft 0.1.4 release features our Image type columns!

New Features

Image Types

Our first Daft Image types have landed!

You can now construct an Image column with .image.decode() on a Binary column.

See PRs:

1. [Images] [1/N] Logical type for variable-shaped and fixed-shaped images. [955](https://github.com/Eventual-Inc/Daft/pull/955)
2. [Images] [3/N] Add image decoding for uint8 images. [981](https://github.com/Eventual-Inc/Daft/pull/981)
3. Image Resize for ImageType [967](https://github.com/Eventual-Inc/Daft/pull/967)
4. [Images] [2/N] Add scaffolding for image decoding and other ops. [965](https://github.com/Eventual-Inc/Daft/pull/965)

Documentation

1. Fix modin typo; add partial scale numbers; highlight highlights [986](https://github.com/Eventual-Inc/Daft/pull/986)
2. Add Page on benchmarking [980](https://github.com/Eventual-Inc/Daft/pull/980)
3. Fix link for broken link checker [972](https://github.com/Eventual-Inc/Daft/pull/972)
4. Clean up DataType userguide/API docs [966](https://github.com/Eventual-Inc/Daft/pull/966)
5. Add more complex datatypes to docs [961](https://github.com/Eventual-Inc/Daft/pull/961)
6. Datatype docs [894](https://github.com/Eventual-Inc/Daft/pull/894)

Bug Fixes

1. [Scheduler] Fix join performance bug [985](https://github.com/Eventual-Inc/Daft/pull/985)
2. Table Slice and IntoPartitions Fix [962](https://github.com/Eventual-Inc/Daft/pull/962)
3. size_bytes fix: Guard against calculating variance of one item [957](https://github.com/Eventual-Inc/Daft/pull/957)

Build Changes

1. Bump hypothesis from 6.75.8 to 6.75.9 [979](https://github.com/Eventual-Inc/Daft/pull/979)
2. Bump orjson from 3.8.14 to 3.9.0 [978](https://github.com/Eventual-Inc/Daft/pull/978)
3. Bump hypothesis from 6.75.6 to 6.75.8 [976](https://github.com/Eventual-Inc/Daft/pull/976)
4. Add s3fs to dev requirements [975](https://github.com/Eventual-Inc/Daft/pull/975)
5. Bump python version to 3.9 for profiling and pin Dask version for python 3.8 [973](https://github.com/Eventual-Inc/Daft/pull/973)
6. Bump log from 0.4.17 to 0.4.18 [971](https://github.com/Eventual-Inc/Daft/pull/971)
7. Bump dask from 2023.5.0 to 2023.5.1 [970](https://github.com/Eventual-Inc/Daft/pull/970)
8. Bump pandas from 2.0.1 to 2.0.2 [969](https://github.com/Eventual-Inc/Daft/pull/969)
9. Bump hypothesis from 6.75.5 to 6.75.6 [968](https://github.com/Eventual-Inc/Daft/pull/968)
10. Bump orjson from 3.8.13 to 3.8.14 [964](https://github.com/Eventual-Inc/Daft/pull/964)
11. Bump hypothesis from 6.75.3 to 6.75.5 [963](https://github.com/Eventual-Inc/Daft/pull/963)
12. finer feature flags for arrow2 for faster compile [960](https://github.com/Eventual-Inc/Daft/pull/960)
13. Bump pytest-cov from 4.0.0 to 4.1.0 [959](https://github.com/Eventual-Inc/Daft/pull/959)
14. Bump orjson from 3.8.12 to 3.8.13 [958](https://github.com/Eventual-Inc/Daft/pull/958)