Daft

Changes

✨ New Features

- [FEAT] Read from LanceDB jaychia (2195)
- .sqrt() expression dmaymay (2180)

👾 Bug Fixes

- [BUG] Propagate errors when hitting them in parquet byte stream samster25 (2214)

📖 Documentation

- .sqrt() expression dmaymay (2180)
- [Docs] Add Dask to DF Comparison FAQ avriiil (2210)

🧰 Maintenance

- [CHORE]: statically link liblzma universalmind303 (2213)

Changes
This is the last release that will support Python 3.7 which has been EOL for about a year now.

✨ New Features

- [FEAT] Rust side exceptions for Transient Errors samster25 (2197)
- [FEAT] Enable anonymous S3 access for Delta jaychia (2206)
- [FEAT] Improve Hudi support for more scenarios xushiyan (2149)
- [FEAT] try python 3.11 for releases samster25 (2184)
- [FEAT] Timestamp Truncation colin-ho (2158)
- [FEAT] Enhance temporal arithmetic functionalities colin-ho (2146)
- [FEAT] Add logarithmic expressions murex971 (2168)
- [PERF] Move `with_column` and `exclude` function logic to Rust side, add `with_columns` kevinzwang (2167)
- [FEAT] Allow for variadic kwargs in UDFs jaychia (2162)
- [FEAT] hide stack traces of wrappers for pytest / ipython samster25 (2159)
- [FEAT] Improve query building in read\_sql colin-ho (2144)

🚀 Performance Improvements

- [PERF] Move `with_column` and `exclude` function logic to Rust side, add `with_columns` kevinzwang (2167)
- [PERF] Refactor TreeNode to be native to Arc\<TreeNode> samster25 (2175)

👾 Bug Fixes

- [FEAT] Improve Hudi support for more scenarios xushiyan (2149)
- [BUG] bump arrow2 to use copy ptr instead of from\_raw\_parts samster25 (2194)
- [BUG] Fix empty inputs case for string kernels jaychia (2165)
- [BUG] Fix tuple inputs in UDF jaychia (2161)

📖 Documentation

- [DOCS] Add Hudi integration entry xushiyan (2208)
- [FEAT] Improve Hudi support for more scenarios xushiyan (2149)
- [FEAT] Timestamp Truncation colin-ho (2158)
- [FEAT] Add logarithmic expressions murex971 (2168)
- [DOCS] add Dask migration guide avriiil (2169)
- [CHORE] enable codespell and fix mispellings samster25 (2177)
- [PERF] Move `with_column` and `exclude` function logic to Rust side, add `with_columns` kevinzwang (2167)

🧰 Maintenance

- [CHORE] Upgrade Rust toolchain to 2024 04 01 samster25 (2192)
- [CHORE] upgrade dask again for publish pipeline samster25 (2193)
- [CHORE] remove fsspec http filesystem samster25 (2191)
- [CHORE] add ray min version for windows to remove upper pin on pyarrow samster25 (2189)
- [CHORE] upgrade dask for tests for 3.11 samster25 (2186)
- [CHORE] disable macos check for testing in publish pipeline samster25 (2185)
- [CHORE] upgrade publish pipeline python version to 3.12 samster25 (2182)
- [CHORE] use python 3.9 for publishing samster25 (2181)
- [CHORE] enable codespell and fix mispellings samster25 (2177)
- [CHORE] Add usr msg for df.explain(show\_all=True) avriiil (2081)
- [CHORE] improve recursive listing error msg avriiil (2145)
- [CHORE] Enforce physical types in logical array jaychia (2160)
- [CHORE] Fix style build on main jaychia (2163)
- [CHORE] unify expr children around expr ref samster25 (2156)
- [CHORE] Rearrange modules in `daft-plan` crate samster25 (2151)
- [CHORE] collect all tests before running pytest so check for import errors samster25 (2150)
- [CHORE] Bottom Up Logical To Physical translation samster25 (2147)

⬆️ Dependencies

<details>
<summary>5 changes</summary>

- Bump comfy-table from 7.1.0 to 7.1.1 dependabot (2199)
- Bump num-traits from 0.2.17 to 0.2.18 dependabot (2200)
- Bump slackapi/slack-github-action from 1.25.0 to 1.26.0 dependabot (2174)
- Bump regex from 1.10.3 to 1.10.4 dependabot (2172)
- Bump serde\_json from 1.0.108 to 1.0.116 dependabot (2173)
</details>

Changes

✨ New Features

- [FEAT] Add S3Config.from\_env functionality jaychia (2137)
- deltalake \_delta\_lake.py: Allow Glue catalog cross account access pang-wu (2113)
- [FEAT] Enable Ruff samster25 (2121)
- [FEAT] Implements other trigonometry expressions MeepoWin (2123)
- [FEAT] exp expression implementation MeepoWin (2115)
- [FEAT] sin/cos/tan expression implementation reswqa (2112)
- [CHORE] Using `uv` in MakeFile MeepoWin (2114)
- [FEAT] Add option to S3Config to force virtual addressing samster25 (2106)
- [FEAT] fill\_null expression colin-ho (2089)
- [FEAT] Add basic list aggregations kevinzwang (2032)
- [FEAT] Allow sql alchemy connection factory as input to read\_sql colin-ho (2071)
- [FEAT] Add daft-sketch subcrate and arrow2 serialization functionality jaychia (2090)

👾 Bug Fixes

- [BUG] Fix reading partition key columns in DeltaLake jaychia (2118)

📖 Documentation

- [CHORE] Fix underlines in README jaychia (2143)
- [DOCS] Update iceberg integration docs to add writes jaychia (2110)
- [DOCS] Create CODE\_OF\_CONDUCT.md samster25 (2101)
- [CHORE] Skip deltalake notebooks for CI jaychia (2097)
- [CHORE] Add link to good first issues in readme colin-ho (2088)
- [DOCS] Fix docs typo avriiil (2075)
- [DOCS] Typos in user guide avriiil (2079)
- [DOCS] Fix typos on 10-min tutorial avriiil (2082)
- [DOCS] Add ml batch inference tutorials jaychia (2057)
- [CHORE] Fix autolabeller CI step for forks jaychia (2138)

🧰 Maintenance

- [CHORE] Fix underlines in README jaychia (2143)
- [CHORE] Split labelling and update release CI steps jaychia (2142)
- [CHORE] Fix the labeller CI step which is not triggering jaychia (2141)
- [CHORE] Fixing readthedocs build jaychia (2135)
- [CHORE] Fix documentation build with uv jaychia (2134)
- [CHORE] Fix build command MeepoWin (2126)
- [CHORE] Rename virtual env folder to `.venv` MeepoWin (2122)
- [CHORE] refactors for ruff [1/n] samster25 (2120)
- [CHORE] FunctionExpr and exp samster25 (2119)
- [CHORE] FunctionEvaluator directly receive `FunctionExpr` MeepoWin (2117)
- [CHORE] Update .gitignore for JetBrains IDE and pyenv user MeepoWin (2116)
- [CHORE] Refactor string kernels colin-ho (2087)
- [CHORE] Skip deltalake notebooks for CI jaychia (2097)
- [CHORE] Add link to good first issues in readme colin-ho (2088)
- [CHORE] fix empty data and pattern case in str expressions murex971 (2085)

⬆️ Dependencies

<details>
<summary>5 changes</summary>

- Bump bytes from 1.5.0 to 1.6.0 dependabot (2131)
- Bump futures from 0.3.28 to 0.3.30 dependabot (2130)
- Bump isbang/compose-action from 1.5.1 to 2.0.0 dependabot (2091)
- Bump dyn-clone from 1.0.16 to 1.0.17 dependabot (2093)
- Bump tokio from 1.33.0 to 1.37.0 dependabot (2092)
</details>

Changes
✨ New Features

- [FEAT] improve error message for s3 streaming error samster25 (2055)
- [FEAT] Add str.replace expression colin-ho (2048)
- [FEAT] Enable str.split using regex pattern colin-ho (2044)
- [FEAT] Support Hudi reader xushiyan (2011)
- [FEAT] Add find functionality for string murex971 (2046)
- [FEAT] round expression implemtation sherlockbeard (2041)
- [FEAT] Add str.extract\_all expression colin-ho (2038)
- [FEAT] Add str.right() function murex971 (2031)
- [FEAT] Sign expression implemtation sherlockbeard (2037)
- [FEAT] drop psutil in favor of our own tool samster25 (2035)
- [FEAT] Allow passing on\_error="null" to ignore decoding errors in image decode jaychia (2033)
- [FEAT] Add str.extract() function colin-ho (2020)
- [FEAT] Add str.left() funtion murex971 (2027)

🚀 Performance Improvements

- [PERF] [Delta Lake] Add IO multithreading arg to `daft.read_delta_lake()`. clarkzinzow (2029)

👾 Bug Fixes

- [BUG] Allow for writes to s3a and s3n paths jaychia (2054)
- [BUG] Fix if\_else series naming from predicate broadcast colin-ho (2051)
- [BUG] enable dependabot for iceberg int tests samster25 (2042)
- [BUG] Fix all-null ImageArray length issues jaychia (2034)
- [BUG] produce only a single sdist samster25 (2078)

📖 Documentation

- [FEAT] Add str.replace expression colin-ho (2048)
- [DOCS] Add docs for write\_iceberg jaychia (2053)
- [FEAT] Add str.extract\_all expression colin-ho (2038)
- [FEAT] Add str.extract() function colin-ho (2020)
- [CHORE] Add global aggregation docs and error on improper aggregation usage kevinzwang (2025)
- [DOCS] Fix typos and broken links kaytsui (2052)

🧰 Maintenance

- [CHORE] Remove autouse from gen\_tpch fixture colin-ho (2049)
- [CHORE] Refactor sql tpch tests colin-ho (2047)
- [CHORE] Add tpch test for read sql colin-ho (2026)
- [CHORE] Add column range stats from read\_sql colin-ho (2015)
- [CHORE] upgrade upload/download artifact github action samster25 (2043)
- [CHORE] Exclude Twitter and LinkedIn from broken link checker colin-ho (2030)
- [CHORE] Add global aggregation docs and error on improper aggregation usage kevinzwang (2025)
- [CHORE] Use docker compose instead of docker-compose in builds jaychia (2077)

⬆️ Dependencies

<details>
<summary>10 changes</summary>

- Bump release-drafter/release-drafter from 5 to 6 dependabot (2065)
- Bump nick-fields/retry from 2 to 3 dependabot (2066)
- Bump openssl-sys from 0.9.93 to 0.9.102 dependabot (2062)
- Bump async-trait from 0.1.74 to 0.1.79 dependabot (2061)
- Bump base64 from 0.21.5 to 0.22.0 dependabot (2059)
- Bump async-compression from 0.4.5 to 0.4.7 dependabot (2060)
- Bump lxml from 4.9.3 to 5.1.0 dependabot (1764)
- Bump slackapi/slack-github-action from 1.24.0 to 1.25.0 dependabot (1822)
- Bump actions/setup-python from 4 to 5 dependabot (1717)
- Bump conda-incubator/setup-miniconda from 2 to 3 dependabot (1666)
</details>

Changes

✨ New Features

- [FEAT] iceberg writes unpartitioned samster25 (2016)
- [FEAT] Add str.match() function colin-ho (2007)
- [FEAT] read\_sql colin-ho (1943)

👾 Bug Fixes

- [BUG] Fix connector-x and psycopg dependencies for CI colin-ho (2017)
- [BUG] Disable Numeric and String comparison samster25 (2019)
- [BUG] deltalake read pq splitting bug jaychia (2013)

📖 Documentation

- [DOCS] [Hotfix] [Delta Lake] Break data skipping optimizations into different section. clarkzinzow (2018)
- [CHORE] Fix is\_in docs colin-ho (2022)
- [FEAT] Add str.match() function colin-ho (2007)
- [FEAT] read\_sql colin-ho (1943)

🧰 Maintenance

- [CHORE] Unskip tensor tests in test\_parquet\_roundtrip.py jaychia (2024)
- [CHORE] Fix is\_in docs colin-ho (2022)

Changes

✨ New Features

- [FEAT] Top level global expressions kevinzwang (2000)
- [FEAT] Add str.capitalize() function murex971 (2003)
- [FEAT] Support reading Parquet files with Field ID jaychia (1990)
- [FEAT] Enable JQ style JSON accessors on strings colin-ho (2001)
- [FEAT] [Catalogs] [Delta Lake] Add support for AWS Glue Catalog and Databricks Unity Catalog integrations to Delta Lake reader clarkzinzow (1991)
- [FEAT] Enable UDF to handle arbitrary number of Daft series gmweaver (1984)

👾 Bug Fixes

- [BUG] skip metadata check for field equality samster25 (2006)
- [BUG] Fix struct getters on logical types jaychia (2008)
- [BUG] Filter out marker files from glob scan colin-ho (1999)

📖 Documentation

- [DOCS] Fix stale docs in df.write\_csv jaychia (2012)
- [FEAT] Enable JQ style JSON accessors on strings colin-ho (2001)

🧰 Maintenance

- [CHORE] [Hotfix] Remove pyarrow upper bound for Windows. clarkzinzow (2002)
- [CHORE] [Catalogs] [Delta Lake] Add test coverage for Delta Lake reads on Azure. clarkzinzow (1970)
- [CHORE] [Repartitioning] Refactor + hide `PartitionSpec` and rename to `ClusteringSpec`. clarkzinzow (1961)
- [CHORE] Simplify cast to schema jaychia (1982)
- [CHORE] Disables anonymous mode for S3 accesses in DeltaLake jaychia (1975)
- [CHORE] Set DAFT\_ANALYTICS\_ENABLED=0 in nightly tests jaychia (1972)