Daft

Changes

✨ New Features

- [FEAT] Anonymous Scan Operator samster25 (1526)
- [FEAT] Micropartition integration and tests jaychia (1502)
- [FEAT] Make Binary Type Comparable samster25 (1528)
- [FEAT] implement series serde samster25 (1519)
- [FEAT] Add streaming + parallel CSV reader, with decompression support. clarkzinzow (1501)
- [FEAT] IOStats for Native Reader samster25 (1493)

🚀 Performance Improvements

- [PERF] Add "eager mode" to limits and use in .show() jaychia (1498)
- [PERF] Micropartition, lazy loading and Column Stats samster25 (1470)
- [PERF] Use pyarrow table for pickling rather than ChunkedArray samster25 (1488)
- [PERF] Use region from system and leverage cached credentials when making new clients samster25 (1490)
- [PERF] Update default max\_connections 64->8 because it is now per-io-thread jaychia (1485)
- [PERF] Pass-through multithreaded\_io flag in read\_parquet jaychia (1484)

👾 Bug Fixes

- [BUG] Fix timestamp timezone parsing bug in CSVs jaychia (1530)
- [BUG] Re-raise exceptions in rayrunner jaychia (1522)
- [BUG] [CSV Reader] Fix CSV parsing bugs around nulls and timestamps. clarkzinzow (1523)
- [BUG] Fix handling of special characters in S3LikeSource jaychia (1495)
- [BUG] Fix local globbing of current directory jaychia (1494)
- [BUG] fix script to upload file 1 at a time samster25 (1492)
- [CHORE] Add tests and fixes for Azure globbing jaychia (1482)

📖 Documentation

- [FEAT] Micropartition integration and tests jaychia (1502)

🧰 Maintenance

- [CHORE] Allow release-drafter to increment minor version jaychia (1532)
- [CHORE] Soft deprecation of fsspec from user-facing APIs jaychia (1467)
- [CHORE] bring up fixtures for iceberg samster25 (1527)
- [CHORE] Skip IO integration tests if being run from dependabot jaychia (1521)
- [CHORE] Better logging for physical plan jaychia (1499)
- [CHORE] Refactor logging jaychia (1489)
- [CHORE] Add Workflow to build artifacts and upload to S3 samster25 (1491)
- [CHORE] Update default num\_tries on S3Config to 25 jaychia (1487)
- [CHORE] Add tests and fixes for Azure globbing jaychia (1482)

Changes

✨ New Features

- [FEAT] Streaming CSV reads xcharleslin (1479)
- [FEAT] [Native I/O] Add a native CSV reader. clarkzinzow (1475)

🚀 Performance Improvements

- [PERF] Update number of cores on every iteration jaychia (1480)
- [Hotfix] Change to streaming reader for CSV schema inference. clarkzinzow (1471)

👾 Bug Fixes

- [BUG] Properly dispatch limited reads in new query planner xcharleslin (1476)
- [BUG] Fixes globbing on windows by consolidating on posix-style paths jaychia (1472)

🧰 Maintenance

- [CHORE] Create SECURITY.md samster25 (1481)

Changes

✨ New Features

- [FEAT] Native globbing for other backends jaychia (1460)
- [FEAT] Native `glob` functionality jaychia (1450)
- [FEAT] ls/list\_dir for AzureBlobStorage xcharleslin (1408)
- [FEAT] Add `.str.split()` API for splitting string columns. clarkzinzow (1409)
- [FEAT] Add local native filesystem globbing. clarkzinzow (1449)
- [FEAT] Native listing of http URLs jaychia (1405)

🚀 Performance Improvements

- [PERF] Local filesystem parquet reader samster25 (1461)
- [PERF] Native globbing early stopping jaychia (1452)

👾 Bug Fixes

- [BUG] fix circ import with pythonpath is set samster25 (1474)
- [BUG] Don't remove all handles and Only use handler for files in `src/` samster25 (1473)

🧰 Maintenance

- [FEAT] Native globbing for other backends jaychia (1460)
- [CHORE] update s3 connection defaults samster25 (1451)

Changes

✨ New Features

- [FEAT] Add support for windows in daft samster25 (1386)
- [FEAT] Add debug logging to s3 native apis samster25 (1414)
- [FEAT] enable path style for s3 custom endpoints by default samster25 (1410)
- [FEAT] Native S3 Lister, support trailing slashes and fix panics when connection is dropped for tokio samster25 (1404)
- [FEAT] Native Rust listing of GCS jaychia (1392)
- [FEAT] [New Query Planner] Enable new query planner by default. clarkzinzow (1398)
- [FEAT] Parameter to set num\_parallel\_tasks for bulk readers samster25 (1399)
- [FEAT] Native S3 Client: allow disabling ssl verification or checking hostnames samster25 (1395)
- [FEAT] Improved projection folding. xcharleslin (1374)
- [FEAT] bulk parquet pyarrow reader samster25 (1396)
- [FEAT] Native Recursive File Lister samster25 (1353)
- [FEAT] Implement .dt.year/month/day for timestamp types jaychia (1385)
- [FEAT] [New Query Planner] Add support for fsspec filesystems to new query planner. clarkzinzow (1357)
- [FEAT] Common subexpression elimination in Projection construction xcharleslin (1347)

👾 Bug Fixes

- [BUG] Fix num input partitions in coalesce. clarkzinzow (1442)
- [BUG] Fix scheme bug in GCS anonymous mode jaychia (1443)
- [BUG] Fix runner check at plan execution time for new query planner clarkzinzow (1435)
- [BUG] [Docs] Allow source code discovery to fail silently for pyo3-defined classes when generating docs. clarkzinzow (1430)
- [BUG] patch workspace version when building wheels samster25 (1418)
- [BUG] Anaconda client don't upload src wheels samster25 (1415)
- [BUG] Anaconda client needs wildcard for upload samster25 (1413)
- [BUG] Fix gs listing to include 0 sized marker files jaychia (1412)
- [BUG] force upload of anaconda nightly wheels samster25 (1411)
- [BUG] add test cases for bulk minio reading samster25 (1402)
- [BUG] Fixes to S3 Native Lister with correct Error propagation samster25 (1401)
- [BUG] Fix public API decorator type annotations. clarkzinzow (1397)
- [BUG] Fix partition spec bugs from old query planner xcharleslin (1372)

📖 Documentation

- [BUG] [Docs] Allow source code discovery to fail silently for pyo3-defined classes when generating docs. clarkzinzow (1430)
- [FEAT] Implement .dt.year/month/day for timestamp types jaychia (1385)

🧰 Maintenance

- [CHORE] disable windows pytest after building samster25 (1420)
- [CHORE] add caching for pip wheels samster25 (1419)
- [CHORE] macos xl runners are 0.32/minute not hour... samster25 (1417)
- [CHORE] Centralize pyo3 pickling around `__reduce__` + bincode macro. clarkzinzow (1394)
- [CHORE] larger macos runner for builds samster25 (1403)
- [CHORE] Add stubs and improve comments for pyo3-exposed abstractions, + driveby type/bug fixes. clarkzinzow (1377)
- [CHORE] add retries for broken link checker samster25 (1378)
- [CHORE] pin azure-storage-blob due to breaking new version samster25 (1373)
- [CHORE] [New Query Planner] Misc. user-facing error tweaks to improve UX. clarkzinzow (1358)

Changes

✨ New Features

- [FEAT] Native Parquet Reader into pyarrow directly samster25 (1366)
- [FEAT] Add configurable io thread pool size samster25 (1363)
- [FEAT] Add flag to limit number of connections to S3 samster25 (1360)
- [FEAT] export jemalloc arm64 flag inside container samster25 (1362)

🚀 Performance Improvements

- [PERF] Used owned Stream in Parquet Page Iterator samster25 (1365)
- [PERF] enable jemalloc with background threads samster25 (1361)
- [PERF] Add microbenchmarks for takes jaychia (1350)
- [PERF] Optimize filter on nested growables jaychia (1349)

👾 Bug Fixes

- [BUG] Respect `multithreaded_io` flag when reading parquet samster25 (1359)
- [BUG] Schema Display should use dtype Display instead of Debug jaychia (1355)
- [BUG] propagate parquet io error instead of panicking samster25 (1352)

🧰 Maintenance

- [CHORE] [New Query Planner] Add simple `df.explain()` option; change to fixed-point policy for rule batch clarkzinzow (1354)
- [CHORE] Add status code to IO integration tests jaychia (1356)
- [CHORE] Fix List/FixedSizeList DataType to hold a dtype instead of Field jaychia (1351)
- [CHORE] Add Series::full\_null/empty/from\_arrow to reduce code duplication jaychia (1331)
- [CHORE] Add a Growable factory method jaychia (1330)
- [CHORE] Add new ListArray jaychia (1329)

⬆️ Dependencies

<details>
<summary>5 changes</summary>

- Bump tokio from 1.29.1 to 1.32.0 dependabot (1371)
- Bump tempfile from 3.7.1 to 3.8.0 dependabot (1285)
- Bump pyo3 from 0.19.1 to 0.19.2 dependabot (1312)
- Bump pytest from 7.4.0 to 7.4.1 dependabot (1339)
- Bump actions/checkout from 3 to 4 dependabot (1337)
</details>

Changes

✨ New Features

- [FEAT] \_\_repr\_\_ for ResourceRequest xcharleslin (1343)
- [FEAT] [New Query Planner] Refactor file globbing logic by exposing `FileInfos` to Python clarkzinzow (1307)
- [FEAT] S3 Native List Impl for a directory samster25 (1324)
- [FEAT] [New Query Planner] Add support for `DropRepartition` clarkzinzow (1302)
- [FEAT] Add all projection optimization rules to new query planner. xcharleslin (1288)
- [FEAT] [New Query Planner] Add support for `PushDownLimit` clarkzinzow (1300)

👾 Bug Fixes

- [BUG] Fix Table.read\_parquet behavior when it encounters arrow\_schema jaychia (1336)
- [BUG] [New Query Planner] Revert file info partition column names. clarkzinzow (1333)
- [BUG] Fix fixed size list array FullNull implementation jaychia (1320)

🧰 Maintenance

- [CHORE] install perl before maturin samster25 (1345)
- [CHORE] Switch to openssl samster25 (1344)
- [CHORE] [New Query Planner] pyo3-agnostic `LogicalPlanBuilder`, op constructor arg orderings clarkzinzow (1332)
- [CHORE] factor io config into common code samster25 (1335)
- [CHORE] [New Query Planner] Remove `ExpressionsProjection` from builder, move validation into `Op::try_new()` clarkzinzow (1327)
- [CHORE] StructArray refactors jaychia (1326)
- [CHORE] drop flag for non native compile for daft profiling samster25 (1323)
- [CHORE] pin pyarrow to 12 for ray compat tests samster25 (1322)
- [CHORE] Move FixedSizeListArray to array/fixed\_size\_list\_array.rs jaychia (1319)
- [CHORE] Add fix for list schema inference tests using PyArrow 13.0.0 jaychia (1318)
- [CHORE] Implementations of FixedSizeListArray jaychia (1281)

⬆️ Dependencies

- Bump ray[data,default] from 2.6.0 to 2.6.3 dependabot (1315)
- Bump orjson from 3.9.4 to 3.9.5 dependabot (1316)
- Bump aws-actions/configure-aws-credentials from 2 to 3 dependabot (1317)