Daft

What's Changed 🚀

✨ Features

- feat: add dt.{millisecond,microsecond,nanosecond} universalmind303 (4128)
- feat: adds verbose and documents no format behavior rchowell (4116)
- feat: nikkthegreek adds support for creating managed tables rchowell (4121)
- feat: adds formatting arguments to show rchowell (4100)
- feat(connect): overwrite mode for writing csv \& parquet colin-ho (4089)
- feat: adds create\_table\_if\_not\_exists and create\_namespace\_if\_not\_exists rchowell (4091)
- feat: adds try\_encode and try\_decode with utf-8 special-case rchowell (4060)
- feat: adds has\_table and has\_namespace catalog methods rchowell (4088)
- feat: non-equi join in the logical plan kevinzwang (3922)
- feat: Cross column exprs colin-ho (4071)
- feat(dashboard): Add Daft CLI with dashboard command universalmind303 (4015)
- feat: Set batch size as morsel size for project colin-ho (4042)
- feat: add `llm_generate` expr universalmind303 (4009)
- feat(sql): Add basic ROLLUP support universalmind303 (4022)

🐛 Bug Fixes

- fix(join ordering): Fix bug with aliasing columns between joins desmondcheongzx (4133)
- fix: skip drop\_nan when no float columns rchowell (4127)
- fix: Disable warc merging colin-ho (4115)
- fix: Allow aliased monotonically increasing id functions to coexist with columns named "id" desmondcheongzx (4102)
- fix: Use io-runtime for task in remote parquet reader colin-ho (4110)
- fix: surface non 'NotFound' errors during table resolution rchowell (4073)
- fix: improve column error messages + add PlanRef to outer column kevinzwang (4037)
- fix: Fix local to ray runner switch test colin-ho (4066)
- fix: only install vllm in requirements-dev.txt for linux kevinzwang (4064)
- fix: Collect iterators in read\_generator to list colin-ho (4061)
- fix: dashboard flaking universalmind303 (4051)
- fix: List tables under all namespaces for iceberg `list_tables` if namespace is none colin-ho (4048)
- fix: Modify ray runner switching tests to check error in stderr instead of endswith colin-ho (4046)

🚀 Performance

- perf(optimizer): push-down join predicate kevinzwang (4123)
- perf(optimizer): simplify join types with null-eliminating filter predicates kevinzwang (4023)

📖 Documentation

- docs: Make wording around distinct more consistent desmondcheongzx (4000)
- docs: fix deltalake type docs rchowell (4119)
- docs: Add numbering rows section colin-ho (4103)
- docs: Cross col expr docs colin-ho (4104)
- docs: add S3 io\_config usage example to read\_lance docstring gabrielaugz (4087)
- docs: permalink works and takes to docs site gabrielaugz (4086)
- docs: s3tables typo fixes rchowell (4084)
- docs: add link to sql reference page inside cast universalmind303 (4054)
- docs: Adjusted SQL TYPE docu and adjusted tests accordings datanikkthegreek (4057)
- docs: fix broken links and clarify contributing rchowell (4056)

🔧 Maintenance

- chore(optimizer): move join type simplification into new rule kevinzwang (4122)
- chore: Remove ray compatibility workflow colin-ho (4068)
- chore: clean up pr template kevinzwang (4065)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.8...v0.4.9

What's Changed 🚀

✨ Features

- feat: syntactic sugar for Python list and struct gets kevinzwang (4027)
- feat: Add a memory-efficient iterator for Series desmondcheongzx (4006)
- feat(catalog): adds s3tables iceberg rest endpoint rchowell (4018)
- feat: adds gz as gzip alias for encode, decode methods rchowell (4020)
- feat: Functions: sign, signum, negative, negate petern48 (3941)
- feat(sql): namespace support with in-memory catalog rchowell (4013)
- feat(sql): adds show tables statement and documentation rchowell (4011)
- feat(catalog): adds native s3tables read and catalog apis rchowell (3929)
- feat: offset indices in sparse tensor itzhakstern (3725)
- feat: Flight shuffle colin-ho (3904)
- feat: daft.range function universalmind303 (3956)
- feat: cast using a string type universalmind303 (3951)

🐛 Bug Fixes

- fix: Fix join condition swaps when left/right sides swap desmondcheongzx (4028)
- fix: Fix boolean expression simplifier desmondcheongzx (4016)
- fix: Fix list sort with groupby desmondcheongzx (3990)
- fix: datetime deprecation universalmind303 (3987)
- fix: Fix incorrect numeric identity optimizations desmondcheongzx (3988)
- fix: tutorial code kevinzwang (3972)
- fix: allow decimal precision equal to scale rchowell (3973)
- fix: Add more retries to sql server connection in test colin-ho (3953)
- fix(ci): distributed tpch benchmark kevinzwang (3967)
- fix: depend on pylance instead of lancedb kevinzwang (3962)
- fix(ci): slack failure notification parameters kevinzwang (3952)
- fix: fix error when casting monotonically\_increasing\_id directly f4t4nt (3950)
- fix: Add target dialect when making subquery in read\_sql colin-ho (3948)
- fix: Count bytes read correctly for local WARC reads desmondcheongzx (3946)
- fix: Pass CommitProperties object custom metadata in deltalake tkauf15k (3914)
- fix: iceberg table name is a method rchowell (3949)

🚀 Performance

- perf: Enable join reordering colin-ho (4029)
- perf: Favor smaller relations on the left for join ordering desmondcheongzx (4003)
- perf: Refactor selectivity estimates colin-ho (4010)

📖 Documentation

- docs: update install instructions for daft-lts and nightly kevinzwang (4026)
- docs: Fix s3 tables docs colin-ho (4025)
- docs: change all mentions of getdaft -> daft jaychia (3986)
- docs: fix docs examples and add missing docs kevinzwang (3974)
- docs: initializes sql and data type documentation rchowell (3959)

👷 CI

- ci: update distributed tpch benchmark kevinzwang (3971)
- ci: fix typo in nightly workflow kevinzwang (3968)
- ci: distributed TPC-H benchmarks kevinzwang (3961)

🔧 Maintenance

- chore: Track imports on scarf colin-ho (4024)
- chore: Upgrade kanal to 0.1 colin-ho (4017)
- chore: create in-memory scans using rust arrow arrays rchowell (4005)
- chore(dashboard): update Next.js dependency to version 15.2.2 universalmind303 (3999)
- chore: add pr template ccmao1130 (3981)
- chore: dashboard build cleanup universalmind303 (3931)
- chore: fix slack link in readme kevinzwang (3966)
- chore: Favor OnceLock over lazy\_static for WARC column sizes desmondcheongzx (3939)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.7...v0.4.8

What's Changed 🚀

- build: build and publish daft package kevinzwang (3913)
- build: bump rust toolchain version kevinzwang (3910)

✨ Features

- feat: adds encode and decode for deflate, gzip, zlib rchowell (3907)
- feat(catalog): adds catalog ddl actions like create\_table and create\_namespace rchowell (3902)
- feat(sql): adds the 'use' sql session statement rchowell (3912)
- feat(catalog): adds append and overwrite to table apis rchowell (3889)
- feat(catalog): adds additional table sources for Catalog.from\_pydict rchowell (3901)
- feat: functions sinh, cosh, tanh petern48 (3903)
- feat: Functions log1p and expm1 petern48 (3887)
- feat: trig functions csc and sec petern48 (3884)

🐛 Bug Fixes

- fix: nightly build and local tpch benchmark workflow kevinzwang (3898)
- fix: add retry to getting GCS client config kevinzwang (3930)
- fix: bun install in build-wheel.yml kevinzwang (3932)
- fix: allow resolving tables at catalog root rchowell (3928)
- fix: Don't use `_position_to_field_name` Fokko (3917)
- fix: write\_lance append mode when storage\_options required ascillitoe (3924)
- fix(dashboard): get dashboard working again universalmind303 (3918)
- fix: coalesce panics, supertype handling, and null handling bugs rchowell (3908)
- fix: small fix for pyspark+ray. universalmind303 (3899)
- fix: map.get on empty dataset universalmind303 (3892)
- fix: remove dashboard imports and dep samster25 (3888)

🚀 Performance

- perf: Reduce memory consumption for WARC reads and improve estimates desmondcheongzx (3935)

📖 Documentation

- docs: adds additional catalog and session documentation rchowell (3926)
- docs: add spark connect doc page universalmind303 (3919)
- docs: adds a usage doc for catalogs rchowell (3878)
- docs: Add documentation for functions module f4t4nt (3880)
- docs: remove cairo ccmao1130 (3900)

👷 CI

- ci: update all --release workflows universalmind303 (3915)
- ci: replace build-artifact-s3 with new workflow, add local tpch benches kevinzwang (3864)

🔧 Maintenance

- chore: use ref name instead of ref in tpch bench metadata kevinzwang (3937)
- chore: use stdlib importlib.metadata for python>3.9 kevinzwang (3916)
- chore: move dashboard in to main project universalmind303 (3909)
- chore: make dashboard assets part of build process. universalmind303 (3905)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.6...v0.4.7

What's Changed 🚀

✨ Features

- feat: Add WARC reader desmondcheongzx (3871)
- feat(functions): add monotonically\_increasing\_id expression function f4t4nt (3838)
- feat: union ops universalmind303 (3872)
- feat: Enable capturing and broadcasting logs when running on the `Native` runner raunakab (3875)
- feat(connect): joins universalmind303 (3849)

🐛 Bug Fixes

- fix: Add check for numpy in from\_pylist colin-ho (3881)
- fix: Fix ray data link colin-ho (3874)
- fix: arrow to Series for nested map array kevinzwang (3870)
- fix: Add metadata to subgraph options in python colin-ho (3869)
- fix: Update dashboard import raunakab (3865)

🚀 Performance

- perf: Clear task inputs upon dispatch colin-ho (3877)
- perf: Fix join cost estimates desmondcheongzx (3831)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.5...v0.4.6

What's Changed 🚀

💥 Breaking Changes

- refactor!: split column expression into unresolved and resolved types kevinzwang (3804)

✨ Features

- feat(connect): `daft.pyspark` module universalmind303 (3861)
- feat: Emit children of join before shuffle + add stats to explain analyze colin-ho (3852)
- feat: Stageify plan on shuffle boundaries colin-ho (3781)
- feat(sql): adds session sql for leveraging attached catalogs rchowell (3860)
- feat(catalog): Cutover deprecated APIs to use session, catalog, table abstractions [3/3] rchowell (3830)
- feat(connect): read csv/parquet/json options universalmind303 (3791)
- feat(sql): select from multiple joins kevinzwang (3842)
- feat(catalog): Integrate session and catalog actions alongside existing APIs [2/3] rchowell (3825)
- feat(catalog): Prepare existing catalog APIs for integration [1/3] rchowell (3820)
- feat(sql): supports schemas in read\_json, read\_csv, read\_parquet rchowell (3836)
- feat(sql): supports array of paths in read\_ table-value functions rchowell (3835)
- feat: Add a daft dashboard to display queries plans and stats raunakab (3790)

🐛 Bug Fixes

- fix: sql round without precision universalmind303 (3863)
- fix: pypi publish workflow kevinzwang (3862)
- fix: build wheel Github action inputs kevinzwang (3858)
- fix: protocol in iceberg writes colin-ho (3851)
- fix: LogicalPlan::get\_schema\_for\_alias should stop when it hits any alias kevinzwang (3848)
- fix: Reduce number of nodes in random join graph test desmondcheongzx (3839)
- fix: Add excludes to broken link checker colin-ho (3834)
- fix: Grab Daft config from environment variables for new contexts desmondcheongzx (3832)
- fix: create series of np.datetime64['D'] rchowell (3829)

🚀 Performance

- perf(optimizer): Infer additional join graph edges during join reordering desmondcheongzx (3807)

♻️ Refactor

- refactor!: split column expression into unresolved and resolved types kevinzwang (3804)

📖 Documentation

- docs: respect daft analytics env var ccmao1130 (3856)
- docs: Update configuration docs to show `set_runner_native` colin-ho (3833)

🔧 Maintenance

- chore: replace anaconda with S3 for nightly build publish kevinzwang (3857)
- chore: minor cleanup to table-value functions rchowell (3854)
- chore: remove accidental printlins universalmind303 (3845)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.4...v0.4.5

What's Changed 🚀

- build: update python-publish workflow ccmao1130 (3797)
- build(docs): fix docgen failed workflow ccmao1130 (3766)

✨ Features

- feat: Adds .summarize() to compute statistics rchowell (3810)
- feat(sql): SELECT without FROM rchowell (3814)
- feat: Simplify is ins to an OR chain of eqs colin-ho (3800)
- feat(session): Adds session class to python rchowell (3809)
- feat(session): Replaces direct usage of DaftCatalog with Session rchowell (3794)
- feat: Sequentially materialize left and right sides during hash join colin-ho (3735)
- feat(connect): add temporal functions universalmind303 (3799)
- feat: nulls first kernels universalmind303 (3789)
- feat(table): implement list\_unique and Set aggregation f4t4nt (3710)
- feat: add functions to daft-connect universalmind303 (3780)
- feat(catalog): Defines a session for connection state rchowell (3782)
- feat: implement bool\_and and bool\_or f4t4nt (3754)
- feat(catalog): Defines an identifier for use across catalogs rchowell (3763)
- feat(optimizer): Brute force join ordering desmondcheongzx (3688)
- feat(swordfish): Properly buffer unordered scan tasks colin-ho (3751)
- feat: better sql datatype support universalmind303 (3750)
- feat: Adds list constructor to Expression and SQL APIs rchowell (3737)
- feat: spark connect set operations universalmind303 (3739)
- feat: add spark explain universalmind303 (3741)

🐛 Bug Fixes

- fix: unity managed table reads pmogren (3806)
- fix: boolean casts to strings and null propagation rchowell (3770)
- fix: catalog table names universalmind303 (3760)

🚀 Performance

- perf(swordfish): Parallel expression evaluation colin-ho (3593)
- perf: Use parquet metadata from schema inference for accurate scan task statistics desmondcheongzx (3784)

♻️ Refactor

- refactor: rename `table` to `recordbatch` universalmind303 (3771)
- refactor: port DaftContext to rust side universalmind303 (3767)
- refactor: renames to\_struct to just struct rchowell (3755)

📖 Documentation

- docs: fix readthedocs build ccmao1130 (3824)
- docs: add scarf analytics ccmao1130 (3773)
- docs: Update distributed docs to add byoc mode, change name to daft cli jessie-young (3768)
- docs: update README.rst diagram ccmao1130 (3803)
- docs: update links in readme ccmao1130 (3779)
- docs: add footer and update broken links ccmao1130 (3764)

👷 CI

- ci: Allow TPCH benchmarks to use ARM cluster profile desmondcheongzx (3777)
- ci: Record info for TPCH benchmarks desmondcheongzx (3729)
- ci: send slack notification for broken links ccmao1130 (3742)

**Full Changelog**: https://github.com/Eventual-Inc/Daft/compare/v0.4.3...v0.4.4