Duckdb

This preview release of DuckDB is named "Spectabilis" after the [King Eider](https://en.wikipedia.org/wiki/King_eider)

Binary builds are listed below. Feedback is very welcome.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the [documentation](https://duckdb.org/docs/sql/statements/export) for details.

Below a list of changes in this release

**Features**
- 2393: Switch to Push-Based Execution Model
- 2417: Add support for GROUPING SETS, ROLLUP, CUBE and GROUPING/GROUPING_ID
- 2347: Support for ENUM Types & 2404: Native mapping between R factors and DuckDB ENUMs
- 2419: Allow WHERE clause referring aliases defined in SELECT clause
- 2473: Adding Compression Option for Column Definitions to SQL Parser
- 2489: Implement MAD (Moving Absolute Deviation) Aggregate
- 2520: Add LIST_CONCAT and LIST_APPEND functions
- 2522: MSD (Most Significant Digit) Radix Sort
- 2529: Implement REGEXP_EXTRACT
- 2482: Add support for EVEN
- 2555: Adding BINARY_AS_STRING parameter to the parquet scan

**Minor Changes & Bug Fixes**
- 2377: Fix current_schema() and current_schemas()
- 2380: Add tests for "did you mean" error message
- 2381: Let FTS index creation respect the current schema.
- 2382: Dropping support to Python 2
- 2385: BLOB support for JDBC
- 2389: Replace __getattr__ on PyBind11 classes with individual properties.
- 2390: Moving big categorical tests to tests_slow folder
- 2394: Add SET s3_endpoint support for in-house Ceph
- 2397: Add Python .pyi stubs
- 2402: GH Actions Upload: retry asset upload with timeout
- 2403: Not altering original DF when renaming columns in the binder
- 2405: Fix bug with enum::varchar cast on null values
- 2408: Rewrite Arrow table register for R using replacement scans
- 2409: Adding dependency on Enum Types -> Tables
- 2412: Updated src readme to state we use push-based execution.
- 2413: Fix for 2411
- 2421: Fix 2407: use correct template parameters for DATE in arg_min/arg_max
- 2423: Fix 2416: fix binding issues related to binding parameters, null values, etc in list_extract and array_length
- 2424: Issue 2388: QUANTILE_DISC for VARCHAR
- 2430: Fix for 2426
- 2431: More fixes for 2416
- 2434: Issue 2388: Moving VARCHAR QUANTILE_DISC
- 2437: implement GEN_RANDOM_UUID
- 2438: Issue 2432: PERCENTILE_XXXX ignores DESC
- 2439: fix: pass absolute path to `System.load()` in Java
- 2444: Fix 2440: correctly report run-time errors in Python client
- 2445: Several OSS Fuzz Fixes
- 2448: Move to codecov v2, and use add_library for vector operations for low RAM machines
- 2449: Enum to Enum Comparisons
- 2451: Hooold the loooock for Python Strings under Dataframe Object Columns
- 2453: Fix when getting single values from ENUMs
- 2455: Doc Improve: Trying to Update doc in `QueryResult`
- 2456: Add --test-dir parameter to unittest so we can test out-of-tree extensions with it
- 2462: Fix 2452: Implement Coalesce instead of rewriting to CASE chain
- 2463: More OSS Fuzz fixes
- 2474: Allow Fetch of chunks containing a multiple of vector sizes for the Arrow Record Batch Reader
- 2476: Really holding the lock this time
- 2477: Sorted aggregate: only re-order when ordering count > 0
- 2485: Benchmarks: Handle comparisons for values that do not have a VARCHAR ->TYPE cast (e.g. complex/list types)
- 2487: Conditionally define UNLIKELY in Thrift
- 2488: ODBC: fetching the first chunk in SQLExecute
- 2490: Clean up RadixSort code
- 2491: These tests are now passing with arrow 6
- 2494: Emit full vectors from VALUES lists instead of emitting individual tuples
- 2497: Upgrade Catch to v2.13.7
- 2500: Fix nested string order
- 2501: Add CIFuzz action
- 2503: Fix for py string conversion on large strings
- 2504: More precision for `SUM` and `AVG`
- 2517: Move Kahan sum to separate method (fsum, sum_kahan)
- 2521: Issue 2515: Windowed quantile list
- 2527: testing: Add oss-fuzz fuzzer
- 2536: Fix 2518: in read_csv_auto don't override names if names have been provided
- 2537: Fix 2531: in recursive CTE avoid waiting for events to finish if an event has thrown an error
- 2539: Restructuring CI Workflow
- 2542: Issue 2530: Reset windowed lists
- 2550: ODBC: Running PSQLODBC tests on Win64
- 2556: Fixed directory separator bug
- 2558: Fixing positional reference binding in ORDER BY clause
- 2559: Issue 2552: General ordered aggregates
- 2561: Fix master CI: Python workflow needs auth tokens for deployment
- 2563: Move ExtensionHelper into main DuckDB Class
- 2564: R Client: Moving UTF encoding to R to avoid multithreading issues
- 2567: Fix 2538: crash in CSV auto-detect when reading ZSTD data
- 2573: Move HTTPFS builds to separate test to avoid deploying them by default on Linux
- 2574: Naming optimizer-created aggregates so plans are interpretable
- 2579: Support to Arrow 6
- 2580: Check magic bytes before checksum when opening a DuckDB database file
- 2585: Fix 2584: correctly handle edge cases for bigger than 1 increments in range table function
- 2587: Replacement Scans for Arrow Objects
- 2592: Fix 2577: Rework case statement to avoid rewrite into nested binary cases
- 2593: Fix 2591: avoid using ungrouped aggregate for non-combineable aggregates
- 2594: Fix 2588: timestamp -> date cast is not invertible
- 2595: Fix 2543: case insensitive replacement scans
- 2598: OSS Fuzz Fixes
- 2603: The asset upload script for releases was broken somehow
- 2605: Clean up reupload by splitting it into two functions
- 2604: Fix 2599: maintain correct dependencies between UNION ALL nodes so that output is deterministic/in-line with what a sequential execution would produce

This preview release of DuckDB is named "Gracilis" after the [Grey Teal](https://en.wikipedia.org/wiki/Grey_teal)

Binary builds are listed below. Feedback is very welcome.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the [documentation](https://duckdb.org/docs/sql/statements/export) for details.

This release contains a novel join method, the 1959: Perfect Hash Join. Otherwise there are many improvements and bug fixes however, a list is below:

- 2377: Fix current_schema() and current_schemas()
- 2371: Installing pandoc so R pkg can be fully checked
- 2369: More CI fixes
- 2361: ODBC: First version Winsetup
- 2360: Issue 2348: Support bankers rounding as default function
- 2359: Implement UUID data type
- 2358: Fixes minor flag comment
- 2357: Issue 2351: Ordered aggregate transformation
- 2356: Issue 2286: Numeric SUMMARIZE stats
- 2353: Get rid of a couple of Clang warnings.
- 2346: Avoid re-using RE2 regex between threads, since RE2 regex objects have locks internally
- 2328: CI Fixes
- 2321: Issue 1998: Exact Timestamp Subtraction
- 2320: Change SET's default scope from GLOBAL to (PG-compatible) SESSION.
- 2319: Fix 2315: correctly detect that index joins cannot be used for multi-column indexes, and clean up TPC-DS extension
- 2318: Remove extra semicolon.
- 2317: Support SET SCHEMA and SET SEARCH_PATH
- 2316: Fix for 2304
- 2314: Fix 2313: No Out of Range Error in Index Benchmark
- 2311: Issue 2310: Create Index Benchmark
- 2306: Windows (64 bits) and -DDISABLE_UNITY=1
- 2302: Fix 2301: ART Leaf Node shrink
- 2300: Fix 2293: Correctly escape all special characters (quotes, newlines, tabs, etc) in JSON output of query profiler
- 2299: Fix 2296: Avoid requesting O_RDWR permissions when we only need O_WRONLY so we can write to FIFO streams
- 2298: Fix 2294: In CSV reader correctly generate column names with many columns
- 2297: Issue 2241: Transacted Index Reinsert
- 2290: Fix 2289: align default continue prompt
- 2287: ODBC: list catalog, schema and test_blob
- 2284: Make S3 credentials session scoped
- 2282: Fix for bug 2281
- 2280: Initial support to read lists from R client
- 2279: Fix 2277: add support for lists of structs to LIST_EXTRACT
- 2276: Add support for EXCLUDE and REPLACE clauses
- 2275: Run only CRAN tests for valgrind
- 2274: Fix 2267: For structs, get the required amount of rows from a non-list child if there is one
- 2272: Rename force_parallelism to verify_parallelism
- 2271: Make regression run parallel with 2 Threads
- 2266: A refactoring around FileSystem
- 2265: Allow optional extensions when building the R package
- 2263: Regression Test: TPC-DS/H20AI and other adjustments
- 2262: Fix 2261: add support for filters pushed down into decimal columns in Parquet files
- 2259: Fix a TSAN error for test/sql/window/test_partition_flushing.test
- 2255: Make sorting even faster
- 2254: More descriptive out-of-memory error when db is launched in-memory
- 2253: Regression Tests
- 2251: Changes from CRAN 0.2.9 release
- 2234: Add support for S3 session token (STS)
- 2228: Date functions on Windows

This preview release of DuckDB is named "Platyrhynchos" after the very well-known [Mallard](https://en.wikipedia.org/wiki/Mallard), πλατυρυγχος meaning "broad-billed".

Binary builds are listed below. Feedback is very welcome.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the [documentation](https://duckdb.org/docs/sql/statements/export) for details.

Major changes (not listing minor bug fixes and small changes):

Storage
- 2099: **Persistent Storage Compression Framework** & RLE compression for numeric columns
- 2157: Multi-Occupancy Blocks in the Storage


SQL
- 2247: Support session and global `SET` statement variable scopes.
- 2230: Full Case Insensitivity in the Binder & Catalog
- 2208: Allow duplicate column names in subqueries/`CREATE TABLE AS`, similar to SQLite
- 2206: Allow unnamed subqueries
- 2189: Add support for `SUMMARIZE` of SQL statement/table
- 2158: `ORDER BY` aggregates
- 2171: Implement `LAST`
- 2146: Support for `RANGE` and `ARRAY_LENGTH` scalar functions
- 2133: Implement `DATESUB` & 2090: Implement `DATEDIFF`
- 2128: Fall back to VARCHAR when the type of a prepared statement parameter is ambiguous

Performance Improvements
- 2226: Issue 1657: Reduce Window copying
- 2221: Prefetch Parquet Meta Data
- 2172: Top-N Rework using new sort code
- 2167: Use new sort code in window functions
- 2098: Reduce unnecessary sorting overhead
- 2077: Fetch Arrow - Streaming

C Client:
- 2173: Don't strdup error msg
- 2115: Improve performance of duckdb_value functions, add support for HUGEINT type and add extra result helper functions
- 2107: CAPI Cleanup & DATE/TIME/TIMESTAMP rework

R Client
- 2250: Support column subsets in dbAppendTable()
- 2134: Implement dbAppendTable()
- 2136: Implement dbBind() according to specification
- 2119: Enable more DBItest tests

ODBC Client
- 2184: Enabling pyodbc & 2124: Enable nanodbc & 2159: Making the R odbc package work with our ODBC driver
- 2169: Supporting SQLPutData, SQLParamData, SQLMoreResults & - 2096: SQLFetchScroll and SQLGetPrivateProfileString
- 2236: Windows x64 Support

This preview release of DuckDB is named "Ceruttii" after a [long-extinct relative of the present-day Harleqin Duck](https://en.wikipedia.org/wiki/Harlequin_duck#Taxonomy) (Histrionicus Ceruttii).
Binary builds are listed below. Feedback is very welcome.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the [documentation](https://duckdb.org/docs/sql/statements/export) for details.

SQL

- 2064: `RANGE`/`GENERATE_SERIES` for timestamp + interval
- 1905: Add `PARQUET_METADATA` and `PARQUET_SCHEMA` functions
- 2059, 1995, 2020 & 1960: Window `RANGE` framing, `NTH_VALUE` and other improvements

APIs

- [Many Arrow integration improvements](https://github.com/duckdb/duckdb/pulls?q=is%3Apr+is%3Aclosed+arrow)
- [Many ODBC driver improvements](https://github.com/duckdb/duckdb/pulls?q=is%3Apr+is%3Aclosed+odbc)
- 1815: Initial version: SQLite UDF API
- 2001: Support DBConfig in C API

Engine

- 1975, 1876 & 2009: Unified row layout for sorting, aggregate & joins
- 1930 & 1904: List Storage
- 2050: CSV Reader/Casting Framework Refactor & add support for `TRY_CAST`
- 1950: Add Constant Segment Compression to Storage
- 1957: Add pipe/stream file system

This preview release of DuckDB is named "Mollissima" after the Common Eider (Somateria mollissima).
Binary builds are listed below. Feedback is very welcome.

Note: This release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the documentation for details.

Major changes:

SQL
- 1847: Unify catalog access functions, and provide views for common PostgreSQL catalog functions
- 1822: Python/JSON-Style Struct & List Syntax
- 1862: 1584 Implementing `NEXTAFTER` for float and double
- 1860: `FIRST` implementation for nested types
- 1858: `UNNEST` table function & array syntax in parser
- 1761: Issue 1746: Moving `QUANTILE`

APIs

- 1852, 1840, 1831, 1819 and 1779: Improvements to Arrow Integration
- 1843: First iteration of ODBC driver
- 1832: Add visualizer extension
- 1803: Converting Nested Types to native python
- 1773: Add support for key/value style configuration, and expose this in the Python API

Engine
- 1808: Row-Group Based Storage
- 1842: Add (Persistent) Struct Storage Support
- 1859: Read and write atomically with offsets
- 1851: Internal Type Rework
- 1845: Nested join payloads
- 1813: Aggregate Row Layout
- 1836: Join Row Layout
- 1804: Use Allocator class in buffer manager and add a test for a custom allocator usage

This preview release of DuckDB is named "Jamaicensis" after the [blue-billed Ruddy Duck (Oxyura jamaicensis)](https://en.wikipedia.org/wiki/Ruddy_duck). Binary builds are listed below. Feedback is very welcome.

Note: This release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the documentation for details.

Also note: Due to changes in the internal storage (1530), databases created with this release wil require somewhat more disk space. This is transient as we are working hard to finalise the on-disk storage format.

Major changes:

Engine
- 1666: External merge sort, 1580: Parallel scan of ordered result and 1561: Rework physical ORDER BY
- 1520 & 1574: Window function computation parallelism
- 1540: Add table functions that take a subquery parameter
- 1533: Using vectors, instead of column chunks as lists
- 1530: Store null values separate from main data in a Validity Segment

SQL
- 1568: Positional Reference Operator `1` etc.
- 1671: `QUANTILE` variants and 1685: Temporal quantiles
- 1695: New Timestamp Types `TIMESTAMP_NS`, `TIMESTAMP_MS` and `TIMESTAMP_NS`
- 1647: Add support for UTC offset timestamp parsing to regular timestamp conversion
- 1659: Add support for `USING` keyword in `DELETE` statement
- 1638, 1663, 1621 & 1484: Many changes arount `ARRAY` syntax
- 1610: Add support for `CURRVAL`
- 1544: Add `SKIP` as an option to `READ_CSV` and `COPY`

APIs
- 1525: Add loadable extensions support
- 1711: Parallel Arrow Scans
- 1569: Map-style UDFs for Python API
- 1534: Extensible Replacement Scans & Automatic Pandas Scans and 1487: Automatically use parquet or CSV scan when using a table name that ends in `.parquet` or `.csv`
- 1649: Add a QueryRelation object that can be used to convert a query directly into a relation object, 1665: Adding from_query to python api
- 1550: Shell: Add support for Ctrl + arrow keys to linenoise, and make Ctrl+C terminate the current query instead of the process
- 1514: Using `ALTREP` to speed up string column transfer to R
- 1502: R: implementation of Rstudio connection-contract tab