Dlt

Core Library
* feat: add incremental lag (attribution window) for datetime, int, and float cursors by donotpush in https://github.com/dlt-hub/dlt/pull/1957
* LanceDB - (1) support merge key to merge chunked documents correctly - removes orphaned chunks (2) huge performance upgrade by loading data via arrow by Pipboyguy in https://github.com/dlt-hub/dlt/pull/1620
* Move `exclude_keys()` to `dlt.common.utils` by burnash in https://github.com/dlt-hub/dlt/pull/1966
* Fix BigQueryLoadJob hiding root cause exception by xneg in https://github.com/dlt-hub/dlt/pull/1992
* loads secrets from colab userdata and steamlit + bugfixes by rudolfix in https://github.com/dlt-hub/dlt/pull/1994
* Fix pagination issue in `JSONResponseCursorPaginator` with empty string cursor value by kang8 in https://github.com/dlt-hub/dlt/pull/2016
* fix: if name of distribution is None by senickel in https://github.com/dlt-hub/dlt/pull/2024
* allows to pass default values when writing specs by rudolfix in https://github.com/dlt-hub/dlt/pull/2018
* enable `delta` partitioning on arrow normalizer load id by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/2022
* add session token to duckdb s3 secret by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/2007
* Add user agent for Databricks by VioletM in https://github.com/dlt-hub/dlt/pull/1987
* Fix an incorrect missing dependency error by burnash in https://github.com/dlt-hub/dlt/pull/2001
* fix resource level max_table_nesting and normalizer performance tuning by sh-rp in https://github.com/dlt-hub/dlt/pull/2026
* move default pipelines of cores sources into source folders by sh-rp in https://github.com/dlt-hub/dlt/pull/1888
* duckdb filesystem custom secrets by sh-rp in https://github.com/dlt-hub/dlt/pull/2017
* allows for empty dataset clickhouse by rudolfix in https://github.com/dlt-hub/dlt/pull/2045
* add GCP default credential handling for `delta` table format by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/2048
* enables merges for bigquery autodetect schema by sh-rp in https://github.com/dlt-hub/dlt/pull/2035
* logs warning if deduplication state is large by willi-mueller in https://github.com/dlt-hub/dlt/pull/1877
* Add core sources extras to requirements in `dlt init` by burnash in https://github.com/dlt-hub/dlt/pull/2028
* Fix merge write disposition for pyarrow and ClickHouse by burnash in https://github.com/dlt-hub/dlt/pull/2042
Experimental interfaces
dlt dataset public interface and docs coming next week.
* 1990 - dataset columns select and limit by sh-rp in https://github.com/dlt-hub/dlt/pull/2000

Docs
* Updated databricks destination documentation by dat-a-man in https://github.com/dlt-hub/dlt/pull/1984
* Docs: fix capitalization of some terms, fix typos by burnash in https://github.com/dlt-hub/dlt/pull/1988
* fix typo by mariarice15 in https://github.com/dlt-hub/dlt/pull/1995
* Fix Zendesk example: make test resilient to data changes by burnash in https://github.com/dlt-hub/dlt/pull/1999
* fix s3 credentials environment variable names by seunggs in https://github.com/dlt-hub/dlt/pull/2010
* remove ga add tm by alexanderfifefd in https://github.com/dlt-hub/dlt/pull/2008
* Super fast snippet linting & type checking by sh-rp in https://github.com/dlt-hub/dlt/pull/2019
* Fix the deprecation warning in `.common.configuration.container` by burnash in https://github.com/dlt-hub/dlt/pull/2025
* Added deploy with modal. by dat-a-man in https://github.com/dlt-hub/dlt/pull/1805
* Updated google cloud function documentation by dat-a-man in https://github.com/dlt-hub/dlt/pull/2034
* add warning for large delta memory footprint on filesystem docs page by sh-rp in https://github.com/dlt-hub/dlt/pull/2036
* simplify advanced section by kning in https://github.com/dlt-hub/dlt/pull/2037
* Added docs on how to deploy a pipeline using Google Cloud run by dat-a-man in https://github.com/dlt-hub/dlt/pull/2038
* Format Delta table section in the filesystem destination by burnash in https://github.com/dlt-hub/dlt/pull/2057
* Docs: add table formats to the sidebar by burnash in https://github.com/dlt-hub/dlt/pull/2060

New Contributors
* xneg made their first contribution in https://github.com/dlt-hub/dlt/pull/1992
* seunggs made their first contribution in https://github.com/dlt-hub/dlt/pull/2010
* alexanderfifefd made their first contribution in https://github.com/dlt-hub/dlt/pull/2008
* kang8 made their first contribution in https://github.com/dlt-hub/dlt/pull/2016
* senickel made their first contribution in https://github.com/dlt-hub/dlt/pull/2024
* kning made their first contribution in https://github.com/dlt-hub/dlt/pull/2037

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/1.3.0...1.4.0

Core Library
* Fix try/except in from_reference shadowing MissingDependencyException by burnash in https://github.com/dlt-hub/dlt/pull/1939
* prefers uv over pip if found (when creating virtual envs) by rudolfix in https://github.com/dlt-hub/dlt/pull/1940
* allows to plug new or updated dlt cli commands by sh-rp in https://github.com/dlt-hub/dlt/pull/1938
* Feat/557 rest api add oauth2clientcredentials to built in auth methods by willi-mueller in https://github.com/dlt-hub/dlt/pull/1871
* uses path normalize for columns in arrow tables by rudolfix in https://github.com/dlt-hub/dlt/pull/1947
* Added extended jsonpath_ng parser (rest_api) by francescomucio in https://github.com/dlt-hub/dlt/pull/1941
* Fix/1897 support https endpoints clickhouse by sh-rp in https://github.com/dlt-hub/dlt/pull/1931
* Fix for multiple ignores is not working (rest_api) by burnash in https://github.com/dlt-hub/dlt/pull/1956
* SQL Database: Support including/excluding NULL cursor values by steinitzu in https://github.com/dlt-hub/dlt/pull/1946
* Add `references` table hint and reflect them in `sql_database` by steinitzu in https://github.com/dlt-hub/dlt/pull/1925
* only truncate or delete from existing tables in refresh modes by sh-rp in https://github.com/dlt-hub/dlt/pull/1926
* adds bigquery partition expiration and motherduck connection string by rudolfix in https://github.com/dlt-hub/dlt/pull/1968

Experimental interfaces
Below we expose a new `pipeline._dataset` and `dlt._dataset` interfaces that provide unified access to data loaded into destination. We also implement `duckdb`-based SQL client on a `filesystem` destination to access data in data lakes. We'll add documentation once we stabilize dataset interface. However already now you can benefit from new `cursor` implementation of `sql_client` that allows to take data frames, arrow tables also in batches:
* dataset factory by sh-rp in https://github.com/dlt-hub/dlt/pull/1945
* expose readable datasets as dataframes and arrow tables by sh-rp in https://github.com/dlt-hub/dlt/pull/1507

PRs below adds `pluggy` and a few first plugin hooks. The idea is to make a lot of functionalities in dlt pluggable. Currently you can plug new cli command (or upgrade existing) and you can also plug your own runtime environment (how dlt looks for data, secrets etc.)
* adds registries and plugins by rudolfix in https://github.com/dlt-hub/dlt/pull/1894
* unifies run configuration and run context by rudolfix in https://github.com/dlt-hub/dlt/pull/1944

Docs
* Update url in deploy-with-airflow-composer.md by FriedrichtenHagen in https://github.com/dlt-hub/dlt/pull/1942
* Added info about backend kwargs in pyarrow by dat-a-man in https://github.com/dlt-hub/dlt/pull/1903
* Docs: sync styles with dlthub by burnash in https://github.com/dlt-hub/dlt/pull/1936
* Docs: styles: remove underline for cards in dark mode by burnash in https://github.com/dlt-hub/dlt/pull/1967


New Contributors
* FriedrichtenHagen made their first contribution in https://github.com/dlt-hub/dlt/pull/1942

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/1.2.0...1.3.0

Core Library
* Sqlalchemy merge support by steinitzu in https://github.com/dlt-hub/dlt/pull/1842
* Fix config sections for synching destinations and accessing destination clients by sh-rp in https://github.com/dlt-hub/dlt/pull/1887
* incremental `scd2` with `merge_key` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1818
* fix: UUIDs are not an unknown data type (logging) by neuromantik33 in https://github.com/dlt-hub/dlt/pull/1914
* fix: PageNumberPaginator not reset when iterating through multiple pa… by paul-godhouse in https://github.com/dlt-hub/dlt/pull/1924
* Feat/1922 rest api source add mulitple path parameters by TheOneTrueAnt in https://github.com/dlt-hub/dlt/pull/1923
* enables gcs staging for databricks by rudolfix in https://github.com/dlt-hub/dlt/pull/1933

Docs
* Update weaviate reference by emmanuel-ferdman in https://github.com/dlt-hub/dlt/pull/1896
* Docs: Add sftp option for filesystem source by VioletM in https://github.com/dlt-hub/dlt/pull/1845
* Update installation.md by erikjamesmason in https://github.com/dlt-hub/dlt/pull/1899
* Added troubleshooting section to filesystem docs by dat-a-man in https://github.com/dlt-hub/dlt/pull/1900
* Docs: make naming consistent in the cloud storage & file system source by burnash in https://github.com/dlt-hub/dlt/pull/1835
* Docs: add section on resolving multiple path parameters by burnash in https://github.com/dlt-hub/dlt/pull/1929


New Contributors
* emmanuel-ferdman made their first contribution in https://github.com/dlt-hub/dlt/pull/1896
* erikjamesmason made their first contribution in https://github.com/dlt-hub/dlt/pull/1899
* neuromantik33 made their first contribution in https://github.com/dlt-hub/dlt/pull/1914
* paul-godhouse made their first contribution in https://github.com/dlt-hub/dlt/pull/1924

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/1.1.0...1.2.0

What's Changed
* fix intermittent `delta` panic issue by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1832
* Sqlalchemy staging dataset support and docs by steinitzu in https://github.com/dlt-hub/dlt/pull/1841
* rest_api: allow specifying custom session (feat/1843) by willi-mueller in https://github.com/dlt-hub/dlt/pull/1844
* Allows any duckdb version, fixes databricks az credentials by rudolfix in https://github.com/dlt-hub/dlt/pull/1854
* Fix/1849 Do Not Parse Ignored Empty Responses by TheOneTrueAnt in https://github.com/dlt-hub/dlt/pull/1851
* feat: filesystem delete old pipeline state files by donotpush in https://github.com/dlt-hub/dlt/pull/1838
* supports adding DltResource in RESTAPIConfig dict by willi-mueller in https://github.com/dlt-hub/dlt/pull/1865
* Fix/1858 make all connection string credentials optional by rudolfix in https://github.com/dlt-hub/dlt/pull/1867

Docs
* sqlalchemy destination docs steinitzu in https://github.com/dlt-hub/dlt/pull/1841
* Docs: move REST API helpers to the REST API category by burnash in https://github.com/dlt-hub/dlt/pull/1852
* Docs: rest_api: document `processing_steps` by burnash in https://github.com/dlt-hub/dlt/pull/1872
* Fix the paginator's doc heading by burnash in https://github.com/dlt-hub/dlt/pull/1869

Verified Sources
* Custom filter clauses supported, pyarrow/arrowmongo requirement optional for Mongo by Pipboyguy

New Contributors
* TheOneTrueAnt made their first contribution in https://github.com/dlt-hub/dlt/pull/1851

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/1.0.0...1.1.0

**This is a major `dlt` release**. Please check the list of breaking changes and deprecations: https://github.com/dlt-hub/dlt/issues/1778

Core Library
* move rest_api, sql_database and filesystem sources to dlt core by willi-mueller in https://github.com/dlt-hub/dlt/pull/1728
* drops `foreign_key`, adds nested references (`row_key` - `parent_key`) by rudolfix in https://github.com/dlt-hub/dlt/pull/1774
* deprecates `complex` data type, changes to `json` by rudolfix in https://github.com/dlt-hub/dlt/pull/1792
* Feat/1749 abort load package and raise exception on terminal errors in jobs by willi-mueller in https://github.com/dlt-hub/dlt/pull/1781
* Feat/1492 extend timestamp config to handle naive timestamps (without timezone) by donotpush in https://github.com/dlt-hub/dlt/pull/1669
* Fix/1571 Incremental: Optionally load or ignore/exclude/include records with `cursor_path` missing or None value by willi-mueller in https://github.com/dlt-hub/dlt/pull/1576
* creates a single source in extract for all resource instances passed as list by rudolfix in https://github.com/dlt-hub/dlt/pull/1535
* Enable BigQuery schema auto-detection with partitioning and clustering hints by Pipboyguy in https://github.com/dlt-hub/dlt/pull/1806
* Sqlalchemy destination (merge support and docs still in progress) by steinitzu in https://github.com/dlt-hub/dlt/pull/1734
* Feat/1730 extend filesystem sftp by donotpush in https://github.com/dlt-hub/dlt/pull/1769
* Stops dumping secrets to dlt traces. by willi-mueller in https://github.com/dlt-hub/dlt/pull/1797
* Don't use Custom Embedding Functions on LanceDB by Pipboyguy in https://github.com/dlt-hub/dlt/pull/1771
* sets default concurrency for blob upload for adlfs to 1 to avoid massive memory usage on large files by rudolfix in https://github.com/dlt-hub/dlt/pull/1779
* Fix/1790 support incremental load with arrow when cursor column is not nullable by willi-mueller in https://github.com/dlt-hub/dlt/pull/1791
* controls row group size and empty tables in memory buffer when writing parquet by rudolfix in https://github.com/dlt-hub/dlt/pull/1782
* fix installation command" by novica in https://github.com/dlt-hub/dlt/pull/1741
* skips tables without jobs when merging delta tables by rudolfix in https://github.com/dlt-hub/dlt/pull/1803

Docs
* display past versions of the documentation (0.5.x / 1.0.0 / devel) by sh-rp in https://github.com/dlt-hub/dlt/pull/1770
* Refactor filesystem doc by VioletM in https://github.com/dlt-hub/dlt/pull/1745
* Update REST API docs by akelad in https://github.com/dlt-hub/dlt/pull/1795
* Add filesystem tutorial by VioletM in https://github.com/dlt-hub/dlt/pull/1775
* adding the sql_database tutorial by rahuljo in https://github.com/dlt-hub/dlt/pull/1796
* structural and content changes to the sql_database doc by rahuljo in https://github.com/dlt-hub/dlt/pull/1623
* Docs: update the introduction, add the rest_api tutorial by burnash in https://github.com/dlt-hub/dlt/pull/1729
* Docs/update deploy dagster by mariarice15 in https://github.com/dlt-hub/dlt/pull/1761
* Correct wrong code example for apply_hints( incremental(xx) ) by w0ut0 in https://github.com/dlt-hub/dlt/pull/1785
* Moves sources and destinations to the top level in docs navigation by VioletM in https://github.com/dlt-hub/dlt/pull/1750
* Fix typo "frequenly" by ruudwelten in https://github.com/dlt-hub/dlt/pull/1800
* Reorder sidebar by mariarice15 in https://github.com/dlt-hub/dlt/pull/1787

New Contributors
* novica made their first contribution in https://github.com/dlt-hub/dlt/pull/1741
* mariarice15 made their first contribution in https://github.com/dlt-hub/dlt/pull/1761
* w0ut0 made their first contribution in https://github.com/dlt-hub/dlt/pull/1785
* ruudwelten made their first contribution in https://github.com/dlt-hub/dlt/pull/1800

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.5.4...1.0.0

Core Library
* BigQuery project_id may be different from credentials project_id by VioletM in https://github.com/dlt-hub/dlt/pull/1680
* Enable schema evolution for `merge` write disposition with `delta` table format by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1742
* Add `storage_options` to `DeltaTable.create` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1686
* Fix `delta` table dangling Parquet file bug by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1695
* Add `delta` table partitioning support by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1696
* fixes load job counter displayed in progress by rudolfix in https://github.com/dlt-hub/dlt/pull/1702
* RESTClient: stops pagination after empty page (Feat/1637) by willi-mueller in https://github.com/dlt-hub/dlt/pull/1677
* Enable `scd2` record reinsert by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1707
* `scd2` custom "valid from" / "valid to" value feature by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1709
* feat/1681 collects load job metrics and adds remote url to traces by rudolfix in https://github.com/dlt-hub/dlt/pull/1708
* locks trace format with a contract rudolfix in https://github.com/dlt-hub/dlt/pull/1708
* Feat/1711 create with not exists for dlt tables to reduce racing conditions by rudolfix in https://github.com/dlt-hub/dlt/pull/1740
* provides detail exception messages when cursor stored value cannot be coerced to data by rudolfix in https://github.com/dlt-hub/dlt/pull/1748
* Allows to configure if staging destination is truncated or left intact to config by VioletM in https://github.com/dlt-hub/dlt/pull/1717
* enables external location and named credential in databricks, allows abfss://containeraccount Azure urls by rudolfix in https://github.com/dlt-hub/dlt/pull/1755
* fixes 1703 and 1754 by rudolfix in https://github.com/dlt-hub/dlt/pull/1755

Docs:
* rest_api: documents pluggable custom auth by willi-mueller in https://github.com/dlt-hub/dlt/pull/1690
* Update Snowflake docs by akelad in https://github.com/dlt-hub/dlt/pull/1747
* Docs/issue 1661 add tip to source docs and update weaviate docs by dat-a-man in https://github.com/dlt-hub/dlt/pull/1662
* Add custom parent-child relationships example by dat-a-man in https://github.com/dlt-hub/dlt/pull/1678
* Correct the library name for mem stats to `psutil` by deepyaman in https://github.com/dlt-hub/dlt/pull/1733
* Replaced "full_refresh" with "dev_mode" by dat-a-man in https://github.com/dlt-hub/dlt/pull/1735


New Contributors
* deepyaman made their first contribution in https://github.com/dlt-hub/dlt/pull/1733

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.5.3...0.5.4