Dlt

Core Library
* SCD2 support by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1168 https://dlthub.com/devel/general-usage/incremental-loading#scd2-strategy
* A fully configurable layout for filesystem files by sultaniman in https://github.com/dlt-hub/dlt/pull/1182 https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#files-layout
* picks file format matching item format to minimize number of rewrites during loading by rudolfix in https://github.com/dlt-hub/dlt/pull/1222
* fix athena iceberg's trailing location by romanperesypkin in https://github.com/dlt-hub/dlt/pull/1230
* Pass options to parse iso like strings by VioletM in https://github.com/dlt-hub/dlt/pull/1219
* pipeline state can be restored from filesystem destination by sh-rp in https://github.com/dlt-hub/dlt/pull/1184 - https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#syncing-of-dlt-state
* Remove `staging-optimized` replace strategy for `synapse` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1231
* fixes bug, where configs where not injected for async functions by sh-rp in https://github.com/dlt-hub/dlt/pull/1241
* feat(transform): implement columns pivot map function by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1152
* Add max_table_nesting to resource decorator by sultaniman in https://github.com/dlt-hub/dlt/pull/1242
* adds csv options to write headers, change delimiter, quotation style by rudolfix in https://github.com/dlt-hub/dlt/pull/1239
* Check for default schema and schema name in streamlit session by sultaniman in https://github.com/dlt-hub/dlt/pull/1155
* Add seconds and millisecond timestamps to filesystem date placeholders by sultaniman in https://github.com/dlt-hub/dlt/pull/1260
* send dlt telemetry wherever you want, not only segment by zem360 in https://github.com/dlt-hub/dlt/pull/1236
* Make merge write-disposition fall back to staging append if no primary or merge keys are specified by sh-rp in https://github.com/dlt-hub/dlt/pull/1225
* Add snowflake application parameter to configuration by sultaniman in https://github.com/dlt-hub/dlt/pull/1266

Docs
* Added docs for deploying dlt with Prefect. by dat-a-man in https://github.com/dlt-hub/dlt/pull/1138
* a note on scd2 incoming high ts change by rudolfix in https://github.com/dlt-hub/dlt/pull/1273
* adding images and wordsmithing to Prefect walkthrough by WillRaphaelson in https://github.com/dlt-hub/dlt/pull/1276

Verified Sources
* Use `pyarrow`, `pandas`, `connectorx` or `sqlalchemy` backends when reading tables with `sql_database`. See README for details. https://github.com/dlt-hub/verified-sources/pull/425
* Google ads source is available https://github.com/dlt-hub/verified-sources/pull/428
* Pages endpoint for notion https://github.com/dlt-hub/verified-sources/pull/429

New Contributors
* romanperesypkin made their first contribution in https://github.com/dlt-hub/dlt/pull/1230
* WillRaphaelson made their first contribution in https://github.com/dlt-hub/dlt/pull/1276

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.8...0.4.9

A pre-release that allows to try out the following features and includes the following bugfixes:
* SCD2 support by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1168 **We are still working on BigQuery support)** https://dlthub.com/devel/general-usage/incremental-loading#scd2-strategy
* A fully configurable layout for filesystem files by sultaniman in https://github.com/dlt-hub/dlt/pull/1182 https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#files-layout
* picks file format matching item format by rudolfix in https://github.com/dlt-hub/dlt/pull/1222
* fix athena iceberg's trailing location by romanperesypkin in https://github.com/dlt-hub/dlt/pull/1230
* Pass options to parse iso like strings by VioletM in https://github.com/dlt-hub/dlt/pull/1219
* filesystem state sync by sh-rp in https://github.com/dlt-hub/dlt/pull/1184 - https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#syncing-of-dlt-state
* Remove `staging-optimized` replace strategy for `synapse` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1231
* fixes bug, where configs where not injected for async functions by sh-rp in https://github.com/dlt-hub/dlt/pull/1241
* adds options to write csv headers, change delimiter by rudolfix in https://github.com/dlt-hub/dlt/pull/1239

Final release is scheduled for next week

Core Library
* Add Dremio as a destination by maxfirman in https://github.com/dlt-hub/dlt/pull/1026
* adds a fast loading of arrow tables/pandas to postgres via COPY csv by rudolfix in https://github.com/dlt-hub/dlt/pull/1185
* adds a csv writer for filesystem and postgres by rudolfix in https://github.com/dlt-hub/dlt/pull/1185
* saves parquet with all logical types, `spark` flavor is not a default any longer by rudolfix in https://github.com/dlt-hub/dlt/pull/1185
https://github.com/dlt-hub/dlt/pull/1185
* feat(bigquery): add streaming inserts support by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1123
* Feat: parameterize pipeline class in the primary factory method by z3z1ma in https://github.com/dlt-hub/dlt/pull/1176
* Fix: check for typeddict before class or subclass checks which fail by z3z1ma in https://github.com/dlt-hub/dlt/pull/1160
* fixes column order and add hints table variants by rudolfix in https://github.com/dlt-hub/dlt/pull/1127
* fixes schema versioning by rudolfix in https://github.com/dlt-hub/dlt/pull/1140
* regular initializers for credentials / config specs are type checked like dataclasses by rudolfix in https://github.com/dlt-hub/dlt/pull/1142
* fix streamlit app state display: Add yaml representer for pendulum datetime by sultaniman in https://github.com/dlt-hub/dlt/pull/1192
* `synapse` and `mssql` bugfixes and improvements (INSERT VALUES UNION) by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1174
* various improvements to arrow table normalization by rudolfix in https://github.com/dlt-hub/dlt/pull/1185
* arrow tables without rows create tables in destination by rudolfix in https://github.com/dlt-hub/dlt/pull/1185
* fixes Motherduck configuration to use `my_db` default database and makes password / token mandatory by rudolfix in

Docs
* docs: add typechecking to embedded snippets by sh-rp in https://github.com/dlt-hub/dlt/pull/1130
* Fix typo with switched column names in schema evolution docs page by b-per in https://github.com/dlt-hub/dlt/pull/1132
* Docs: deploy with Kestra by dat-a-man in https://github.com/dlt-hub/dlt/pull/1087
* Docs: Deploy dlt on dagster by dat-a-man in https://github.com/dlt-hub/dlt/pull/1086
* Update example connection string by MiConnell in https://github.com/dlt-hub/dlt/pull/1188
* Changed directory of all the blog images to google cloud storage. by dat-a-man in https://github.com/dlt-hub/dlt/pull/1156

Verified Sources
* postgres replication / CDC by jorritsandbrink https://github.com/dlt-hub/verified-sources/pull/392

New Contributors
* b-per made their first contribution in https://github.com/dlt-hub/dlt/pull/1132
* MiConnell made their first contribution in https://github.com/dlt-hub/dlt/pull/1188
* maxfirman made their first contribution in https://github.com/dlt-hub/dlt/pull/1026

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.7...0.4.8

Core Library
* Custom destinations with `dlt.destination` decorator by sh-rp in https://github.com/dlt-hub/dlt/pull/1065
* A BigQuery custom destination supporting STRUCT data types by sh-rp in https://github.com/dlt-hub/dlt/pull/1107
* Built-in Streamlit rewrite, UI improvements, dark theme a by sultaniman in https://github.com/dlt-hub/dlt/pull/1060
* fixes various edge cases with Incremental data deduplication, for ordered and unordered results 971 by rudolfix in https://github.com/dlt-hub/dlt/pull/1062
* Adds new `dlt.mark` marker to materialize table schemas without data by rudolfix in https://github.com/dlt-hub/dlt/pull/1122
* validates class instances in typed dict by rudolfix in https://github.com/dlt-hub/dlt/pull/1082
* feat(airflow): allow re-using sources in airflow wrapper by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1080
* feat(core): drop default value for write disposition by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1057
* splits pandas and arrow imports to fix pyarrow.compute missing by rudolfix in https://github.com/dlt-hub/dlt/pull/1112
* improve no schema upgrade path exception by sh-rp in https://github.com/dlt-hub/dlt/pull/1125

Docs
* docs(airflow): add description of new decompose methods by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1072
* check embedded code blocks by sh-rp in https://github.com/dlt-hub/dlt/pull/1093
* docs(kafka): describe the possible sync issues by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1100
* Docs: schema evolution by dat-a-man in https://github.com/dlt-hub/dlt/pull/1078
* Add example link to the custom destination page by VioletM in https://github.com/dlt-hub/dlt/pull/1120


**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.6...0.4.7

Core Library
* feat(airflow): expose the Airflow runner method to create custom DAGs by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1014
* removes sql alchemy dependency and port parts of URL class by rudolfix in https://github.com/dlt-hub/dlt/pull/1028
* Parallelize decorator - run many regular generators in parallel by steinitzu in https://github.com/dlt-hub/dlt/pull/965
* Add __main__ entry point to support calling dlt as python module by sultaniman in https://github.com/dlt-hub/dlt/pull/1023


Library Bugfixes
* fixes naive datetime bug in incremental by rudolfix in https://github.com/dlt-hub/dlt/pull/1020
* Import missing pyarrow compute for transforms on arrowitems by sh-rp in https://github.com/dlt-hub/dlt/pull/1010
* delete normalized package in case it already existed by sh-rp in https://github.com/dlt-hub/dlt/pull/1012
* fix(core): validation error with TTableHintTemplate by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1039
* adds test case where payload data contains PUA unicode characters by willi-mueller in https://github.com/dlt-hub/dlt/pull/1053
* fix add_limit behavior in edge cases by sh-rp in https://github.com/dlt-hub/dlt/pull/1052
* adds row_order to Incremental - automatically stop taking data when out of range by rudolfix in https://github.com/dlt-hub/dlt/pull/1041
* Fix to serialize load metrics as list instead of a dictionary by sultaniman in https://github.com/dlt-hub/dlt/pull/1051
* fix import schema workflow by sh-rp in https://github.com/dlt-hub/dlt/pull/1013
* rollback all changes to live schemas when extraction fails by sh-rp in https://github.com/dlt-hub/dlt/pull/1013

Docs
* Fix zendesk example test by VioletM in https://github.com/dlt-hub/dlt/pull/1027
* Edit arrow-pandas.md and fix a typo by Bl3f in https://github.com/dlt-hub/dlt/pull/1001
* Added info about file compression to filesystem docs by dat-a-man in https://github.com/dlt-hub/dlt/pull/975
* Update "create destination" docs with new file layouts by steinitzu in https://github.com/dlt-hub/dlt/pull/1032
* Docs update on how to set query limits. by dat-a-man in https://github.com/dlt-hub/dlt/pull/973
* Docs/Updated for slack alerts. by dat-a-man in https://github.com/dlt-hub/dlt/pull/1042

Verified Sources
* scrape web sites with spiders and Scrapy and send data to dlt sultaniman https://github.com/dlt-hub/verified-sources/pull/332
* `sql_database` recoginizes `end_value` and `row_order` to return rows in range and optionally ordered. backfill and proper Airflow intervals support rudolfix https://github.com/dlt-hub/verified-sources/pull/388/

New Contributors
* Bl3f made their first contribution in https://github.com/dlt-hub/dlt/pull/1001

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.5...0.4.6

Core Library
* enables google drive filesystem for sources and destinations (second one experimental, google drive listings are only eventually consistent!) by IlyaFaer in https://github.com/dlt-hub/dlt/pull/932
* creates parallel Airflow DAGs in airflow helper to allow many resources to be executed at once IlyaFaer in https://github.com/dlt-hub/dlt/pull/966
* 855 create bigquery adapter for dlt resources: easily configure partitions, clustering, data retention etc. by Pipboyguy in https://github.com/dlt-hub/dlt/pull/952 and https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery#bigquery-adapter
* Use BIGNUMERIC for large decimals in bigquery by steinitzu in https://github.com/dlt-hub/dlt/pull/984
* Normalize keys for Google secrets config provider by sultaniman in https://github.com/dlt-hub/dlt/pull/963
* does not lowercase postgres and redshift database names by rudolfix in https://github.com/dlt-hub/dlt/pull/990
* Introduce `hard_delete` and `dedup_sort` columns hint for `merge` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/960 and https://dlthub.com/docs/general-usage/incremental-loading#delete-records
* adjustment of pua start in typed json encoding, pass through on decoding errors by rudolfix in https://github.com/dlt-hub/dlt/pull/974
* creates isolated parallel Airflow DAGs in airflow helper to execute resources parallel in isolated pipelines IlyaFaer in https://github.com/dlt-hub/dlt/pull/979
* Fix annotation processing and rebuilding, mark dataclass as complex by sultaniman in https://github.com/dlt-hub/dlt/pull/980
* allows async functions to be decorated with dlt.source by rudolfix in https://github.com/dlt-hub/dlt/pull/985
* allows right pipe operator to feed simple lists into a transformer rudolfix in https://github.com/dlt-hub/dlt/pull/985
* allows pendulum datetime as incremental cursor when loading arrow tables rudolfix in https://github.com/dlt-hub/dlt/pull/985
* enables Python 3.12 (mind that not all extras have python 3.12 libraries!) rudolfix in https://github.com/dlt-hub/dlt/pull/985

Docs
* docs(filesystem): include Google Drive into filesystem tutorial by IlyaFaer in https://github.com/dlt-hub/dlt/pull/962
* Fix typos/grammar in tutorial docs by taljaards in https://github.com/dlt-hub/dlt/pull/972
* add blog post observability by adrianbr in https://github.com/dlt-hub/dlt/pull/989
* Update arrow-pandas.md by snehangsude in https://github.com/dlt-hub/dlt/pull/992
* Clarify info about GoodData in modelling tools article by mhauzirek in https://github.com/dlt-hub/dlt/pull/956
* Fix small typings in contributing guide by VioletM in https://github.com/dlt-hub/dlt/pull/993
* Docs/google sheets update by dat-a-man in https://github.com/dlt-hub/dlt/pull/976
* Added "Incremental Configuration" section to SQL Databases documentat… by dat-a-man in https://github.com/dlt-hub/dlt/pull/977

Verified Sources
* Bing Webmaster source by willi-mueller

New Contributors
* taljaards made their first contribution in https://github.com/dlt-hub/dlt/pull/972
* mhauzirek made their first contribution in https://github.com/dlt-hub/dlt/pull/956
* snehangsude made their first contribution in https://github.com/dlt-hub/dlt/pull/992
* VioletM made their first contribution in https://github.com/dlt-hub/dlt/pull/993

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.4...0.4.5