Dlt

Core Library
* Custom destinations with `dlt.destination` decorator by sh-rp in https://github.com/dlt-hub/dlt/pull/1065
* A BigQuery custom destination supporting STRUCT data types by sh-rp in https://github.com/dlt-hub/dlt/pull/1107
* Built-in Streamlit rewrite, UI improvements, dark theme a by sultaniman in https://github.com/dlt-hub/dlt/pull/1060
* fixes various edge cases with Incremental data deduplication, for ordered and unordered results 971 by rudolfix in https://github.com/dlt-hub/dlt/pull/1062
* Adds new `dlt.mark` marker to materialize table schemas without data by rudolfix in https://github.com/dlt-hub/dlt/pull/1122
* validates class instances in typed dict by rudolfix in https://github.com/dlt-hub/dlt/pull/1082
* feat(airflow): allow re-using sources in airflow wrapper by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1080
* feat(core): drop default value for write disposition by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1057
* splits pandas and arrow imports to fix pyarrow.compute missing by rudolfix in https://github.com/dlt-hub/dlt/pull/1112
* improve no schema upgrade path exception by sh-rp in https://github.com/dlt-hub/dlt/pull/1125

Docs
* docs(airflow): add description of new decompose methods by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1072
* check embedded code blocks by sh-rp in https://github.com/dlt-hub/dlt/pull/1093
* docs(kafka): describe the possible sync issues by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1100
* Docs: schema evolution by dat-a-man in https://github.com/dlt-hub/dlt/pull/1078
* Add example link to the custom destination page by VioletM in https://github.com/dlt-hub/dlt/pull/1120


**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.6...0.4.7

Core Library
* feat(airflow): expose the Airflow runner method to create custom DAGs by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1014
* removes sql alchemy dependency and port parts of URL class by rudolfix in https://github.com/dlt-hub/dlt/pull/1028
* Parallelize decorator - run many regular generators in parallel by steinitzu in https://github.com/dlt-hub/dlt/pull/965
* Add __main__ entry point to support calling dlt as python module by sultaniman in https://github.com/dlt-hub/dlt/pull/1023


Library Bugfixes
* fixes naive datetime bug in incremental by rudolfix in https://github.com/dlt-hub/dlt/pull/1020
* Import missing pyarrow compute for transforms on arrowitems by sh-rp in https://github.com/dlt-hub/dlt/pull/1010
* delete normalized package in case it already existed by sh-rp in https://github.com/dlt-hub/dlt/pull/1012
* fix(core): validation error with TTableHintTemplate by IlyaFaer in https://github.com/dlt-hub/dlt/pull/1039
* adds test case where payload data contains PUA unicode characters by willi-mueller in https://github.com/dlt-hub/dlt/pull/1053
* fix add_limit behavior in edge cases by sh-rp in https://github.com/dlt-hub/dlt/pull/1052
* adds row_order to Incremental - automatically stop taking data when out of range by rudolfix in https://github.com/dlt-hub/dlt/pull/1041
* Fix to serialize load metrics as list instead of a dictionary by sultaniman in https://github.com/dlt-hub/dlt/pull/1051
* fix import schema workflow by sh-rp in https://github.com/dlt-hub/dlt/pull/1013
* rollback all changes to live schemas when extraction fails by sh-rp in https://github.com/dlt-hub/dlt/pull/1013

Docs
* Fix zendesk example test by VioletM in https://github.com/dlt-hub/dlt/pull/1027
* Edit arrow-pandas.md and fix a typo by Bl3f in https://github.com/dlt-hub/dlt/pull/1001
* Added info about file compression to filesystem docs by dat-a-man in https://github.com/dlt-hub/dlt/pull/975
* Update "create destination" docs with new file layouts by steinitzu in https://github.com/dlt-hub/dlt/pull/1032
* Docs update on how to set query limits. by dat-a-man in https://github.com/dlt-hub/dlt/pull/973
* Docs/Updated for slack alerts. by dat-a-man in https://github.com/dlt-hub/dlt/pull/1042

Verified Sources
* scrape web sites with spiders and Scrapy and send data to dlt sultaniman https://github.com/dlt-hub/verified-sources/pull/332
* `sql_database` recoginizes `end_value` and `row_order` to return rows in range and optionally ordered. backfill and proper Airflow intervals support rudolfix https://github.com/dlt-hub/verified-sources/pull/388/

New Contributors
* Bl3f made their first contribution in https://github.com/dlt-hub/dlt/pull/1001

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.5...0.4.6

Core Library
* enables google drive filesystem for sources and destinations (second one experimental, google drive listings are only eventually consistent!) by IlyaFaer in https://github.com/dlt-hub/dlt/pull/932
* creates parallel Airflow DAGs in airflow helper to allow many resources to be executed at once IlyaFaer in https://github.com/dlt-hub/dlt/pull/966
* 855 create bigquery adapter for dlt resources: easily configure partitions, clustering, data retention etc. by Pipboyguy in https://github.com/dlt-hub/dlt/pull/952 and https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery#bigquery-adapter
* Use BIGNUMERIC for large decimals in bigquery by steinitzu in https://github.com/dlt-hub/dlt/pull/984
* Normalize keys for Google secrets config provider by sultaniman in https://github.com/dlt-hub/dlt/pull/963
* does not lowercase postgres and redshift database names by rudolfix in https://github.com/dlt-hub/dlt/pull/990
* Introduce `hard_delete` and `dedup_sort` columns hint for `merge` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/960 and https://dlthub.com/docs/general-usage/incremental-loading#delete-records
* adjustment of pua start in typed json encoding, pass through on decoding errors by rudolfix in https://github.com/dlt-hub/dlt/pull/974
* creates isolated parallel Airflow DAGs in airflow helper to execute resources parallel in isolated pipelines IlyaFaer in https://github.com/dlt-hub/dlt/pull/979
* Fix annotation processing and rebuilding, mark dataclass as complex by sultaniman in https://github.com/dlt-hub/dlt/pull/980
* allows async functions to be decorated with dlt.source by rudolfix in https://github.com/dlt-hub/dlt/pull/985
* allows right pipe operator to feed simple lists into a transformer rudolfix in https://github.com/dlt-hub/dlt/pull/985
* allows pendulum datetime as incremental cursor when loading arrow tables rudolfix in https://github.com/dlt-hub/dlt/pull/985
* enables Python 3.12 (mind that not all extras have python 3.12 libraries!) rudolfix in https://github.com/dlt-hub/dlt/pull/985

Docs
* docs(filesystem): include Google Drive into filesystem tutorial by IlyaFaer in https://github.com/dlt-hub/dlt/pull/962
* Fix typos/grammar in tutorial docs by taljaards in https://github.com/dlt-hub/dlt/pull/972
* add blog post observability by adrianbr in https://github.com/dlt-hub/dlt/pull/989
* Update arrow-pandas.md by snehangsude in https://github.com/dlt-hub/dlt/pull/992
* Clarify info about GoodData in modelling tools article by mhauzirek in https://github.com/dlt-hub/dlt/pull/956
* Fix small typings in contributing guide by VioletM in https://github.com/dlt-hub/dlt/pull/993
* Docs/google sheets update by dat-a-man in https://github.com/dlt-hub/dlt/pull/976
* Added "Incremental Configuration" section to SQL Databases documentat… by dat-a-man in https://github.com/dlt-hub/dlt/pull/977

Verified Sources
* Bing Webmaster source by willi-mueller

New Contributors
* taljaards made their first contribution in https://github.com/dlt-hub/dlt/pull/972
* mhauzirek made their first contribution in https://github.com/dlt-hub/dlt/pull/956
* snehangsude made their first contribution in https://github.com/dlt-hub/dlt/pull/992
* VioletM made their first contribution in https://github.com/dlt-hub/dlt/pull/993

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.4...0.4.5

Core Library
* passes incremental from apply hints to resource function by rudolfix in https://github.com/dlt-hub/dlt/pull/953
* Handle UnionType when checking is_union_type and is_optional_type by sultaniman in https://github.com/dlt-hub/dlt/pull/951
* yanks orjson to <=0.3.10 by rudolfix in https://github.com/dlt-hub/dlt/pull/958

Docs
* Databricks workspace setup docs by steinitzu in https://github.com/dlt-hub/dlt/pull/949

Verified Source
* allows for table reflection at runtime, column selection and buffer control in `sql_database` rudolfix (https://github.com/dlt-hub/verified-sources/pull/351)

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.3...0.4.4

Core Library
* Databricks destination by steinitzu and phillem15 in https://github.com/dlt-hub/dlt/pull/892
* Synapse destination by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/900
* BigQuery Partitioning Improvements by Pipboyguy in https://github.com/dlt-hub/dlt/pull/887
* enable async generators as resources by sh-rp in https://github.com/dlt-hub/dlt/pull/905
* fix: use truthy value in ternary since 0 cause div by zero by z3z1ma in https://github.com/dlt-hub/dlt/pull/902
* feat(filesystem): add compression flag if the read file is GZ by IlyaFaer in https://github.com/dlt-hub/dlt/pull/912
* Enhancements in Filesystem Configuration by Pipboyguy in https://github.com/dlt-hub/dlt/pull/869
* add mark function to emit resource hints from decorated function by rudolfix in https://github.com/dlt-hub/dlt/pull/938
* handles nested Pydantic models when generating dlt schema by sultaniman in https://github.com/dlt-hub/dlt/pull/901

Docs
* Restructure intro, getting started and tutorial by burnash in https://github.com/dlt-hub/dlt/pull/702
* Update the release instructions in CONTRIBUTING.md by burnash in https://github.com/dlt-hub/dlt/pull/867
* Add explicit sub section about streamlit under getting started by sultaniman in https://github.com/dlt-hub/dlt/pull/884
* Examples: google sheets by AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/846
* Added URL-parser documentation by dat-a-man in https://github.com/dlt-hub/dlt/pull/909

Verified Sources
* feat(filesystem): implement a csv reader with duckdb engine IlyaFaer https://github.com/dlt-hub/verified-sources/pull/319
* fix(notion): define payload within the while-loop glebzhidkov (https://github.com/dlt-hub/verified-sources/pull/338)
* sql alchemy + connector x example rudolfix (https://github.com/dlt-hub/verified-sources/pull/334)
* Shopify: Standalone resource for partner API queries steinitzu (https://github.com/dlt-hub/verified-sources/pull/329)
* sql-database: detect precision and scale of supported column types steinitzu (https://github.com/dlt-hub/verified-sources/pull/324)
* feat(sources.kafka): implement Kafka source IlyaFaer (https://github.com/dlt-hub/verified-sources/pull/306)

New Contributors
* Pipboyguy made their first contribution in https://github.com/dlt-hub/dlt/pull/869
* sultaniman made their first contribution in https://github.com/dlt-hub/dlt/pull/883

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.2...0.4.3

Core Library
- Fix the data type used in the `from_db_type()` method from `MsSqlTypeMapper` by jorritsandbrink in https://github.com/dlt-hub/dlt/pull/863
- Use Secret Manager in CI by AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/859
- Move destination adapters to `dlt.destination.adapters` by rudolfix in https://github.com/dlt-hub/dlt/pull/854

Docs
- Improve HubSpot source docs by IlyaFaer in https://github.com/dlt-hub/dlt/pull/864
- Add new topic to docs: Destination; improve Configuration docs by rudolfix in https://github.com/dlt-hub/dlt/pull/861

**Full Changelog**: https://github.com/dlt-hub/dlt/compare/0.4.1...0.4.2