Dbt-databricks

Breaking changes

- Raise an exception when schema contains '.'. ([222](https://github.com/databricks/dbt-databricks/pull/222))
- Containing a catalog in `schema` is not allowed anymore.
- Need to explicitly use `catalog` instead.

Features

- Support Python 3.11 ([233](https://github.com/databricks/dbt-databricks/pull/233))
- Support `incremental_predicates` ([161](https://github.com/databricks/dbt-databricks/pull/161))
- Apply connection retry refactor, add defaults with exponential backoff ([137](https://github.com/databricks/dbt-databricks/pull/137))
- Quote by Default ([241](https://github.com/databricks/dbt-databricks/pull/241))
- Avoid show table extended command. ([231](https://github.com/databricks/dbt-databricks/pull/231))
- Use show table extended with table name list for get_catalog. ([237](https://github.com/databricks/dbt-databricks/pull/237))
- Add support for a glob pattern in the databricks_copy_into macro ([259](https://github.com/databricks/dbt-databricks/pull/259))

Fixes

- Fix copy into macro when passing `expression_list`. ([223](https://github.com/databricks/dbt-databricks/pull/223))
- Partially revert to fix the case where schema config contains uppercase letters. ([224](https://github.com/databricks/dbt-databricks/pull/224))

Under the hood

- Show and log a warning when schema contains '.'. ([221](https://github.com/databricks/dbt-databricks/pull/221))

Features

- Support python model through run command API, currently supported materializations are table and incremental. ([dbt-labs/dbt-spark377](https://github.com/dbt-labs/dbt-spark/pull/377), [#126](https://github.com/databricks/dbt-databricks/pull/126))
- Enable Pandas and Pandas-on-Spark DataFrames for dbt python models ([dbt-labs/dbt-spark469](https://github.com/dbt-labs/dbt-spark/pull/469), [#181](https://github.com/databricks/dbt-databricks/pull/181))
- Support job cluster in notebook submission method ([dbt-labs/dbt-spark467](https://github.com/dbt-labs/dbt-spark/pull/467), [#194](https://github.com/databricks/dbt-databricks/pull/194))
- In `all_purpose_cluster` submission method, a config `http_path` can be specified in Python model config to switch the cluster where Python model runs.
py
def model(dbt, _):
dbt.config(
materialized='table',
http_path='...'
)
...

- Use builtin timestampadd and timestampdiff functions for dateadd/datediff macros if available ([185](https://github.com/databricks/dbt-databricks/pull/185))
- Implement testing for a test for various Python models ([189](https://github.com/databricks/dbt-databricks/pull/189))
- Implement testing for `type_boolean` in Databricks ([dbt-labs/dbt-spark471](https://github.com/dbt-labs/dbt-spark/pull/471), [#188](https://github.com/databricks/dbt-databricks/pull/188))
- Add a macro to support [COPY INTO](https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html) ([#190](https://github.com/databricks/dbt-databricks/pull/190))

Under the hood

- Apply "Initial refactoring of incremental materialization" ([148](https://github.com/databricks/dbt-databricks/pull/148))
- Now dbt-databricks uses `adapter.get_incremental_strategy_macro` instead of `dbt_spark_get_incremental_sql` macro to dispatch the incremental strategy macro. The overwritten `dbt_spark_get_incremental_sql` macro will not work anymore.
- Better interface for python submission ([dbt-labs/dbt-spark452](https://github.com/dbt-labs/dbt-spark/pull/452), [#178](https://github.com/databricks/dbt-databricks/pull/178))

Fixes

- Fix cancellation ([173](https://github.com/databricks/dbt-databricks/pull/173))
- `http_headers` should be dict in the profile ([174](https://github.com/databricks/dbt-databricks/pull/174))

Fixes

- Data is duplicated on reloading seeds that are using an external table ([114](https://github.com/databricks/dbt-databricks/issues/114), [#149](https://github.com/databricks/dbt-databricks/issues/149))

Under the hood

- Explicitly close cursors ([163](https://github.com/databricks/dbt-databricks/pull/163))
- Upgrade databricks-sql-connector to 2.0.5 ([166](https://github.com/databricks/dbt-databricks/pull/166))
- Embed dbt-databricks and databricks-sql-connector versions to SQL comments ([167](https://github.com/databricks/dbt-databricks/pull/167))