Sqlfluff

- Update changelog

- Add a status bar button that runs "Format document with sqlfluff"
- If dbt-core-interface is enabled in the extension
- The "fix" command should use dbt-core-interface /format endpoint
- The "format selection" action should also use dbt-core-interface /format endpoint

- Update for sqlfluff version 3.0.0.
- Add Vertica dialect.
- Removed `--force` from `fix` as it is the default for >= 3.0.0.

Highlights

This introduces some memory optimisations in the linting operation which
prevent a major cause of crashes when linting large projects. As part of that
we've also deprecated the `--force` option on `sqlfluff fix` and made that
the default behaviour (the associated memory optimisations will come shortly).

This also removes the long since deprecated `--disable_progress_bar` option
(which was replaced by the kabab-case `--disable-progress-bar` more than a
year ago).

On top of that this release also introduces the `vertica` dialect for the
first time, and a whole host of bugfixes and improvements to other dialects.

This release should be considered a release candidate for the final `3.0.0`
release which will follow shortly in the next few days unless any other
major issues are found.

Thanks particularly to the **seven** new contributors we saw in this release 🏆🎉.

What’s Changed

* Add Support for Databricks `CREATE FUNCTION` Syntax in SparkSQL Parser [5615](https://github.com/sqlfluff/sqlfluff/pull/5615) [mitchellvanrijkom](https://github.com/mitchellvanrijkom)
* Swap fix `--force` for `--check` [5650](https://github.com/sqlfluff/sqlfluff/pull/5650) [alanmcruickshank](https://github.com/alanmcruickshank)
* Remove `DeprecatedOption` [5649](https://github.com/sqlfluff/sqlfluff/pull/5649) [alanmcruickshank](https://github.com/alanmcruickshank)
* Resolve broken loop limit test [5651](https://github.com/sqlfluff/sqlfluff/pull/5651) [alanmcruickshank](https://github.com/alanmcruickshank)
* Snowflake: Move NOTIFY to non-reserved words [5645](https://github.com/sqlfluff/sqlfluff/pull/5645) [greg-finley](https://github.com/greg-finley)
* BigQuery: GROUP BY ALL [5646](https://github.com/sqlfluff/sqlfluff/pull/5646) [greg-finley](https://github.com/greg-finley)
* chore: use pre-calculated `_code_indices` in `BaseSegment::raw_segmen… [5644](https://github.com/sqlfluff/sqlfluff/pull/5644) [gvozdvmozgu](https://github.com/gvozdvmozgu)
* Fix Snowflake Semistructured identifier parsing regex-expression [5635](https://github.com/sqlfluff/sqlfluff/pull/5635) [DannyMor](https://github.com/DannyMor)
* Postgres: Update ReferentialActionGrammar to support sets of columns [5628](https://github.com/sqlfluff/sqlfluff/pull/5628) [WittierDinosaur](https://github.com/WittierDinosaur)
* Snowflake: Add syntax for masking policy force [5629](https://github.com/sqlfluff/sqlfluff/pull/5629) [WittierDinosaur](https://github.com/WittierDinosaur)
* Postgres: Allow nested block comments [5630](https://github.com/sqlfluff/sqlfluff/pull/5630) [WittierDinosaur](https://github.com/WittierDinosaur)
* Postgres: Add Create, Alter, Drop Statistics [5631](https://github.com/sqlfluff/sqlfluff/pull/5631) [WittierDinosaur](https://github.com/WittierDinosaur)
* T-SQL Fix relative sql filepath lexer [5632](https://github.com/sqlfluff/sqlfluff/pull/5632) [WittierDinosaur](https://github.com/WittierDinosaur)
* Tech Debt: Replace some sequences with their Ref equivalents [5633](https://github.com/sqlfluff/sqlfluff/pull/5633) [WittierDinosaur](https://github.com/WittierDinosaur)
* ANSI/MYSQL: Support Create Role If Not Exists [5634](https://github.com/sqlfluff/sqlfluff/pull/5634) [WittierDinosaur](https://github.com/WittierDinosaur)
* Add Vertica dialect [5640](https://github.com/sqlfluff/sqlfluff/pull/5640) [PolitePp](https://github.com/PolitePp)
* Add Support for Snowflake Materialised View and Column Masking Policy [5637](https://github.com/sqlfluff/sqlfluff/pull/5637) [ulixius9](https://github.com/ulixius9)
* [snowflake dialect] support ALTER TABLE ... ADD COLUMN IF NOT EXISTS [5621](https://github.com/sqlfluff/sqlfluff/pull/5621) [gshen7](https://github.com/gshen7)
* SQLite: Make `DISTINCT FROM` optional; SQLite/TSQL/Exasol: Nothing'd `NanLiteralSegment` [5620](https://github.com/sqlfluff/sqlfluff/pull/5620) [keraion](https://github.com/keraion)
* Upgrade greenplum dialect [5546](https://github.com/sqlfluff/sqlfluff/pull/5546) [kkozhakin](https://github.com/kkozhakin)
* Oracle: parse length qualifier in types [5613](https://github.com/sqlfluff/sqlfluff/pull/5613) [Jefffrey](https://github.com/Jefffrey)
* Multiple Dialects: Fix handling of nested sets expressions [5606](https://github.com/sqlfluff/sqlfluff/pull/5606) [keraion](https://github.com/keraion)
* DB2: Add labeled durations and special registers [5612](https://github.com/sqlfluff/sqlfluff/pull/5612) [keraion](https://github.com/keraion)
* Sparksql: Fix `LATERAL VIEW` following `JOIN`; `CLUSTER|SORT|DISTRIBUTE BY` or `QUALIFY` without `FROM` [5602](https://github.com/sqlfluff/sqlfluff/pull/5602) [keraion](https://github.com/keraion)
* File helpers and config test parameterisation. [5579](https://github.com/sqlfluff/sqlfluff/pull/5579) [alanmcruickshank](https://github.com/alanmcruickshank)
* Memory overhead optimisations during linting [5585](https://github.com/sqlfluff/sqlfluff/pull/5585) [alanmcruickshank](https://github.com/alanmcruickshank)
* fix: multiple columns foreign key constraint (5592) [5594](https://github.com/sqlfluff/sqlfluff/pull/5594) [maoxingda](https://github.com/maoxingda)
* [CI] Add `no_implicit_reexport` mypy check [5509](https://github.com/sqlfluff/sqlfluff/pull/5509) [Koyaani](https://github.com/Koyaani)

New Contributors
* [maoxingda](https://github.com/maoxingda) made their first contribution in [#5594](https://github.com/sqlfluff/sqlfluff/pull/5594)
* [Jefffrey](https://github.com/Jefffrey) made their first contribution in [#5613](https://github.com/sqlfluff/sqlfluff/pull/5613)
* [kkozhakin](https://github.com/kkozhakin) made their first contribution in [#5546](https://github.com/sqlfluff/sqlfluff/pull/5546)
* [gshen7](https://github.com/gshen7) made their first contribution in [#5621](https://github.com/sqlfluff/sqlfluff/pull/5621)
* [PolitePp](https://github.com/PolitePp) made their first contribution in [#5640](https://github.com/sqlfluff/sqlfluff/pull/5640)
* [DannyMor](https://github.com/DannyMor) made their first contribution in [#5635](https://github.com/sqlfluff/sqlfluff/pull/5635)
* [mitchellvanrijkom](https://github.com/mitchellvanrijkom) made their first contribution in [#5615](https://github.com/sqlfluff/sqlfluff/pull/5615)

Highlights

This release primarily brings through a large set of dialect improvements and
bugfixes from over the holiday period. Notably also:

* A change in the default behaviour for `convention.not_equals`. The new default
is to be `consistent`, which is slightly more relaxed than the original
behaviour.
* A new rule (`aliasing.self_alias.column`) which prevents aliasing a column
as itself.
* Disables `AL01` (`aliasing.table`) by default for Oracle.

This release also saw **ELEVEN** new contributors in this release 🎉🎉🏆🏆🎉🎉.
Great to see so many new people getting involved with the project. Thank You 🙏.

What’s Changed

* Add support & test for postgres alter policy with multiple clauses [5577](https://github.com/sqlfluff/sqlfluff/pull/5577) [fnimick](https://github.com/fnimick)
* Update github actions to latest versions [5584](https://github.com/sqlfluff/sqlfluff/pull/5584) [alanmcruickshank](https://github.com/alanmcruickshank)
* Allows using dbt cross project ref in jinja templater [5574](https://github.com/sqlfluff/sqlfluff/pull/5574) [alangner](https://github.com/alangner)
* Improve support for Jinja templater plugins with custom tags [5543](https://github.com/sqlfluff/sqlfluff/pull/5543) [james-johnston-thumbtack](https://github.com/james-johnston-thumbtack)
* Databricks: fix `EXCEPT` with qualified column reference [5557](https://github.com/sqlfluff/sqlfluff/pull/5557) [keraion](https://github.com/keraion)
* Stricter recommended config for not_equals convention [5580](https://github.com/sqlfluff/sqlfluff/pull/5580) [alanmcruickshank](https://github.com/alanmcruickshank)
* CV01: Add options for ANSI and consistent style. [5539](https://github.com/sqlfluff/sqlfluff/pull/5539) [keraion](https://github.com/keraion)
* DuckDB: Fix `REPLACE` after `EXCLUDE`. Fix AL03 linting for wildcard like expression. [5556](https://github.com/sqlfluff/sqlfluff/pull/5556) [keraion](https://github.com/keraion)
* Clickhouse: Add `GLOBAL JOIN`, `GLOBAL IN`, and `PASTE JOIN` [5560](https://github.com/sqlfluff/sqlfluff/pull/5560) [keraion](https://github.com/keraion)
* [Docs] Use extended policy for identifier capitalisation in starter config [5562](https://github.com/sqlfluff/sqlfluff/pull/5562) [j-svensmark](https://github.com/j-svensmark)
* Build: linting black 24.1.0 rules update [5573](https://github.com/sqlfluff/sqlfluff/pull/5573) [keraion](https://github.com/keraion)
* Snowflake: Updating Snowflake dialect to pass acceptable RLS policy objects [5559](https://github.com/sqlfluff/sqlfluff/pull/5559) [k1drobot](https://github.com/k1drobot)
* Redshift Syntax: ALTER APPEND [5545](https://github.com/sqlfluff/sqlfluff/pull/5545) [OTooleMichael](https://github.com/OTooleMichael)
* DuckDB: Add ANTI, SEMI, ASOF, and POSITIONAL joins [5544](https://github.com/sqlfluff/sqlfluff/pull/5544) [keraion](https://github.com/keraion)
* MySQL: fix FIRST keyword in ALTER ADD/MODIFY [5537](https://github.com/sqlfluff/sqlfluff/pull/5537) [archer62](https://github.com/archer62)
* Postgres/DB2/Oracle: Fix comma join `LATERAL`. [5533](https://github.com/sqlfluff/sqlfluff/pull/5533) [keraion](https://github.com/keraion)
* Add new Rule AL09: Avoid Self Alias [5528](https://github.com/sqlfluff/sqlfluff/pull/5528) [aayushr7](https://github.com/aayushr7)
* Rule AL01: disabled for Oracle dialect [5517](https://github.com/sqlfluff/sqlfluff/pull/5517) [keraion](https://github.com/keraion)
* Postgres ALTER EXTENSION support: dialect & tests [5527](https://github.com/sqlfluff/sqlfluff/pull/5527) [remy-gohiring](https://github.com/remy-gohiring)
* SparkSQL: Add `UNPIVOT` syntax. Fix `TABLESAMPLE` aliases. [5524](https://github.com/sqlfluff/sqlfluff/pull/5524) [keraion](https://github.com/keraion)
* DuckDB: Added support for `PIVOT` and `UNPIVOT` [5514](https://github.com/sqlfluff/sqlfluff/pull/5514) [keraion](https://github.com/keraion)
* Fix parse error databricks window function starts with order by [5493](https://github.com/sqlfluff/sqlfluff/pull/5493) [snkekorfus](https://github.com/snkekorfus)
* Hive: allow UDTF to return multiple column aliases in SELECT [5495](https://github.com/sqlfluff/sqlfluff/pull/5495) [reata](https://github.com/reata)
* DB2: Add support for `DECLARE GLOBAL TEMPORARY TABLES`, `OFFSET`, `CALL`, and non-bracketed `VALUES` [5508](https://github.com/sqlfluff/sqlfluff/pull/5508) [keraion](https://github.com/keraion)
* DuckDB: Add CREATE OR REPLACE TABLE syntax [5511](https://github.com/sqlfluff/sqlfluff/pull/5511) [keraion](https://github.com/keraion)
* TSQL: Top and Distinct in same query [5491](https://github.com/sqlfluff/sqlfluff/pull/5491) [greg-finley](https://github.com/greg-finley)
* [Spark/Databricks] Fix: make COLUMNS in APPLY CHANGES INTO optional [5498](https://github.com/sqlfluff/sqlfluff/pull/5498) [rocwang](https://github.com/rocwang)
* SparkSQL: exclamation mark as logical not [5500](https://github.com/sqlfluff/sqlfluff/pull/5500) [reata](https://github.com/reata)
* SparkSQL: allow value in set_statement to be Java class name [5504](https://github.com/sqlfluff/sqlfluff/pull/5504) [reata](https://github.com/reata)
* SparkSQL: allow distribute/sort/cluster by at end of set operation [5502](https://github.com/sqlfluff/sqlfluff/pull/5502) [reata](https://github.com/reata)
* [CI] Add a few more mypy checks [5505](https://github.com/sqlfluff/sqlfluff/pull/5505) [Koyaani](https://github.com/Koyaani)
* Snowflake dialect: Add support for DATABASE ROLE in GRANT/REVOKE [5490](https://github.com/sqlfluff/sqlfluff/pull/5490) [sfc-gh-dgupta](https://github.com/sfc-gh-dgupta)
* DuckDB: Qualify and From-First [5485](https://github.com/sqlfluff/sqlfluff/pull/5485) [keraion](https://github.com/keraion)
* MySql: create table: allow null/not null in any position [5473](https://github.com/sqlfluff/sqlfluff/pull/5473) [archer62](https://github.com/archer62)
* Snowflake dialect: Support for CREATE DATABASE ROLE [5475](https://github.com/sqlfluff/sqlfluff/pull/5475) [sfc-gh-dgupta](https://github.com/sfc-gh-dgupta)
* Clickhouse Dialect - Support BackQuoted Identifiers [5457](https://github.com/sqlfluff/sqlfluff/pull/5457) [kaiyannameighu](https://github.com/kaiyannameighu)
* Change Color.lightgrey to have a white background - dark theme friendly [5458](https://github.com/sqlfluff/sqlfluff/pull/5458) [ryaminal](https://github.com/ryaminal)
* Fix indentation for single cube clause [5462](https://github.com/sqlfluff/sqlfluff/pull/5462) [tunetheweb](https://github.com/tunetheweb)

New Contributors
* [ryaminal](https://github.com/ryaminal) made their first contribution in [#5458](https://github.com/sqlfluff/sqlfluff/pull/5458)
* [sfc-gh-dgupta](https://github.com/sfc-gh-dgupta) made their first contribution in [#5475](https://github.com/sqlfluff/sqlfluff/pull/5475)
* [archer62](https://github.com/archer62) made their first contribution in [#5473](https://github.com/sqlfluff/sqlfluff/pull/5473)
* [keraion](https://github.com/keraion) made their first contribution in [#5485](https://github.com/sqlfluff/sqlfluff/pull/5485)
* [Koyaani](https://github.com/Koyaani) made their first contribution in [#5505](https://github.com/sqlfluff/sqlfluff/pull/5505)
* [snkekorfus](https://github.com/snkekorfus) made their first contribution in [#5493](https://github.com/sqlfluff/sqlfluff/pull/5493)
* [remy-gohiring](https://github.com/remy-gohiring) made their first contribution in [#5527](https://github.com/sqlfluff/sqlfluff/pull/5527)
* [aayushr7](https://github.com/aayushr7) made their first contribution in [#5528](https://github.com/sqlfluff/sqlfluff/pull/5528)
* [k1drobot](https://github.com/k1drobot) made their first contribution in [#5559](https://github.com/sqlfluff/sqlfluff/pull/5559)
* [alangner](https://github.com/alangner) made their first contribution in [#5574](https://github.com/sqlfluff/sqlfluff/pull/5574)
* [fnimick](https://github.com/fnimick) made their first contribution in [#5577](https://github.com/sqlfluff/sqlfluff/pull/5577)

Highlights

This release makes a breaking change to the serialized output of the
CLI (and by extension, any of the serialized outputs of the API).

* The serialised output for `sqlfluff lint` now contains more information
about the span of linting issues and initial proposed fixes. Beside the *new*
fields, the original fields of `line_pos` and `line_no` have been
renamed to `start_line_pos` and `start_line_no`, to distinguish
them from the new fields starting `end_*`.

* The default `annotation_level` set by the `--annotation-level`
option on the `sqlfluff lint` command has been changed from `notice`
to `warning`, to better distinguish linting errors from warnings, which
always now have the level of `notice`. This is only relevant when using
the `github-annotation` or `github-annotation-native` formats.

What’s Changed

* Build out rule and fix serialisation [5364](https://github.com/sqlfluff/sqlfluff/pull/5364) [alanmcruickshank](https://github.com/alanmcruickshank)
* Add warning about github actions annotations limit [5450](https://github.com/sqlfluff/sqlfluff/pull/5450) [alanmcruickshank](https://github.com/alanmcruickshank)
* chore: remove unused line initialization in ParseContext [5448](https://github.com/sqlfluff/sqlfluff/pull/5448) [gvozdvmozgu](https://github.com/gvozdvmozgu)
* Refine CLI testing fixture [5446](https://github.com/sqlfluff/sqlfluff/pull/5446) [alanmcruickshank](https://github.com/alanmcruickshank)
* Update black linting [5447](https://github.com/sqlfluff/sqlfluff/pull/5447) [alanmcruickshank](https://github.com/alanmcruickshank)