Sqlfluff

Highlights

This is the first alpha version for 2.0.0. It brings all of the changes to whitespace
handing, including a total rewrite of indentation and long line logic (L003 & L016).
That brings several breaking changes to the configuration of layout, see the
[layout docs](https://docs.sqlfluff.com/en/stable/layout.html) for more details and
familiarise yourself with the new
[default configuration](https://docs.sqlfluff.com/en/stable/configuration.html#default-configuration).

In addition, for the dbt templater, this introduces a large re-write of the codebase,
dropping support for dbt versions before 1.0.0. This leverages functionality from
[dbt-osmosis](https://github.com/z3z1ma/dbt-osmosis) to reduce the amount of
functionality supported directly by SQLFluff, and performance during testing of the new
version has been reported as significantly faster.

What’s Changed

* Fixed False Positive for L037 [4198](https://github.com/sqlfluff/sqlfluff/pull/4198) [WillAyd](https://github.com/WillAyd)
* Indentation bug [4217](https://github.com/sqlfluff/sqlfluff/pull/4217) [alanmcruickshank](https://github.com/alanmcruickshank)
* Show fatal errors regardless [4214](https://github.com/sqlfluff/sqlfluff/pull/4214) [alanmcruickshank](https://github.com/alanmcruickshank)
* Don't consider templated whitespace [4213](https://github.com/sqlfluff/sqlfluff/pull/4213) [alanmcruickshank](https://github.com/alanmcruickshank)
* Don't pickle the templater [4208](https://github.com/sqlfluff/sqlfluff/pull/4208) [alanmcruickshank](https://github.com/alanmcruickshank)
* MySQL: Support column character set and collation [4204](https://github.com/sqlfluff/sqlfluff/pull/4204) [yoichi](https://github.com/yoichi)
* Fix some issues with Docker Compose environment [4201](https://github.com/sqlfluff/sqlfluff/pull/4201) [barrywhart](https://github.com/barrywhart)
* Implicit Indents [4054](https://github.com/sqlfluff/sqlfluff/pull/4054) [alanmcruickshank](https://github.com/alanmcruickshank)
* Tweak Coveralls settings [4199](https://github.com/sqlfluff/sqlfluff/pull/4199) [barrywhart](https://github.com/barrywhart)
* In addition to Codecov, also upload to Coveralls [4197](https://github.com/sqlfluff/sqlfluff/pull/4197) [barrywhart](https://github.com/barrywhart)
* Fix: create table default cast returns unparsable section [4192](https://github.com/sqlfluff/sqlfluff/pull/4192) [NelsonTorres](https://github.com/NelsonTorres)
* Fix JSON parsing issue with diff-quality plugin [4190](https://github.com/sqlfluff/sqlfluff/pull/4190) [barrywhart](https://github.com/barrywhart)
* Codecov migration [4195](https://github.com/sqlfluff/sqlfluff/pull/4195) [alanmcruickshank](https://github.com/alanmcruickshank)
* Stop adding trailing os.sep if ignore file is on the root of the file… [4182](https://github.com/sqlfluff/sqlfluff/pull/4182) [baa-ableton](https://github.com/baa-ableton)
* Port dbt-osmosis templater changes to SQLFluff [3976](https://github.com/sqlfluff/sqlfluff/pull/3976) [barrywhart](https://github.com/barrywhart)
* Reflow 4: Long Lines [4067](https://github.com/sqlfluff/sqlfluff/pull/4067) [alanmcruickshank](https://github.com/alanmcruickshank)
* Fix comment bug on reindent [4179](https://github.com/sqlfluff/sqlfluff/pull/4179) [alanmcruickshank](https://github.com/alanmcruickshank)
* Reflow 3: Reindent [3942](https://github.com/sqlfluff/sqlfluff/pull/3942) [alanmcruickshank](https://github.com/alanmcruickshank)

New Contributors

* [baa-ableton](https://github.com/baa-ableton) made their first contribution in [#4182](https://github.com/sqlfluff/sqlfluff/pull/4182)
* [WillAyd](https://github.com/WillAyd) made their first contribution in [#4198](https://github.com/sqlfluff/sqlfluff/pull/4198)

Highlights

This is a bugfix release, primarily for diff-quality. In addition, a new rules for spacing
around parenthesis is also included.

This is also the final 1.x.x release. Following releases will be a series of alpha releases
for 2.x.x. If you affected by any outstanding bugs or regressions from this release, consider
either rolling _backward_ to a previous release without those issues, or failing _forward_
to on an alpha release for 2.x.x (or a full release if that's out). Note that 2.x.x will bring
a selection of breaking changes to config file structure, rule categorisation and dbt support.

What’s Changed

* Add rule for space around parenthesis [4131](https://github.com/sqlfluff/sqlfluff/pull/4131) [NelsonTorres](https://github.com/NelsonTorres)
* diff-quality plugin: Print invalid JSON on parse failure [4176](https://github.com/sqlfluff/sqlfluff/pull/4176) [barrywhart](https://github.com/barrywhart)
* Ensure diff-quality runs the correct SQLFluff [4175](https://github.com/sqlfluff/sqlfluff/pull/4175) [barrywhart](https://github.com/barrywhart)

New Contributors

* [NelsonTorres](https://github.com/NelsonTorres) made their first contribution in [#4131](https://github.com/sqlfluff/sqlfluff/pull/4131)

Highlights

Bug fix for 1.4.3 which was incorrectly flagging L006 for concat operators (`||`) and other two-symbol binary operators.

What’s Changed

* Recognise || as an operator to avoid rule L006 flagging it [4168](https://github.com/sqlfluff/sqlfluff/pull/4168) [tunetheweb](https://github.com/tunetheweb)
* :bug: Check verbosity level of pytest run before running certain tests [4167](https://github.com/sqlfluff/sqlfluff/pull/4167) [pwildenhain](https://github.com/pwildenhain)
* [snowflake] Add support for snowflake select * exclude/replace syntax [4160](https://github.com/sqlfluff/sqlfluff/pull/4160) [moreaupascal56](https://github.com/moreaupascal56)

Highlights

* Rewrote `diff-quality` plugin to run SQLFluff as a subprocess. More reliable, easier to switch between `diff-quality` and running `sqlfluff lint` directly.
* New rule L067 enforces consistent syntax for type casts.
* New rule L068 enforces a consistent number of columns in set queries (e.g. UNION).
* Initial support for Materialize dialect.

What's Changed
* Add flyway variables support via placeholder templater [4026](https://github.com/sqlfluff/sqlfluff/pull/4026) [srjonemed](https://github.com/srjonemed)
* Fix Spark comparison parsing [4066](https://github.com/sqlfluff/sqlfluff/pull/4066) [ms32035](https://github.com/ms32035)
* Add errors and fails to pytest summary [4076](https://github.com/sqlfluff/sqlfluff/pull/4076) [alanmcruickshank](https://github.com/alanmcruickshank)
* Storage reference segment [4057](https://github.com/sqlfluff/sqlfluff/pull/4057) [YilangHe](https://github.com/YilangHe)
* New rule L069: Consistent syntax for sql type casting [3747](https://github.com/sqlfluff/sqlfluff/pull/3747) [bolajiwahab](https://github.com/bolajiwahab)
* Postgres: Views and named notations [4073](https://github.com/sqlfluff/sqlfluff/pull/4073) [davetapley](https://github.com/davetapley)
* Switch reflow buffer from LintFix to LintResult [4083](https://github.com/sqlfluff/sqlfluff/pull/4083) [alanmcruickshank](https://github.com/alanmcruickshank)
* Support parallel linting when many individual files specified [4084](https://github.com/sqlfluff/sqlfluff/pull/4084) [barrywhart](https://github.com/barrywhart)
* Rule: check number of columns in set operations match [4028](https://github.com/sqlfluff/sqlfluff/pull/4028) [erevear](https://github.com/erevear)
* Fix syntax in indentation sample code docs [4087](https://github.com/sqlfluff/sqlfluff/pull/4087) [PBalsdon](https://github.com/PBalsdon)
* Remove "mystery" binary file added in PR 2923 [4088](https://github.com/sqlfluff/sqlfluff/pull/4088) [barrywhart](https://github.com/barrywhart)
* Fix mypy issue with regex [4097](https://github.com/sqlfluff/sqlfluff/pull/4097) [barrywhart](https://github.com/barrywhart)
* Enable variable reference names to have leading underscore for snowflake dialect [4098](https://github.com/sqlfluff/sqlfluff/pull/4098) [gavin-tsang](https://github.com/gavin-tsang)
* Fix flake8 issue with segment_predicates.py [4101](https://github.com/sqlfluff/sqlfluff/pull/4101) [barrywhart](https://github.com/barrywhart)
* Fix bug in example rule plugin [4103](https://github.com/sqlfluff/sqlfluff/pull/4103) [barrywhart](https://github.com/barrywhart)
* Fix bug where L034 should ignore INSERT or "CREATE TABLE AS SELECT" with CTE [4108](https://github.com/sqlfluff/sqlfluff/pull/4108) [barrywhart](https://github.com/barrywhart)
* Postgres: Alter type rename value [4100](https://github.com/sqlfluff/sqlfluff/pull/4100) [greg-finley](https://github.com/greg-finley)
* Bug fix: dbt templater ignores .sqlfluff file encoding on Windows [4109](https://github.com/sqlfluff/sqlfluff/pull/4109) [barrywhart](https://github.com/barrywhart)
* Add initial Materialize dialect [4112](https://github.com/sqlfluff/sqlfluff/pull/4112) [bobbyiliev](https://github.com/bobbyiliev)
* L015: Handle COUNT(DISTINCT(col)) [4110](https://github.com/sqlfluff/sqlfluff/pull/4110) [barrywhart](https://github.com/barrywhart)
* [Snowflake] format type options extensions for copy_into_location [4129](https://github.com/sqlfluff/sqlfluff/pull/4129) [YilangHe](https://github.com/YilangHe)
* Fix tox arguments [4144](https://github.com/sqlfluff/sqlfluff/pull/4144) [greg-finley](https://github.com/greg-finley)
* [DB2] Fix parsing of string identifiers [4134](https://github.com/sqlfluff/sqlfluff/pull/4134) [borchero](https://github.com/borchero)
* BigQuery: Allow double quoted literal in export_option_list [4126](https://github.com/sqlfluff/sqlfluff/pull/4126) [yoichi](https://github.com/yoichi)
* Only upload 3 sets of test results to codecov (possible workaround for hanging builds) [4147](https://github.com/sqlfluff/sqlfluff/pull/4147) [barrywhart](https://github.com/barrywhart)
* SparkSQL: ILIKE [4138](https://github.com/sqlfluff/sqlfluff/pull/4138) [greg-finley](https://github.com/greg-finley)
* SparkSQL: Mark `AS` as optional keyword for CTE & CTS [4127](https://github.com/sqlfluff/sqlfluff/pull/4127) [ulixius9](https://github.com/ulixius9)
* Fix passenv to work with tox 4 [4154](https://github.com/sqlfluff/sqlfluff/pull/4154) [tunetheweb](https://github.com/tunetheweb)
* Allow deprecated --disable_progress_bar flag for fix command [4151](https://github.com/sqlfluff/sqlfluff/pull/4151) [pdebelak](https://github.com/pdebelak)
* Implement diff_quality_plugin using command-line rather than Python [4148](https://github.com/sqlfluff/sqlfluff/pull/4148) [barrywhart](https://github.com/barrywhart)
* L037: insert ASC just after column_reference [4149](https://github.com/sqlfluff/sqlfluff/pull/4149) [yoichi](https://github.com/yoichi)

New Contributors
* [srjonemed](https://github.com/srjonemed) made their first contribution in [#4026](https://github.com/sqlfluff/sqlfluff/pull/4026)
* [ms32035](https://github.com/ms32035) made their first contribution in [#4066](https://github.com/sqlfluff/sqlfluff/pull/4066)
* [davetapley](https://github.com/davetapley) made their first contribution in [#4073](https://github.com/sqlfluff/sqlfluff/pull/4073)
* [PBalsdon](https://github.com/PBalsdon) made their first contribution in [#4087](https://github.com/sqlfluff/sqlfluff/pull/4087)
* [gavin-tsang](https://github.com/gavin-tsang) made their first contribution in [#4098](https://github.com/sqlfluff/sqlfluff/pull/4098)
* [bobbyiliev](https://github.com/bobbyiliev) made their first contribution in [#4112](https://github.com/sqlfluff/sqlfluff/pull/4112)
* [ulixius9](https://github.com/ulixius9) made their first contribution in [#4127](https://github.com/sqlfluff/sqlfluff/pull/4127)

Highlights

This release is less about internals and much more about some quality of life
improvements and dialect changes. The most notable are:
- The introduction of a `sqlfluff render` command to preview the results of
templated sql.
- Linting errors within templated loops should now only appear once in the
linting output.
- Indentation around jinja `{% set %}` statements should now be more consistent.
- Linting errors around unparsable code are now more appropriately handled (with
more to come soon on that front).
- Error messages when specified files aren't found are now more specific.

We've also got dialect improvements for Redshift, SOQL & SparkSQL.

What’s Changed

* Fix type error in `get_rules` hook of plugin example [4060](https://github.com/sqlfluff/sqlfluff/pull/4060) [Samyak2](https://github.com/Samyak2)
* L003: Add missing "pragma: no cover" [4058](https://github.com/sqlfluff/sqlfluff/pull/4058) [barrywhart](https://github.com/barrywhart)
* Fix bug in sparksql SELECT statement termination at UNION 4050 [4052](https://github.com/sqlfluff/sqlfluff/pull/4052) [anna-azizian](https://github.com/anna-azizian)
* Deduplicate violations in the source space [4041](https://github.com/sqlfluff/sqlfluff/pull/4041) [alanmcruickshank](https://github.com/alanmcruickshank)
* Use "docker compose", not "docker-compose" [4055](https://github.com/sqlfluff/sqlfluff/pull/4055) [barrywhart](https://github.com/barrywhart)
* Allow warnings for specific rules [4053](https://github.com/sqlfluff/sqlfluff/pull/4053) [alanmcruickshank](https://github.com/alanmcruickshank)
* Better file not found error 1023 [4051](https://github.com/sqlfluff/sqlfluff/pull/4051) [alanmcruickshank](https://github.com/alanmcruickshank)
* Filter out issues in unparsable sections [4032](https://github.com/sqlfluff/sqlfluff/pull/4032) [alanmcruickshank](https://github.com/alanmcruickshank)
* Snowflake: ADD and DROP without COLUMN [4049](https://github.com/sqlfluff/sqlfluff/pull/4049) [greg-finley](https://github.com/greg-finley)
* Make render command [4043](https://github.com/sqlfluff/sqlfluff/pull/4043) [alanmcruickshank](https://github.com/alanmcruickshank)
* Bump after_n_builds to 10 [4046](https://github.com/sqlfluff/sqlfluff/pull/4046) [greg-finley](https://github.com/greg-finley)
* Redshift: allows for parenthesis around FROM content [3962](https://github.com/sqlfluff/sqlfluff/pull/3962) [adam-tokarski](https://github.com/adam-tokarski)
* Update CI to use Python 3.11 [4038](https://github.com/sqlfluff/sqlfluff/pull/4038) [greg-finley](https://github.com/greg-finley)
* Classify self contained set statements as templated [4034](https://github.com/sqlfluff/sqlfluff/pull/4034) [alanmcruickshank](https://github.com/alanmcruickshank)
* Date and Datetime literals in SOQL [4037](https://github.com/sqlfluff/sqlfluff/pull/4037) [alanmcruickshank](https://github.com/alanmcruickshank)
* mypy edits for 0.990 [4035](https://github.com/sqlfluff/sqlfluff/pull/4035) [alanmcruickshank](https://github.com/alanmcruickshank)
* sparksql: support for create/remove widget clause [4021](https://github.com/sqlfluff/sqlfluff/pull/4021) [Coola4kov](https://github.com/Coola4kov)
* Redshift CREATE EXTERNAL FUNCTION statement [4011](https://github.com/sqlfluff/sqlfluff/pull/4011) [rpr-ableton](https://github.com/rpr-ableton)
* Update Redshift bare functions [4012](https://github.com/sqlfluff/sqlfluff/pull/4012) [rpr-ableton](https://github.com/rpr-ableton)


New Contributors
* [Coola4kov](https://github.com/Coola4kov) made their first contribution in [#4021](https://github.com/sqlfluff/sqlfluff/pull/4021)
* [anna-azizian](https://github.com/anna-azizian) made their first contribution in [#4052](https://github.com/sqlfluff/sqlfluff/pull/4052)

Highlights

This is a fix to the configuration migration from 1.4.0. In that release, the configuration
of leading/trailing operators would be migrated the wrong way around and precedence between
new and old configuration values behaved unexpectedly.

What’s Changed

* Config precedence [4007](https://github.com/sqlfluff/sqlfluff/pull/4007) [alanmcruickshank](https://github.com/alanmcruickshank)
* Redshift CREATE/ATTACH/DETACH/DROP RLS POLICY statements [4004](https://github.com/sqlfluff/sqlfluff/pull/4004) [rpr-ableton](https://github.com/rpr-ableton)
* Redshift: Add support for APPROXIMATE functions [3997](https://github.com/sqlfluff/sqlfluff/pull/3997) [rpr-ableton](https://github.com/rpr-ableton)
* hotfix to config migration [4005](https://github.com/sqlfluff/sqlfluff/pull/4005) [alanmcruickshank](https://github.com/alanmcruickshank)