Databricks-labs-lsql

* Make hatch a prerequisite ([259](https://github.com/databrickslabs/lsql/issues/259)). In this commit, Eric Vergnaud has introduced a change to make the installation of `hatch` version 1.9.4 a prerequisite for the project to avoid errors related to `pip` command recognition. The Makefile has been updated to handle the installation of hatch automatically, and the `hatch env create` command is now used instead of `pip install hatch==1.7.0`. This change ensures that the development environment is consistent and reliable by maintaining the correct version of hatch and automatically handling its installation. Additionally, the `.venv/bin/python` and `dev` targets have been updated accordingly to reflect these changes. This commit also formats all files using the `make dev fmt` command, which helps maintain consistent code formatting throughout the project.
* add support for exclusions in `fmt` command ([263](https://github.com/databrickslabs/lsql/issues/263)). In this release, we have added support for exclusions to the `fmt` command in the 'databricks/labs/lsql/cli.py' module. This feature allows users to specify a list of directories or files to exclude while formatting SQL files, which is particularly useful when verifying SQL notebooks in ucx. The `fmt` command now accepts a new optional parameter 'exclude', which accepts an iterable of strings that specify the relative paths to exclude. Any `sql_file` that is a descendant of any `exclusion` is skipped during formatting. The exclusions are implemented by converting the relative paths into `Path` objects. This change addresses the issue where single line comments are converted into inlined comments, causing misinterpretation. The added unit test is manually verified, and this pull request fixes issue [#261](https://github.com/databrickslabs/lsql/issues/261). This feature was authored and co-authored by Eric Vergnaud.

* Fixed dataclass field types ([257](https://github.com/databrickslabs/lsql/issues/257)). This PR introduces a workaround to a Python bug affecting the `dataclasses.fields()` function, which sometimes returns field types as string type names instead of types. This can cause the ORM to malfunction. The workaround involves checking if the returned `f.type` is a string, and if so, converting it to a type by looking it up in the `__builtins__` dictionary. This change is global and affects the `_schema_for` function in the `backends.py` file, which is responsible for creating a schema for a given dataclass, taking into account any necessary type conversions. This change ensures consistent and accurate type handling in the face of the Python bug, improving the reliability of our ORM.
* Fixed missing EOL when formatting SQL files ([260](https://github.com/databrickslabs/lsql/issues/260)). In this release, we have addressed an issue related to the inconsistent addition of end-of-line (EOL) characters in formatted SQL files. The `QueryTile.format()` method has been updated to ensure that an EOL character is always added, except when the input query already ends with a newline. This change enhances the reliability of the SQL formatting functionality, making the output format more predictable and improving the overall user experience. The new implementation is demonstrated in the `test_query_format_preserves_eol()` test case, and existing test cases have been updated to check for the presence of EOL characters, further ensuring consistent and correct formatting.
* Fixed normalize case input in cli ([258](https://github.com/databrickslabs/lsql/issues/258)). In this release, we have updated the `fmt` command in the `cli.py` file to allow users to specify whether they want to normalize the case of SQL files when formatting. The `normalize_case` parameter now defaults to the string `"true"` and checks if it is in the `STRING_AFFIRMATIVES` list to determine whether to normalize the case of SQL files. Additionally, we have introduced a new optional `normalize_case` parameter in the `format` method of the `dashboards.py` file in the Databricks CLI, which normalizes the identifiers in the query to lower case when set to `True`. We have also added support for a new `normalize_case` parameter in the `QueryTile.format()` method, which prevents the automatic normalization of string input to uppercase when set to `False`. This change allows for more flexibility in handling string input and ensures that the input string is preserved as-is. These updates improve the functionality and usability of the open-source library, providing more control to users over formatting and handling of string input.

* Added design for filter file ([251](https://github.com/databrickslabs/lsql/issues/251)). A new feature has been added to enable the creation of filters for multiple widgets in a dashboard using a `.filter.json` file. This file allows users to specify columns to be filtered, the filter type, title, description, order, and a unique ID for each filter. Both the `column` and `columns` flags are supported, with the former taking a single string and the latter taking a list of strings. The filter type can be set to a drop-down menu or another type as desired. The `.filter.json` file schema also supports optional `title` and `description` strings, as well as `order` and `ID` flags. An example of a `.filter.json` file is provided in the commit message. Additionally, the `dashboard.yml` file documentation has been updated to include information on how to use the new `.filter.json` file.
* adding normalize-case option to databricks labs lsql fmt cmd ([254](https://github.com/databrickslabs/lsql/issues/254)). In this open-source library release, the `databricks labs lsql` tool's `fmt` command now supports a new flag, `normalize-case`. This flag allows users to control the normalization of query text to lowercase, providing more flexibility when formatting SQL queries. By default, query text is still normalized to lowercase, but users can now prevent this behavior by setting the `normalize-case` flag to `False`. This change addresses an issue where some queries are case sensitive, such as those using map field keys in UCX dashboards. Additionally, a new parameter `normalize_case` has been added to the `format` method in the `dashboards.py` file, with updated method documentation. A new test function, `test_query_formats_no_normalize()`, has also been included to ensure consistent formatter behavior.

* Removed deploy_dashboard method ([240](https://github.com/databrickslabs/lsql/issues/240)). In this release, the `deploy_dashboard` method has been removed from the `dashboards.py` file and the legacy deployment method has been deprecated. The `deploy_dashboard` method was previously used to deploy a dashboard to a workspace, but it has been replaced with the `create` method of the `lakeview` attribute of the WorkspaceClient object. Additionally, the `test_dashboards_creates_dashboard_via_legacy_method` method has been removed. A new test has been added to ensure that the `deploy_dashboard` method is no longer being used, utilizing the `deprecated_call` function from pytest to verify that calling the method raises a deprecation warning. This change simplifies the code and improves the overall design of the system, resolving issue [#232](https://github.com/databrickslabs/lsql/issues/232). The `_with_better_names` method and `create_dashboard` method remain unchanged.
* Skip test that fails due to insufficient permission to create schema ([248](https://github.com/databrickslabs/lsql/issues/248)). A new test function, `test_dashboards_creates_dashboard_with_replace_database`, has been added to the open-source library, but it is currently marked to be skipped due to missing permissions to create a schema. This function creates an instance of the `Dashboards` class with the `ws` parameter, creates a dashboard using the `make_dashboard` function, and performs various actions using the created dashboard, as well as functions such as `tmp_path` and `sql_backend`. This test function aims to ensure that the `Dashboards` class functions as expected when creating a dashboard with a replaced database. Once the necessary permissions for creating a schema are acquired, this test function can be enabled for further testing and validation.
* Updates to use the Databricks Python sdk 0.30.0 ([247](https://github.com/databrickslabs/lsql/issues/247)). In this release, we have updated the project to use Databricks Python SDK version 0.30.0. This update includes changes to the `execute` and `fetch_value` functions, which now use the new `StatementResponse` type instead of `ExecuteStatementResponse`. A conditional import statement has been added to maintain compatibility with both Databricks SDK versions 0.30.0 and below. The `execute` function now raises `TimeoutError` when the specified timeout is greater than 50 seconds and the statement execution hasn't finished. Additionally, the `fetch_value` function has been updated to handle the case when the `execute` function returns `None`. The unit test file `test_backends.py` has also been updated to reflect these changes, with multiple test functions now using the `StatementResponse` class instead of `ExecuteStatementResponse`. These changes improve the system's compatibility with the latest version of the Databricks SDK, ensuring that the core functionality of the SDK continues to work as expected.

* Fixed missing widget name suffixes ([243](https://github.com/databrickslabs/lsql/issues/243)). In this release, we have addressed an issue related to missing widget name suffixes ([#243](https://github.com/databrickslabs/lsql/issues/243)) by adding a `_widget` suffix to the name of the widget object in the dashboards.py file. This change ensures consistency between the widget name and the id of the query, facilitating user understanding of the relationship between the two. A new method, _get_query_widget_spec, has also been added, although its specific functionality requires further investigation. Additionally, the unit tests in the `tests/unit/test_dashboards.py` file have been updated to check for the presence of the `_widget` suffix in widget names, ensuring that the tests accurately reflect the desired behavior. These changes improve the consistency of dashboard widget naming, thus benefiting software engineers utilizing or extending the project's widget-ordering functionalities.

* Fixed dataset/widget name uniqueness requirement that was preventing dashboards being deployed ([241](https://github.com/databrickslabs/lsql/issues/241)). A fix has been implemented to address a uniqueness requirement issue with the dataset/widget name that was preventing dashboard deployment. A new `widget` instance is now created with a unique name, generated by appending `_widget` to the metadata ID, in the `get_layouts` method. This ensures that multiple widgets with the same ID but different content can exist in a single dashboard, thereby meeting the name uniqueness requirement. In the `save_to_folder` method, the widget name is modified by removing the `_widget` suffix before writing the textbox specification to a markdown file, maintaining consistency between the widget ID and file name. These changes are localized to the `get_layouts` and `save_to_folder` methods, and no new methods have been added. The existing functionality related to the creation, validation, and saving of dashboard layouts remains unaltered.