Databricks-labs-lsql

* Fixed PyPI badge ([72](https://github.com/databrickslabs/lsql/issues/72)). In this release, we have implemented a fix to the PyPI badge in the README file of our open-source library. The PyPI badge displays the version of the package and serves as a quick reference for users. This fix ensures the accuracy and proper functioning of the badge, without involving any changes to the functionality or methods within the project. Software engineers can be assured that this update is limited to the README file, specifically the PyPI badge, and will not affect the overall functionality of the library.
* Fixed `no-cheat` check ([71](https://github.com/databrickslabs/lsql/issues/71)). In this release, we have made improvements to the `no-cheat` verification process for new code. Previously, the check for disabling the linter was prone to false positives when the string '# pylint: disable' appeared for reasons other than disabling the linter. The updated code now includes an additional filter to exclude the string `CHEAT` from the search, and the number of characters in the output is counted using the `wc -c` command. If the count is not zero, the script will terminate with an error message. This change enhances the accuracy of the `no-cheat` check, ensuring that the linter is being used correctly and that all new code meets our quality standards.
* Removed upper bound on `sqlglot` dependency ([70](https://github.com/databrickslabs/lsql/issues/70)). In this update, we have removed the upper bound on the `sqlglot` dependency version in the project's `pyproject.toml` file. Previously, the version constraint required `sqlglot` to be at least 22.3.1 but less than 22.5.0. With this modification, there will be no upper limit, enabling the project to utilize any version greater than or equal to 22.3.1. This change provides the project with the flexibility to take advantage of future bug fixes, performance improvements, and new features available in newer `sqlglot` package versions. Developers should thoroughly test the updated package version to ensure compatibility with the existing codebase.

* Fixed `Builder` object is not callable error ([67](https://github.com/databrickslabs/lsql/issues/67)). In this release, we have made an enhancement to the `Backends` class in the `databricks/labs/lsql/backends.py` file. The `DatabricksSession.builder()` method call in the `__init__` method has been changed to `DatabricksSession.builder`. This update uses the `builder` attribute to create a new instance of `DatabricksSession` without calling it like a function. The `sdk_config` method is then used to configure the instance with the required settings. Finally, the `getOrCreate` method is utilized to obtain a `SparkSession` object, which is then passed as a parameter to the parent class constructor. This modification simplifies the code and eliminates the error caused by treating the `builder` attribute as a callable object. Software engineers may benefit from this change by having a more streamlined and error-free codebase when working with the open-source library.
* Prevent silencing of `pylint` ([65](https://github.com/databrickslabs/lsql/issues/65)). In this release, we have introduced a new job, "no-lint-disabled", to the GitHub Actions workflow for the repository. This job runs on the latest Ubuntu version and checks out the codebase with a full history. It verifies that no new instances of code suppressing `pylint` checks have been added, by filtering the differences between the current branch and the main branch for new lines of code, and then checking if any of those new lines contain a `pylint` disable comment. If any such lines are found, the job will fail and print a message indicating the offending lines of code, thereby ensuring that the codebase maintains a consistent level of quality by not allowing linting checks to be bypassed.
* Updated `_SparkBackend.fetch()` to return iterator instead of list ([62](https://github.com/databrickslabs/lsql/issues/62)). In this release, the `fetch()` method of the `_SparkBackend` class has been updated to return an iterator instead of a list, which can result in reduced memory usage and improved performance, as the results of the SQL query can now be processed one element at a time. A new exception has been introduced to wrap any exceptions that occur during query execution, providing better debugging and error handling capabilities. The `test_runtime_backend_fetch()` unit test has been updated to reflect this change, and users of the `fetch()` method should be aware that it now returns an iterator and must be consumed to obtain the desired data. Thorough testing is recommended to ensure that the updated method still meets the needs of the application.

* Added support for common parameters in StatementExecutionBackend ([59](https://github.com/databrickslabs/lsql/issues/59)). The `StatementExecutionBackend` class in the `databricks.labs.lsql` package's `backends.py` file now supports the passing of common parameters through keyword arguments (kwargs). This enhancement allows for greater customization and flexibility in the backend's operation, as the kwargs are passed to the `StatementExecutionExt` constructor. This change empowers users to control the behavior of the backend, making it more adaptable to various use cases. The key modification in this commit is the addition of the `**kwargs` parameter in the constructor signature and passing it to `StatementExecutionExt`, with no changes made to any methods within the class.

* Updating packages. In this update, the dependencies specified in the pyproject.toml file have been updated to more recent versions. The outdated packages "databricks-labs-blueprint~=0.4.0" and "databricks-sdk~=0.21.0" have been replaced with "databricks-labs-blueprint>=0.4.2" and "databricks-sdk>=0.22.0", respectively. These updates are expected to bring new features and bug fixes to the software. The dependency `sqlglot` remains unchanged, with the same version requirement range of "sqlglot>=22.3.1,<22.5.0". These updates ensure that the software will function as intended, while also taking advantage of the enhancements provided by the more recent versions of the packages.

* Fixed row converter to properly handle nullable values ([53](https://github.com/databrickslabs/lsql/issues/53)). In this release, the row converter in the `databricks.labs.lsql.core` module has been updated to handle nullable values correctly. A new method `StatementExecutionExt` has been added, which manages the handling of nullable values during SQL statement execution. The `Row` class has also been modified to include nullable values, improving the robustness and flexibility of SQL execution in dealing with various data types, including null values. These enhancements increase the overall reliability of the system, making it more production-ready.
* Improved integration test coverage ([52](https://github.com/databrickslabs/lsql/issues/52)). In this release, the project's integration test coverage has been significantly improved through several changes. A new function, `make_random()`, has been added to the `conftest.py` file to generate a random string of fixed length, aiding in the creation of more meaningful and readable random strings for integration tests. A new file, `test_deployment.py`, has been introduced, containing a test function for deploying a database schema and verifying successful data retrieval via a view. The `test_integration.py` file has been renamed to `test_core.py`, with updates to the `test_fetch_one` function to test the `fetch_one` method using a SQL query with an aliased value. Additionally, a new `Foo` dataclass has been added to the `tests/integration/views/__init__.py` file, supporting integration test coverage. Lastly, a new SQL query has been added to the integration test suite, located in the `some.sql` file, which retrieves data from a table named `foo` in the `inventory` schema. These changes aim to enhance the overall stability, reliability, and coverage of the project's integration tests. Note: The changes to the `.gitignore` file and the improvements to the `StatementExecutionBackend` class in the `backends.py` file are not included in this summary, as they were described in the opening statement.
* Rely on `hatch` being present on the build machine ([54](https://github.com/databrickslabs/lsql/issues/54)). In this release, we have made significant changes to how we manage our build process and toolchain configuration. We have removed the need to manually install `hatch` version 1.7.0 in the build machine, and instead, rely on its presence, adding it to the list of required tools in the toolchain configuration. The command to create a virtual environment using `hatch` has also been added, and the `pre_setup` section no longer includes installing `hatch`, assuming its availability. We have also updated the `hatch` package version from 1.7.0 to 1.9.4, which may include bug fixes, performance improvements, or new features. This change may impact the behavior of any existing functionality that relies on `hatch`. The `pyproject.toml` file has been modified to update the `fmt` and `verify` sections, with `ruff check . --fix` replacing `ruff . --fix` and the removal of `black --check .` and `isort . --check-only`. A new configuration for `isort` has also been added to specify the `databricks.labs.blueprint` package as a known first-party package, enabling more precise management of imports related to that package. These changes simplify the build process and ensure that the project is using a more recent version of the `hatch` package for packaging and distributing Python projects.
* Updated sqlglot requirement from ~=22.3.1 to >=22.3.1,<22.5.0 ([51](https://github.com/databrickslabs/lsql/issues/51)). In this release, we have updated the version constraint for the `sqlglot` package in our project's `pyproject.toml` file. Previously, we had set the constraint to `~=22.3.1`, allowing for any version with the same major and minor numbers but different patch numbers. With this update, we have changed the constraint to `>=22.3.1,<22.5.0`. This change enables our project to utilize bug fixes and improvements made in the latest patch versions of `sqlglot`, while still preventing it from inadvertently using any breaking changes introduced in version 22.5.0 or later versions. This modification allows us to take advantage of the latest features and improvements in `sqlglot` while maintaining compatibility and stability in our project.

Dependency updates:

* Updated sqlglot requirement from ~=22.3.1 to >=22.3.1,<22.5.0 ([51](https://github.com/databrickslabs/lsql/pull/51)).

* Added `MockBackend.rows("col1", "col2")[(...), (...)]` helper ([49](https://github.com/databrickslabs/lsql/issues/49)). In this release, we have added a new helper method `MockBackend.rows("col1", "col2")[(...), (...)]` to simplify testing with `MockBackend`. This method allows for the creation of rows using a more concise syntax, taking in the column names and a list of values to be used for each column, and returning a list of `Row` objects with the specified columns and values. Additionally, a `__eq__` method has been introduced to check if two rows are equal by converting the rows to dictionaries using the existing `as_dict` method and comparing them. The `__contains__` method has also been modified to improve the behavior of the `in` keyword when used with rows, ensuring columns can be checked for membership in the row in a more intuitive and predictable manner. These changes make it easier to test and work with `MockBackend`, improving overall quality and maintainability of the project.