Databricks-labs-ucx

* Added ACL migration to `migrate-tables` workflow ([1135](https://github.com/databrickslabs/ucx/issues/1135)).
* Added AVRO to supported format to be upgraded by SYNC ([1134](https://github.com/databrickslabs/ucx/issues/1134)). In this release, the `hive_metastore` package's `tables.py` file has been updated to add AVRO as a supported format for the SYNC upgrade functionality. This change includes AVRO in the list of supported table formats in the `is_format_supported_for_sync` method, which checks if the table format is not `None` and if the format's uppercase value is one of the supported formats. The addition of AVRO enables it to be upgraded using the SYNC functionality. Moreover, a new format called BINARYFILE has been introduced, which is not supported for SYNC upgrade. This release is part of the implementation of issue [#1134](https://github.com/databrickslabs/ucx/issues/1134), improving the compatibility of the SYNC upgrade functionality with various data formats.
* Added `is_partitioned` column ([1130](https://github.com/databrickslabs/ucx/issues/1130)). A new column, `is_partitioned`, has been added to the `ucx.tables` table in the assessment module, indicating whether the table is partitioned or not with values `Yes` or "No". This change addresses issue [#871](https://github.com/databrickslabs/ucx/issues/871) and has been manually tested. The commit also includes updated documentation for the modified table. No new methods, CLI commands, workflows, or tests (unit, integration) have been introduced as part of this change.
* Added assessment of interactive cluster usage compared to UC compute limitations ([1123](https://github.com/databrickslabs/ucx/issues/1123)).
* Added external location validation when creating catalogs with `create-catalogs-schemas` command ([1110](https://github.com/databrickslabs/ucx/issues/1110)).
* Added flag to Job to identify Job submitted by jar ([1088](https://github.com/databrickslabs/ucx/issues/1088)). The open-source library has been updated with several new features aimed at enhancing user functionality and convenience. These updates include the addition of a new sorting algorithm, which provides users with an efficient and customizable method for organizing data. Additionally, a new caching mechanism has been implemented, improving the library's performance and reducing the amount of time required to access frequently used data. Furthermore, the library now supports multi-threading, enabling users to perform multiple operations simultaneously and increase overall productivity. Lastly, a new error handling system has been developed, providing users with more informative and actionable feedback when unexpected issues arise. These changes are a significant step forward in improving the library's performance, functionality, and usability for all users.
* Bump databricks-sdk from 0.22.0 to 0.23.0 ([1121](https://github.com/databrickslabs/ucx/issues/1121)). In this version update, `databricks-sdk` is upgraded from 0.22.0 to 0.23.0, introducing significant changes to the handling of AWS and Azure identities. The `AwsIamRole` class is replaced with `AwsIamRoleRequest` in the `databricks.sdk.service.catalog` module, affecting the creation of AWS storage credentials using IAM roles. The `create` function in `src/databricks/labs/ucx/aws/credentials.py` is updated to accommodate this modification. Additionally, the `AwsIamRole` argument in the `create` function of `fixtures.py` in the `databricks/labs/ucx/mixins` directory is replaced with `AwsIamRoleRequest`. The tests in `tests/integration/aws/test_access.py` are also updated to utilize `AwsIamRoleRequest`, and `StorageCredentialInfo` in `tests/unit/azure/test_credentials.py` now uses `AwsIamRoleResponse` instead of `AwsIamRole`. The new classes, `AwsIamRoleRequest` and `AwsIamRoleResponse`, likely include new features or bug fixes for AWS IAM roles. These changes require software engineers to thoroughly assess their codebase and adjust any relevant functions accordingly.
* Deploy static views needed by [1123](https://github.com/databrickslabs/ucx/issues/1123) interactive dashboard ([#1139](https://github.com/databrickslabs/ucx/issues/1139)). In this update, we have added two new views, `misc_patterns_vw` and `code_patterns_vw`, to the `install.py` script in the `databricks/labs/ucx` directory. These views were originally intended to be deployed with a previous update ([#1123](https://github.com/databrickslabs/ucx/issues/1123)) but were inadvertently overlooked. The addition of these views addresses issues with queries in the `interactive` dashboard. The `deploy_schema` function has been updated with two new lines, `deployer.deploy_view("misc_patterns", "queries/views/misc_patterns.sql")` and `deployer.deploy_view("code_patterns", "queries/views/code_patterns.sql")`, to deploy the new views using their respective SQL files from the `queries/views` directory. No other modifications have been made to the file.
* Fixed Table ACL migration logic ([1149](https://github.com/databrickslabs/ucx/issues/1149)). The open-source library has been updated with several new features, providing enhanced functionality for software engineers. A new utility class has been added to simplify the process of working with collections, offering methods to filter, map, and reduce elements in a performant manner. Additionally, a new configuration system has been implemented, allowing users to easily customize library behavior through a simple JSON format. Finally, we have added support for asynchronous processing, enabling efficient handling of I/O-bound tasks and improving overall application performance. These features have been thoroughly tested and are ready for use in your projects.
* Fixed `AssertionError: assert '14.3.x-scala2.12' == '15.0.x-scala2.12'` from nightly integration tests ([1120](https://github.com/databrickslabs/ucx/issues/1120)). In this release, the open-source library has been updated with several new features to enhance functionality and provide more options to users. The library now supports multi-threading, allowing for more efficient processing of large datasets. Additionally, a new algorithm for data compression has been implemented, resulting in reduced memory usage and faster data transfer. The library API has also been expanded, with new methods for sorting and filtering data, as well as improved error handling. These changes aim to provide a more robust and performant library, making it an even more valuable tool for software engineers.
* Increase code coverage by 1 percent ([1125](https://github.com/databrickslabs/ucx/issues/1125)).
* Skip installation if remote and local version is the same, provide prompt to override ([1084](https://github.com/databrickslabs/ucx/issues/1084)). In this release, the `new_installation` workflow in the open-source library has been enhanced to include a new use case for handling identical remote and local versions of UCX. When the remote and local versions are the same, the user is now prompted and if no override is requested, a RuntimeWarning is raised. Additionally, users are now prompted to update the existing installation and if confirmed, the installation proceeds. These modifications include manual testing and new unit tests to ensure functionality. These changes provide users with more control over their installation process and address a specific use case for handling identical UCX versions.
* Updated databricks-labs-lsql requirement from ~=0.2.2 to >=0.2.2,<0.4.0 ([1137](https://github.com/databrickslabs/ucx/issues/1137)). The open-source library has been updated with several new features to enhance usability and functionality. Firstly, we have added support for asynchronous processing, allowing for more efficient handling of large data sets and improving overall performance. Additionally, a new configuration system has been implemented, which simplifies the setup process for users and increases customization options. We have also included a new error handling mechanism that provides more detailed and actionable information, making it easier to diagnose and resolve issues. Lastly, we have made significant improvements to the library's documentation, including updated examples, guides, and an expanded API reference. These changes are part of our ongoing commitment to improving the library and providing the best possible user experience.
* [Experimental] Add support for permission migration API ([1080](https://github.com/databrickslabs/ucx/issues/1080)).

Dependency updates:

* Updated databricks-labs-lsql requirement from ~=0.2.2 to >=0.2.2,<0.4.0 ([1137](https://github.com/databrickslabs/ucx/pull/1137)).

* Added instance pool id to WorkspaceConfig ([1087](https://github.com/databrickslabs/ucx/issues/1087)). In this release, the `create` method of the `_policy_installer` object has been updated to return an additional value, `instance_pool_id`, which is then assigned and passed as an argument to the `WorkspaceConfig` object in the `_configure_new_installation` method. The `ClusterPolicyInstaller` class in the `v0.15.0_added_cluster_policy.py` file has also been updated to return a fourth value, `instance_pool_id`, from the `create` method, allowing for more flexibility in future enhancements. Additionally, the test function `test_table_migration_job` in the `test_installation.py` file has been updated to skip when the script is not being run as part of a nightly test job or in debug mode, and the test functions in the `test_policy.py` file have been updated to reflect the new return value in the `create` method. These changes enable better management and scaling of resources through instance pools, provide more granular control in the WorkspaceConfig, and improve testing efficiency.
* Added more cross-linking between CLI commands ([1091](https://github.com/databrickslabs/ucx/issues/1091)). In this release, we have introduced several enhancements to our open-source library's Command Line Interface (CLI) and documentation. Specifically, we have added more cross-linking between CLI commands to improve navigation and usability. The documentation has been updated to include a new step in the UCX installation process, where users are required to run the assessment workflow after installing UCX. This workflow is the first step in the migration process and checks the compatibility of the user's workspace with Unity Catalog. Additionally, we have added new commands for principal-prefix-access, migrate-credentials, and migrate-locations, which are part of the table migration process. These new commands require the assessment workflow and group migration workflow to be completed before they can be executed. Overall, these changes aim to provide a more streamlined and detailed installation and migration process, improving the user experience for software engineers.
* Fixed command references in README.md ([1093](https://github.com/databrickslabs/ucx/issues/1093)). In this release, we have made improvements to the command references in the README.md file to enhance the overall readability and usability of the documentation for software engineers. Specifically, we have updated the links for the `migrate-locations` and `validate_external_locations` commands to use the correct syntax, enclosing them in backticks to denote code. This change ensures that the links are correctly interpreted as commands and addresses any issues that may have arisen with their previous formatting. It is important to note that no new methods have been added in this release, and the existing functionality of the commands has not been changed in scope or functionality.
* Fixing the issue in workspace id flag in create-account-group command ([1094](https://github.com/databrickslabs/ucx/issues/1094)). In this update, we have improved the `create_account_group` command related to the `workspace_ids` flag in our open-source library. The `workspace_ids` flag's type has been changed from `list[int] | None` to `str | None`, allowing for easier input of multiple workspace IDs as a string of comma-separated integers. The `create_account_level_groups` function in the `AccountWorkspaces` class has been updated to accept this string and convert it to a list of integers before proceeding. To ensure proper functioning, we added a new test case `test_create_account_groups_with_id()` to check if the command handles the case when no workspace IDs are provided in the configuration. The `create_account_groups()` method now checks for this condition and raises a `ValueError`. Furthermore, the `manual_workspace_info()` method has been updated to handle workspace name input by the user, receiving the `ws` object, along with prompts that contain the user input for the workspace name and the next workspace ID.
* Rely UCX on the latest 14.3 LTS DBR instead of 15.x ([1097](https://github.com/databrickslabs/ucx/issues/1097)). In this release, we have implemented a quick fix to rely on the Long Term Support (LTS) version 14.3 of the Databricks Runtime (DBR) instead of 15.x for UCX, addressing issue [#1096](https://github.com/databrickslabs/ucx/issues/1096). This change affects the `_definition` function, which has been modified to use the latest LTS DBR instead of the latest Spark version. The `latest_lts_dbr` variable is now assigned the value returned by the `select_spark_version` method with the `latest=True` and `long_term_support=True` parameters. The `spark_version` key in the `policy_definition` dictionary is set to the value returned by the `_policy_config` method with `latest_lts_dbr` as the argument. Additionally, in the `tests/unit/installer/test_policy.py` file, the `select_spark_version` method of the `clusters` object has been updated to accept any number of arguments and consistently return the string "14.2.x-scala2.12", allowing for greater flexibility. This is a temporary solution, with a more comprehensive fix being tracked in issue [#1098](https://github.com/databrickslabs/ucx/issues/1098). Developers should be aware of how the `clusters` object is used in the codebase when adopting this project.

* Added Legacy Table ACL grants migration ([1054](https://github.com/databrickslabs/ucx/issues/1054)). This commit introduces a legacy table ACL grants migration to the `migrate-tables` workflow, resolving issue [#340](https://github.com/databrickslabs/ucx/issues/340) and paving the way for follow-up PRs [#887](https://github.com/databrickslabs/ucx/issues/887) and [#907](https://github.com/databrickslabs/ucx/issues/907). A new `GrantsCrawler` class is added for crawling grants, along with a `GroupManager` class to manage groups during migration. The `TablesMigrate` class is updated to accept an instance of `GrantsCrawler` and `GroupManager` in its constructor. The migration process has been thoroughly tested with unit tests, integration tests, and manual testing on a staging environment. The changes include the addition of a new Enum class `AclMigrationWhat` and updates to the `Table` dataclass, and affect the way tables are selected for migration based on rules. The logging and error handling have been improved in the `skip_schema` function.
* Added `databricks labs ucx cluster-remap` command to remap legacy cluster configurations to UC-compatible ([994](https://github.com/databrickslabs/ucx/issues/994)). In this open-source library update, we have developed and added the `databricks labs ucx cluster-remap` command, which facilitates the remapping of legacy cluster configurations to UC-compatible ones. This new CLI command comes with user documentation to guide the cluster remapping process. Additionally, we have expanded the functionality of creating and managing UC external catalogs and schemas with the inclusion of `create-catalogs-schemas` and `revert-cluster-remap` commands. This change does not modify existing commands or workflows and does not introduce new tables. The `databricks labs ucx cluster-remap` command allows users to re-map and revert the re-mapping of clusters from Unity Catalog (UC) using the CLI, ensuring compatibility and streamlining the migration process. The new command and associated functions have been manually tested for functionality.
* Added `migrate-tables` workflow ([1051](https://github.com/databrickslabs/ucx/issues/1051)). The `migrate-tables` workflow has been added, which allows for more fine-grained control over the resources allocated to the workspace. This workflow includes two new instance variables `min_workers` and `max_workers` in the `WorkspaceConfig` class, with default values of 1 and 10 respectively. A new `trigger` function has also been introduced, which initializes a configuration, SQL backend, and WorkspaceClient based on the provided configuration file. The `run_task` function has been added, which looks up the specified task, logs relevant information, and runs the task's function with the provided arguments. The `Task` class's `fn` attribute now includes an `Installation` object as a parameter. Additionally, a new `migrate-tables` workflow has been added for migrating tables from the Hive Metastore to the Unity Catalog, along with new classes and methods for table mapping, migration status refreshing, and migrating tables. The `migrate_dbfs_root_delta_tables` and `migrate_external_tables_sync` methods perform migrations for Delta tables located in the DBFS root and synchronize external tables, respectively. These functions use the workspace client to access the catalogs and ensure proper migration. Integration tests have also been added for these new methods to ensure their correct operation.
* Added handling for `SYNC` command failures ([1073](https://github.com/databrickslabs/ucx/issues/1073)). This pull request introduces changes to improve handling of `SYNC` command failures during external table migrations in the Hive metastore. Previously, the `SYNC` command's result was not checked, and failures were not logged. Now, the `_migrate_external_table` method in `table_migrate.py` fetches the result of the `SYNC` command execution, logs a warning message for failures, and returns `False` if the command fails. A new integration test has been added to simulate a failed `SYNC` command due to a non-existent catalog and schema, ensuring the migration tool handles such failures. A new test case has also been added to verify the handling of `SYNC` command failures during external table migrations, using a mock backend to simulate failures and checking for appropriate log messages. These changes enhance the reliability and robustness of the migration process, providing clearer error diagnosis and handling for potential `SYNC` command failures.
* Added initial version of `databricks labs ucx migrate-local-code` command ([1067](https://github.com/databrickslabs/ucx/issues/1067)). A new `databricks labs ucx migrate-local-code` command has been added to facilitate migration of local code to a Databricks environment, specifically targeting Python and SQL files. This initial version is experimental and aims to help users and administrators manage code migration, maintain consistency across workspaces, and enhance compatibility with the Unity Catalog, a component of Databricks' data and AI offerings. The command introduces a new `Files` class for applying migrations to code files, considering their language. It also updates the `.gitignore` file and the pyproject.toml file to ensure appropriate version control management. Additionally, new classes and methods have been added to support code analysis, transformation, and linting for various programming languages. These improvements will aid in streamlining the migration process and ensuring compatibility with Databricks' environment.
* Added instance pool to cluster policy ([1078](https://github.com/databrickslabs/ucx/issues/1078)). A new field, `instance_pool_id`, has been added to the cluster policy configuration in `policy.py`, allowing users to specify the ID of an instance pool to be applied to all workflow clusters in the policy. This ID can be manually set or automatically retrieved by the system. A new private method, `_get_instance_pool_id()`, has been added to handle the retrieval of the instance pool ID. Additionally, a new test for table migration jobs has been added to `test_installation.py` to ensure the migration job is correctly configured with the specified parallelism, minimum and maximum number of workers, and instance pool ID. A new test case for creating a cluster policy with an instance pool has also been added to `tests/unit/installer/test_policy.py` to ensure the instance pool is added to the cluster policy during creation. These changes provide users with more control over instance pools and cluster policies, and improve the overall functionality of the library.
* Fixed `ucx move` logic for `MANAGED` & `EXTERNAL` tables ([1062](https://github.com/databrickslabs/ucx/issues/1062)). The `ucx move` command has been updated to allow for the movement of UC tables/views after the table upgrade process, providing flexibility in managing catalog structure. The command now supports moving multiple tables simultaneously, dropping managed tables/views upon confirmation, and deep-cloning managed tables while dropping and recreating external tables. A refactoring of the `TableMove` class has improved code organization and readability, and the associated unit tests have been updated to reflect these changes. This feature is targeted towards developers and administrators seeking to adjust their catalog structure after table upgrades, with the added ability to manage exceptional conditions gracefully.
* Fixed integration testing with random product names ([1074](https://github.com/databrickslabs/ucx/issues/1074)). In the recent update, the `trigger` function in the `tasks.py` module of the `ucx` framework has undergone modification to incorporate a new argument, `install_folder`, within the `Installation` object. This object is now generated locally within the `trigger` function and subsequently passed to the `run_task` function. The `install_folder` is determined by obtaining the parent directory of the `config_path` variable, transforming it into a POSIX-style path, and eliminating the leading "/Workspace" prefix. This enhancement guarantees that the `run_task` function acquires the correct installation folder for the `ucx` framework, thereby improving the overall functionality and precision of the framework. Furthermore, the `Installation.current` method has been supplanted with the newly formed `Installation` object, which now encompasses the `install_folder` argument.
* Refactor installer to separate workflows methods from the installer class ([1055](https://github.com/databrickslabs/ucx/issues/1055)). In this release, the installer in the `cli.py` file has been refactored to improve modularity and maintainability. The installation and workflow functionalities have been separated by importing a new class called `WorkflowsInstallation` from `databricks.labs.ucx.installer.workflows`. The `WorkspaceInstallation` class is no longer used in various functions, and the new `WorkflowsInstallation` class is used instead. Additionally, a new mixin class called `InstallationMixin` has been introduced, which includes methods for uninstalling UCX, removing jobs, and validating installation steps. The `WorkflowsInstallation` class now inherits from this mixin class. A new file, `workflows.py`, has been added to the `databricks/labs/ucx/installer` directory, which contains methods for managing Databricks jobs. The new `WorkflowsInstallation` class is responsible for deploying workflows, uploading wheels to DBFS or WSFS, and creating debug notebooks. The refactoring also includes the addition of new methods for handling specific workflows, such as `run_workflow`, `validate_step`, and `repair_run`, which are now contained in the `WorkflowsInstallation` class. The `test_install.py` file in the `tests/unit` directory has also been updated to include new imports and test functions to accommodate these changes.
* Skip unsupported locations while migrating to external location in Azure ([1066](https://github.com/databrickslabs/ucx/issues/1066)). In this release, we have updated the functionality of migrating to an external location in Azure. A new private method `_filter_unsupported_location` has been added to the `locations.py` file, which checks if the location URLs are supported and removes the unsupported ones from the list. Only locations starting with "abfss://" are considered supported. Unsupported locations are logged with a warning message. Additionally, a new test `test_skip_unsupported_location` has been introduced to verify that the `location_migration` function correctly skips unsupported locations during migration to external locations in Azure. The test checks if the correct log messages are generated for skipped unsupported locations, and it mocks various scenarios such as crawled HMS external locations, storage credentials, UC external locations, and installation with permission mapping. The mock crawled HMS external locations contain two unsupported locations: `adl://` and `wasbs://`. This ensures that the function handles unsupported locations correctly, avoiding any unnecessary errors or exceptions during migration.
* Triggering Assessment Workflow from Installer based on User Prompt ([1007](https://github.com/databrickslabs/ucx/issues/1007)). A new functionality has been added to the installer that allows users to trigger an assessment workflow based on a prompt during the installation process. The `_trigger_workflow` method has been implemented, which can be initiated with a step string argument. This method retrieves the job ID for the specified step from the `_state.jobs` dictionary, generates the job URL, and triggers the job using the `run_now` method from the `jobs` class of the Workspace object. Users will be asked to confirm triggering the assessment workflow and will have the option to open the job URL in a web browser after triggering it. A new unit test, `test_triggering_assessment_wf`, has been introduced to the `test_install.py` file to verify the functionality of triggering an assessment workflow based on user prompt. This test uses existing classes and functions, such as `MockBackend`, `MockPrompts`, `WorkspaceConfig`, and `WorkspaceInstallation`, to run the `WorkspaceInstallation.run` method with a mocked `WorkspaceConfig` object and a mock installation. The test also includes a user prompt to confirm triggering the assessment job and opening the assessment job URL. The new functionality and test improve the installation process by enabling users to easily trigger the assessment workflow based on their specific needs.
* Updated README.md for Service Principal Installation Limit ([1076](https://github.com/databrickslabs/ucx/issues/1076)). This release includes an update to the README.md file to clarify that installing UCX with a Service Principal is not supported. Previously, the file indicated that Databricks Workspace Administrator privileges were required for the user running the installation, but did not explicitly state that Service Principal installation is not supported. The updated text now includes this information, ensuring that users have a clear understanding of the requirements and limitations of the installation process. The rest of the file remains unchanged and continues to provide instructions for installing UCX, including required software and network access. No new methods or functionality have been added, and no existing functionality has been changed beyond the addition of this clarification. The changes in this release have been manually tested to ensure they are functioning as intended.

* Added AWS IAM role support to `databricks labs ucx create-uber-principal` command ([993](https://github.com/databrickslabs/ucx/issues/993)). The `databricks labs ucx create-uber-principal` command now supports AWS Identity and Access Management (IAM) roles for external table migration. This new feature introduces a CLI command to create an `uber-IAM` profile, which checks for the UCX migration cluster policy and updates or adds the migration policy to provide access to the relevant table locations. If no IAM instance profile or role is specified in the cluster policy, a new one is created and the new migration policy is added. This change includes new methods and functions to handle AWS IAM roles, instance profiles, and related trust policies. Additionally, new unit and integration tests have been added and verified on the staging environment. The implementation also identifies all S3 buckets used by the Instance Profiles configured in the workspace.
* Added Dashboard widget to show the list of cluster policies along with DBR version ([1013](https://github.com/databrickslabs/ucx/issues/1013)). In this code revision, the `assessment` module of the 'databricks/labs/ucx' package has been updated to include a new `PoliciesCrawler` class, which fetches, assesses, and snapshots cluster policies. This class extends `CrawlerBase` and `CheckClusterMixin` and introduces the '_crawl', '_assess_policies', '_try_fetch', and `snapshot` methods. The `PolicyInfo` dataclass has been added to hold policy information, with a structure similar to the `ClusterInfo` dataclass. The `ClusterInfo` dataclass has been updated to include `spark_version` and `policy_id` attributes. A new table for policies has been added, and cluster policies along with the DBR version are loaded into this table. Relevant user documentation, tests, and a Dashboard widget have been added to support this feature. The `create` function in 'fixtures.py' has been updated to enable a Delta preview feature in Spark configurations, and a new SQL file has been included for querying cluster policies. Additionally, a new `crawl_cluster_policies` method has been added to scan and store cluster policies with matching configurations.
* Added `migration_status` table to capture a snapshot of migrated tables ([1041](https://github.com/databrickslabs/ucx/issues/1041)). A `migration_status` table has been added to track the status of migrated tables in the database, enabling improved management and tracking of migrations. The new `MigrationStatus` class, which is a dataclass that holds the source and destination schema, table, and updated timestamp, is added. The `TablesMigrate` class now has a new `_migration_status_refresher` attribute that is an instance of the new `MigrationStatusRefresher` class. This class crawls the `migration_status` table and returns a snapshot of the migration status, which is used to refresh the migration status and check if the table is upgraded. Additionally, the `_init_seen_tables` method is updated to get the seen tables from the `_migration_status_refresher` instead of fetching from the table properties. The `MigrationStatusRefresher` class fetches the migration status table and returns a snapshot of the migration status. This change also adds new test functions in the test file for the Hive metastore, which covers various scenarios such as migrating managed tables with and without caching, migrating external tables, and reverting migrated tables.
* Added a check for existing inventory database to avoid losing existing, inject installation objects in tests and try fetching existing installation before setting global as default ([1043](https://github.com/databrickslabs/ucx/issues/1043)). In this release, we have added a new method, `_check_inventory_database_exists`, to the `WorkspaceInstallation` class, which checks if an inventory database with a given name already exists in the Workspace. This prevents accidental overwriting of existing data and improves the robustness of handling inventory databases. The `validate_and_run` method has been updated to call `app.current_installation(workspace_client)`, allowing for a more flexible handling of installations. The `Installation` class import has been updated to include `SerdeError`, and the test suite has been updated to inject installation objects and check for existing installations before setting the global installation as default. A new argument `inventory_schema_suffix` has been added to the `factory` method for customization of the inventory schema name. We have also added a new method `check_inventory_database_exists` to the `WorkspaceInstaller` class, which checks if an inventory database already exists for a given installation type and raises an `AlreadyExists` error if it does. The behavior of the `download` method in the `WorkspaceClient` class has been mocked, and the `get_status` method has been updated to return `NotFound` in certain tests. These changes aim to improve the robustness, flexibility, and safety of the installation process in the Workspace.
* Added a check for external metastore in SQL warehouse configuration ([1046](https://github.com/databrickslabs/ucx/issues/1046)). In this release, we have added new functionality to the Unity Catalog (UCX) installation process to enable checking for and connecting to an external Hive metastore configuration. A new method, `_get_warehouse_config_with_external_hive_metastore`, has been introduced to retrieve the workspace warehouse config and identify if it is set up for an external Hive metastore. If so, and the user confirms the prompt, UCX will be configured to connect to the external metastore. Additionally, new methods `_extract_external_hive_metastore_sql_conf` and `test_cluster_policy_definition_<cloud_provider>_hms_warehouse()` have been added to handle the external metastore configuration for Azure, AWS, and GCP, and to handle the case when the data_access_config is empty. These changes provide more flexibility and ease of use when installing UCX with external Hive metastore configurations. The new imports `EndpointConfPair`, `GetWorkspaceWarehouseConfigResponse` from the `databricks.sdk.service.sql` package are used to handle the endpoint configuration of the SQL warehouse.
* Added integration tests for AWS - create locations ([1026](https://github.com/databrickslabs/ucx/issues/1026)). In this release, we have added comprehensive integration tests for AWS resources and their management in the `tests/unit/assessment/test_aws.py` file. The `AWSResources` class has been updated with new methods (AwsIamRole, add_uc_role, add_uc_role_policy, and validate_connection) and the regular expression for matching S3 resource ARN has been modified. The `create_external_locations` method now allows for creating external locations without validating them, and the `_identify_missing_external_locations` function has been enhanced to match roles with a wildcard pattern. The new tests include validating the integration of AWS services with the system, testing the CLI's behavior when it is missing, and introducing new configuration scenarios with the addition of a Key Management Service (KMS) key during the creation of IAM roles and policies. These changes improve the robustness and reliability of AWS resource integration and handling in our system.
* Bump Databricks SDK to v0.22.0 ([1059](https://github.com/databrickslabs/ucx/issues/1059)). In this release, we are bumping the Databricks SDK version to 0.22.0 and upgrading the `databricks-labs-lsql` package to ~0.2.2. The new dependencies for this release include `databricks-sdk==0.22.0`, `databricks-labs-lsql~=0.2.2`, `databricks-labs-blueprint~=0.4.3`, and `PyYAML>=6.0.0,<7.0.0`. In the `fixtures.py` file, we have added `PermissionLevel.CAN_QUERY` to the `CAN_VIEW` and `CAN_MANAGE` permissions in the `_path` function, allowing users to query the endpoint. Additionally, we have updated the `test_endpoints` function in the `test_generic.py` file as part of the integration tests for workspace access. This change updates the permission level for creating a serving endpoint from `CAN_MANAGE` to `CAN_QUERY`, meaning that the assigned group can now only query the endpoint. We have also included the `test_feature_tables` function in the commit, which tests the behavior of feature tables in the Databricks workspace. This change only affects the `test_endpoints` function and its assert statements, and does not impact the functionality of the `test_feature_tables` function.
* Changed default UCX installation folder to `/Applications/ucx` from `/Users/<me>/.ucx` to allow multiple users users utilising the same installation ([854](https://github.com/databrickslabs/ucx/issues/854)). In this release, we've added a new advanced feature that allows users to force the installation of UCX over an existing installation using the `UCX_FORCE_INSTALL` environment variable. This variable can take two values `global` and 'user', providing more control and flexibility in installing UCX. The default UCX installation folder has been changed to /Applications/ucx from /Users/<me>/.ucx to enable multiple users to utilize the same installation. A table detailing the expected install location, `install_folder`, and mode for each combination of global and user values has been added to the README file. We've also added user prompts to confirm the installation if UCX is already installed and the `UCX_FORCE_INSTALL` variable is set to 'user'. This feature is useful when users want to install UCX in a specific location or force the installation over an existing installation. However, it is recommended to use this feature with caution, as it can potentially break existing installations if not used correctly. Additionally, several changes to the implementation of the UCX installation process have been made, as well as new tests to ensure that the installation process works correctly in various scenarios.
* Fix: Recover lost fix for `webbrowser.open` mock ([1052](https://github.com/databrickslabs/ucx/issues/1052)). A fix has been implemented to address an issue related to the mock for `webbrowser.open` in the tests `test_repair_run` and `test_get_existing_installation_global`. This change prevents the `webbrowser.open` function from being called during these tests, which helps improve test stability and consistency. No new methods have been added, and the existing functionality of these tests has only been modified to include the `webbrowser.open` mock. This modification aims to enhance the reliability and predictability of these specific tests, ensuring accurate and consistent results.
* Improved table migrations logic ([1050](https://github.com/databrickslabs/ucx/issues/1050)). This change introduces improvements to table migrations logic by refactoring unit tests to load table mappings from JSON instead of inline structs, adding an `escape_sql_identifier` function where missing, and preparing for ACLs migration. The `uc_grant_sql` method in `grants.py` has been updated to accept optional `object_type` and `object_key` parameters, and the hive-to-UC mapping has been expanded to include mappings for views. Additionally, new JSON files for external source table configuration have been added, and new functions have been introduced for loading fixture data from JSON files and creating mocked `WorkspaceClient` and `TableMapping` objects for testing. The changes improve the maintainability and security of the codebase, prepare it for future migration tasks, and ensure that the code is more adaptable and robust. The changes have been manually tested and verified on the staging environment.
* Moved `SqlBackend` implementation to `databricks-labs-lsql` dependency ([1042](https://github.com/databrickslabs/ucx/issues/1042)). In this change, the `SqlBackend` implementation, including classes such as `StatementExecutionBackend` and `RuntimeBackend`, has been moved to a separate library, `databricks-labs-lsql`, which is managed at <https://github.com/databrickslabs/lsql>. This refactoring simplifies the current repository, promotes code reuse, and improves modularity by leveraging an external dependency. The modification includes adding a new line in the .gitignore file to exclude `*.out` files from version control.
* Prepare for a PyPI release ([1038](https://github.com/databrickslabs/ucx/issues/1038)). In preparation for a PyPI release, this change introduces a new GitHub Actions workflow that automates the package release process and ensures the integrity of the released packages by signing them with Sigstore. When a new git tag starting with `v` is pushed, this workflow is triggered, building wheels using hatch, drafting a new GitHub release, publishing the package distributions to PyPI, and signing the artifacts with Sigstore. The `pyproject.toml` file is now used for metadata, replacing `setup.cfg` and `setup.py`, and is cached to improve build performance. In addition, the `pyproject.toml` file has been updated with recent metadata in preparation for the release, including updates to the package's authors, development status, classifiers, and dependencies.
* Prevent fragile `mock.patch('databricks...')` in the test code ([1037](https://github.com/databrickslabs/ucx/issues/1037)). This change introduces a custom `pylint` checker to improve code flexibility and maintainability by preventing fragile `mock.patch` designs in test code. The new checker discourages the use of `MagicMock` and encourages the use of `create_autospec` to ensure that mocks have the same attributes and methods as the original class. This change has been implemented in multiple test files, including `test_cli.py`, `test_locations.py`, `test_mapping.py`, `test_table_migrate.py`, `test_table_move.py`, `test_workspace_access.py`, `test_redash.py`, `test_scim.py`, and `test_verification.py`, to improve the robustness and maintainability of the test code. Additionally, the commit removes the `verification.py` file, which contained a `VerificationManager` class for verifying applied permissions, scope ACLs, roles, and entitlements for various objects in a Databricks workspace.
* Removed `mocker.patch("databricks...)` from `test_cli` ([1047](https://github.com/databrickslabs/ucx/issues/1047)). In this release, we have made significant updates to the library's handling of Azure and AWS workspaces. We have added new parameters `azure_resource_permissions` and `aws_permissions` to the `_execute_for_cloud` function in `cli.py`, which are passed to the `func_azure` and `func_aws` functions respectively. The `create_uber_principal` and `principal_prefix_access` commands have also been updated to include these new parameters. Additionally, the `_azure_setup_uber_principal` and `_aws_setup_uber_principal` functions have been updated to accept the new `azure_resource_permissions` and `aws_resource_permissions` parameters. The `_azure_principal_prefix_access` and `_aws_principal_prefix_access` functions have also been updated similarly. We have also introduced a new `aws_resources` parameter in the `migrate_credentials` command, which is used to migrate Azure Service Principals in ADLS Gen2 locations to UC storage credentials. In terms of testing, we have replaced the `mocker.patch` calls with the creation of `AzureResourcePermissions` and `AWSResourcePermissions` objects, improving the code's readability and maintainability. Overall, these changes significantly enhance the library's functionality and maintainability in handling Azure and AWS workspaces.
* Require Hatch v1.9.4 on build machines ([1049](https://github.com/databrickslabs/ucx/issues/1049)). In this release, we have updated the Hatch package version to 1.9.4 on build machines, addressing issue [#1049](https://github.com/databrickslabs/ucx/issues/1049). The changes include updating the toolchain dependencies and setup in the `.codegen.json` file, which simplifies the setup process and now relies on a pre-existing Hatch environment and Python 3. The acceptance workflow has also been updated to use the latest version of Hatch and the `databrickslabs/sandbox/acceptance` GitHub action version `v0.1.4`. Hatch is a Python package manager that simplifies package development and management, and this update provides new features and bug fixes that can help improve the reliability and performance of the acceptance workflow. This change requires version 1.9.4 of the Hatch package on build machines, and it will affect the build process for the project but will not have any impact on the functionality of the project itself. As a software engineer adopting this project, it's important to note this change to ensure that the build process runs smoothly and takes advantage of any new features or improvements in Hatch 1.9.4.
* Set acceptance tests to timeout after 45 minutes ([1036](https://github.com/databrickslabs/ucx/issues/1036)). As part of issue [#1036](https://github.com/databrickslabs/ucx/issues/1036), the acceptance tests in this open-source library now have a 45-minute timeout configured, improving the reliability and stability of the testing environment. This change has been implemented in the `.github/workflows/acceptance.yml` file by adding the `timeout` parameter to the step where the `databrickslabs/sandbox/acceptance` action is called. This ensures that the acceptance tests will not run indefinitely and prevents any potential issues caused by long-running tests. By adopting this project, software engineers can now benefit from a more stable and reliable testing environment, with acceptance tests that are guaranteed to complete within a maximum of 45 minutes.
* Updated databricks-labs-blueprint requirement from ~0.4.1 to ~0.4.3 ([1058](https://github.com/databrickslabs/ucx/issues/1058)). In this release, the version requirement for the `databricks-labs-blueprint` library has been updated from ~0.4.1 to ~0.4.3 in the pyproject.toml file. This change is necessary to support issues [#1056](https://github.com/databrickslabs/ucx/issues/1056) and [#1057](https://github.com/databrickslabs/ucx/issues/1057). The code has been manually tested and is ready for further testing to ensure the compatibility and smooth functioning of the software. It is essential to thoroughly test the latest version of the `databricks-labs-blueprint` library with the existing codebase before deploying it to production. This includes running a comprehensive suite of tests such as unit tests, integration tests, and verification on the staging environment. This modification allows the software to use the latest version of the library, improving its functionality and overall performance.
* Use `MockPrompts.extend()` functionality in test_install to supply multiple prompts ([1057](https://github.com/databrickslabs/ucx/issues/1057)). This diff introduces the `MockPrompts.extend()` functionality in the `test_install` module to enable the supplying of multiple prompts for testing purposes. A new `base_prompts` dictionary with default prompts has been added and is extended with additional prompts for specific test cases. This allows for the testing of various scenarios, such as when UCX is already installed on the workspace and the user is prompted to choose between global or user installation. Additionally, new `force_user_environ` and `force_global_env` dictionaries have been added to simulate different installation environments. The functionality of the `WorkspaceInstaller` class and mocking of `webbrowser.open` are also utilized in the test cases. These changes aim to ensure the proper functioning of the configuration process for different installation scenarios.

* Added AWS IAM roles support to `databricks labs ucx migrate-credentials` command ([973](https://github.com/databrickslabs/ucx/issues/973)). This commit adds AWS Identity and Access Management (IAM) roles support to the `databricks labs ucx migrate-credentials` command, resolving issue [#862](https://github.com/databrickslabs/ucx/issues/862) and being related to pull request [#874](https://github.com/databrickslabs/ucx/issues/874). It includes the addition of a `load` function to `AWSResourcePermissions` to return identified instance profiles and the creation of an `IamRoleMigration` class under `aws/credentials.py` to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command `databricks labs ucx migrate-credentials` have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as `add_uc_role_policy` and `update_uc_trust_role`, among others, designed to facilitate the migration process for AWS IAM roles.
* Added `create-catalogs-schemas` command to prepare destination catalogs and schemas before table migration ([1028](https://github.com/databrickslabs/ucx/issues/1028)). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new `create-catalogs-schemas` command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the `create-table-mapping` command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new `CatalogSchema` class has been added to the `hive_metastore` package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the `tests/unit/hive_metastore` directory to verify the behavior of the `CatalogSchema` class and the new `create-catalogs-schemas` command. This command is intended for use in contexts where GCP is not supported.
* Added automated upgrade option to set up cluster policy ([1024](https://github.com/databrickslabs/ucx/issues/1024)). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, `ClusterPolicyInstaller`, is added to the `policy.py` file in the `installer` package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project.
* Added crawling for init scripts on local files to assessment workflow ([960](https://github.com/databrickslabs/ucx/issues/960)). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue [#9](https://github.com/databrickslabs/ucx/issues/9)
* Added database filter for the `assessment` workflow ([989](https://github.com/databrickslabs/ucx/issues/989)). In this release, we have added a new configuration option, `include_databases`, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the `TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the associated functions such as `_all_databases`, `getIncludeDatabases`, `_select_databases`. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment.
* Estimate migration effort based on assessment database ([1008](https://github.com/databrickslabs/ucx/issues/1008)). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new `query` parameter has been added to the `SimpleQuery` class to support this feature. Additional changes include the update of the `_install_viz` and `_install_query` methods, the inclusion of the `data_source_id` in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, `mock_installation_with_jobs`, has been added to support testing of the assessment estimates dashboard.
* Explicitly write to `hive_metastore` from `crawl_tables` task ([1021](https://github.com/databrickslabs/ucx/issues/1021)). In this release, we have improved the clarity and specificity of our handling of the `hive_metastore` in the `crawl_tables` task. Previously, the `df.write.saveAsTable` method was used without explicitly specifying the `hive_metastore` database, which could result in ambiguity. To address this issue, we have updated the `saveAsTable` method to include the `hive_metastore` database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and affect the `crawl_tables` task. While no new methods have been added, the existing `saveAsTable` method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore.
* Improved documentation for `databricks labs ucx move` command ([1025](https://github.com/databrickslabs/ucx/issues/1025)). The `databricks labs ucx move` command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security.
* Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 ([1030](https://github.com/databrickslabs/ucx/issues/1030)). In this update, the `databricks-sdk` package requirement has been updated to version `~=0.21.0` from `~=0.20.0`. This new version addresses several bugs and provides enhancements, including the fix for the `get_workspace_client` method in GCP, the use of the `all-apis` scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the `databricks-labs-blueprint` and `PyYAML` dependencies in this commit.

* Added AWS S3 support for `migrate-locations` command ([1009](https://github.com/databrickslabs/ucx/issues/1009)). In this release, the open-source library has been enhanced with AWS S3 support for the `migrate-locations` command, enabling efficient and secure management of S3 data. The new functionality includes the identification of missing S3 prefixes and the creation of corresponding roles and policies through the addition of methods `_identify_missing_paths`, `_get_existing_credentials_dict`, and `create_external_locations`. The library now also includes new classes `AwsIamRole`, `ExternalLocationInfo`, and `StorageCredentialInfo` for better handling of AWS-related functionality. Additionally, two new tests, `test_create_external_locations` and `test_create_external_locations_skip_existing`, have been added to ensure the correct behavior of the new AWS-related functionality. The new test function `test_migrate_locations_aws` checks the AWS-specific implementation of the `migrate-locations` command, while `test_missing_aws_cli` verifies the correct error message is displayed when the AWS CLI is not found in the system path. These changes enhance the library's capabilities, improving data security, privacy, and overall performance for users working with AWS S3.
* Added `databricks labs ucx create-uber-principal` command to create Azure Service Principal for migration ([976](https://github.com/databrickslabs/ucx/issues/976)). The new CLI command, `databricks labs ucx create-uber-principal`, has been introduced to create an Azure Service Principal (SPN) and grant it STORAGE BLOB READER access on all the storage accounts used by the tables in the workspace. The SPN information is then stored in the UCX cluster policy. A new class, AzureApiClient, has been added to isolate Azure API calls, and unit and integration tests have been included to verify the functionality. This development enhances migration capabilities for Azure workspaces, providing a more streamlined and automated way to create and manage Service Principals, and improves the functionality and usability of the UCX tool. The changes are well-documented and follow the project's coding standards.
* Added `migrate-locations` command ([1016](https://github.com/databrickslabs/ucx/issues/1016)). In this release, we've added a new CLI command, `migrate_locations`, to create Unity Catalog (UC) external locations. This command extracts candidates for location creation from the `guess_external_locations` assessment task and checks if corresponding UC Storage Credentials exist before creating the locations. Currently, the command only supports Azure, with plans to add support for AWS and GCP in the future. The `migrate_locations` function is marked with the `ucx.command` decorator and is available as a command-line interface (CLI) command. The pull request also includes unit tests for this new command, which check the environment (Azure, AWS, or GCP) before executing the migration and log a message if the environment is AWS or GCP, indicating that the migration is not yet supported on those platforms. No changes have been made to existing workflows, commands, or tables.
* Added handling for widget delete on upgrade platform bug ([1011](https://github.com/databrickslabs/ucx/issues/1011)). In this release, the `_install_dashboard` method in `dashboards.py` has been updated to handle a platform bug that occurred during the deletion of dashboard widgets during an upgrade process (issue [#1011](https://github.com/databrickslabs/ucx/issues/1011)). Previously, the method attempted to delete each widget using the `self._ws.dashboard_widgets.delete(widget.id)` command, which resulted in a `TypeError` when attempting to delete a widget. The updated method now includes a try/except block that catches this `TypeError` and logs a warning message, while also tracking the issue under bug ES-1061370. The rest of the method remains unchanged, creating a dashboard with the given name, role, and parent folder ID if no widgets are present. This enhancement improves the robustness of the `_install_dashboard` method by adding error handling for the SDK API response when deleting dashboard widgets, ensuring a smoother upgrade process.
* Create UC external locations in Azure based on migrated storage credentials ([992](https://github.com/databrickslabs/ucx/issues/992)). The `locations.py` file in the `databricks.labs.ucx.azure` package has been updated to include a new class `ExternalLocationsMigration`, which creates UC external locations in Azure based on migrated storage credentials. This class takes various arguments, including `WorkspaceClient`, `HiveMetastoreLocations`, `AzureResourcePermissions`, and `AzureResources`. It has a `run()` method that lists any missing external locations in UC, extracts their location URLs, and attempts to create a UC external location with a mapped storage credential name if the missing external location is in the mapping. The class also includes helper methods for generating credential name mappings. Additionally, the `resources.py` file in the same package has been modified to include a new method `managed_identity_client_id`, which retrieves the client ID of a managed identity associated with a given access connector. Test functions for the `ExternalLocationsMigration` class and Azure external locations functionality have been added in the new file `test_locations.py`. The `test_resources.py` file has been updated to include tests for the `managed_identity_client_id` method. A new `mappings.json` file has also been added for tests related to Azure external location mappings based on migrated storage credentials.
* Deprecate legacy installer ([1014](https://github.com/databrickslabs/ucx/issues/1014)). In this release, we have deprecated the legacy installer for the UCX project, which was previously implemented as a bash script. A warning message has been added to inform users about the deprecation and direct them to the UCX installation instructions. The functionality of the script remains unchanged, and it still performs tasks such as installing Python dependencies and building Python bindings. The script will eventually be replaced with the `databricks labs install ucx` command. This change is part of issue [#1014](https://github.com/databrickslabs/ucx/issues/1014) and is intended to streamline the installation process and improve the overall user experience. We recommend that users update their installation process to the new recommended method as soon as possible to avoid any issues with the legacy installer in the future.
* Prompt user if Terraform utilised for deploying infrastructure ([1004](https://github.com/databrickslabs/ucx/issues/1004)). In this update, the `config.py` file has been modified to include a new attribute, `is_terraform_used`, in the `WorkspaceConfig` class. This boolean flag indicates whether Terraform has been used for deploying certain entities in the workspace. Issue [#393](https://github.com/databrickslabs/ucx/issues/393) has been addressed with this change. The `WorkspaceInstaller` configuration has also been updated to take advantage of this new attribute, allowing developers to determine if Terraform was used for infrastructure deployment, thereby increasing visibility into the deployment process. Additionally, a new prompt has been added to the `warehouse_type` function to ascertain if Terraform is being utilized for infrastructure deployment, setting the `is_terraform_used` variable to True if it is. This improvement is intended for software engineers adopting this open-source library.
* Updated CONTRIBUTING.md ([1005](https://github.com/databrickslabs/ucx/issues/1005)). In this contribution to the open-source library, the CONTRIBUTING.md file has been significantly updated with clearer instructions on how to effectively contibute to the project. The previous command to print the Python path has been removed, as the IDE is now advised to be configured to use the Python interpreter from the virtual environment. A new step has been added, recommending the use of a consistent styleguide and formatting of the code before every commit. Moreover, it is now encouraged to run tests before committing to minimize potential issues during the review process. The steps on how to make a Fork from the ucx repo and create a PR have been updated with links to official documentation. Lastly, the commit now includes information on handling dependency errors that may occur after `git pull`.
* Updated databricks-labs-blueprint requirement from ~=0.2.4 to ~=0.3.0 ([1001](https://github.com/databrickslabs/ucx/issues/1001)). In this pull request update, the requirements file, pyproject.toml, has been modified to upgrade the databricks-labs-blueprint package from version ~0.2.4 to ~0.3.0. This update integrates the latest features and bug fixes of the package, including an automated upgrade framework, a brute-forcing approach for handling SerdeError, and enhancements for running nightly integration tests with service principals. These improvements increase the testability and functionality of the software, ensuring its stable operation with service principals during nightly integration tests. Furthermore, the reliability of the test for detecting existing installations has been reinforced by adding a new test function that checks for the correct detection of existing installations and retries the test for up to 15 seconds if they are not.

Dependency updates:

* Updated databricks-labs-blueprint requirement from ~=0.2.4 to ~=0.3.0 ([1001](https://github.com/databrickslabs/ucx/pull/1001)).