Databricks-labs-ucx

* Added a logo for UCX ([605](https://github.com/databrickslabs/ucx/pull/605)).
* Check if the `hatch` is already installed, and install only if it isn't installed yet ([603](https://github.com/databrickslabs/ucx/pull/603)).
* Fixed installation check for git pre-release versions ([600](https://github.com/databrickslabs/ucx/pull/600)).
* Temporarily remove SQL warehouse requirement from `labs.yml` ([604](https://github.com/databrickslabs/ucx/pull/604)).

**Breaking changes** (existing installations need to reinstall UCX and re-run assessment jobs)

* Switched local group migration component to rename groups instead of creating backup groups ([450](https://github.com/databrickslabs/ucx/pull/450)).
* Mitigate permissions loss in Table ACLs by folding grants belonging to the same principal, object id and object type together ([512](https://github.com/databrickslabs/ucx/pull/512)).

**New features**

* Added support for the experimental Databricks CLI launcher ([517](https://github.com/databrickslabs/ucx/pull/517)).
* Added support for external Hive Metastores including AWS Glue ([400](https://github.com/databrickslabs/ucx/pull/400)).
* Added more views to assessment dashboard ([474](https://github.com/databrickslabs/ucx/pull/474)).
* Added rate limit for creating backup group to increase stability ([500](https://github.com/databrickslabs/ucx/pull/500)).
* Added deduplication for mount point list ([569](https://github.com/databrickslabs/ucx/pull/569)).
* Added documentation to describe interaction with external Hive Metastores ([473](https://github.com/databrickslabs/ucx/pull/473)).
* Added failure injection for job failure message propagation ([591](https://github.com/databrickslabs/ucx/pull/591)).
* Added uniqueness in the new warehouse name to avoid conflicts on installation ([542](https://github.com/databrickslabs/ucx/pull/542)).
* Added a global init script to collect Hive Metastore lineage ([513](https://github.com/databrickslabs/ucx/pull/513)).
* Added retry set/update permissions when possible and assess the changes in the workspace ([519](https://github.com/databrickslabs/ucx/pull/519)).
* Use `~/.ucx/state.json` to store the state of both dashboards and jobs ([561](https://github.com/databrickslabs/ucx/pull/561)).

**Bug fixes**

* Fixed handling for `OWN` table permissions ([571](https://github.com/databrickslabs/ucx/pull/571)).
* Fixed handling of keys with and without values. ([514](https://github.com/databrickslabs/ucx/pull/514)).
* Fixed integration test failures related to concurrent group delete ([584](https://github.com/databrickslabs/ucx/pull/584)).
* Fixed issue with workspace listing process on None type `object_type` ([481](https://github.com/databrickslabs/ucx/pull/481)).
* Fixed missing group entitlement migration bug ([583](https://github.com/databrickslabs/ucx/pull/583)).
* Fixed entitlement application for account-level groups ([529](https://github.com/databrickslabs/ucx/pull/529)).
* Fixed assessment throwing an error when the owner of an object is empty ([485](https://github.com/databrickslabs/ucx/pull/485)).
* Fixed installer to migrate between different configuration file versions ([596](https://github.com/databrickslabs/ucx/pull/596)).
* Fixed cluster policy crawler to be aware of deleted policies ([486](https://github.com/databrickslabs/ucx/pull/486)).
* Improved error message for not null constraints violated ([532](https://github.com/databrickslabs/ucx/pull/532)).
* Improved integration test resiliency ([597](https://github.com/databrickslabs/ucx/pull/597), [#594](https://github.com/databrickslabs/ucx/pull/594), [#586](https://github.com/databrickslabs/ucx/pull/586)).
* Introduced Safer access to workspace objects' properties. ([530](https://github.com/databrickslabs/ucx/pull/530)).
* Mitigated permissions loss in Table ACLs by running appliers with single thread ([518](https://github.com/databrickslabs/ucx/pull/518)).
* Running apply permission task before assessment should display message ([487](https://github.com/databrickslabs/ucx/pull/487)).
* Split integration tests from blocking the merge queue ([496](https://github.com/databrickslabs/ucx/pull/496)).
* Support more than one dashboard per step ([472](https://github.com/databrickslabs/ucx/pull/472)).
* Update databricks-sdk requirement from ~=0.11.0 to ~=0.12.0 ([505](https://github.com/databrickslabs/ucx/pull/505)).
* Update databricks-sdk requirement from ~=0.12.0 to ~=0.13.0 ([575](https://github.com/databrickslabs/ucx/pull/575)).

* Added `make install-dev` and a stronger `make clean` for easier dev on-boarding and release upgrades ([458](https://github.com/databrickslabs/ucx/pull/458)).
* Added failure summary in the assessment dashboard ([455](https://github.com/databrickslabs/ucx/pull/455)).
* Added test for checking grants in default schema ([470](https://github.com/databrickslabs/ucx/pull/470)).
* Added unit tests for generic permissions ([457](https://github.com/databrickslabs/ucx/pull/457)).
* Enabled integration tests via OIDC for every pull request ([378](https://github.com/databrickslabs/ucx/pull/378)).
* Added check if permissions are up to date ([421](https://github.com/databrickslabs/ucx/pull/421)).
* Fixed casing in `all_tables.sql` query. ([464](https://github.com/databrickslabs/ucx/pull/464)).
* Fixed missed scans for empty databases and views in `crawl_grants` ([469](https://github.com/databrickslabs/ucx/pull/469)).
* Improved logging colors for dark terminal backgrounds ([468](https://github.com/databrickslabs/ucx/pull/468)).
* Improved local group migration state handling and made log files flush every 10 minutes ([449](https://github.com/databrickslabs/ucx/pull/449)).
* Moved workspace listing as a separate task for an assessment workflow ([437](https://github.com/databrickslabs/ucx/pull/437)).
* Removed rate limit for get or create backup group to speed up the prepare environment ([453](https://github.com/databrickslabs/ucx/pull/453)).
* Updated databricks-sdk requirement from ~=0.10.0 to ~=0.11.0 ([448](https://github.com/databrickslabs/ucx/pull/448)).

* Added exception handling for secret scope not found. ([418](https://github.com/databrickslabs/ucx/pull/418)).
* Added a crawler for creating an inventory of Azure Service Principals ([326](https://github.com/databrickslabs/ucx/pull/326)).
* Added check if account group already exists during failure recovery ([446](https://github.com/databrickslabs/ucx/pull/446)).
* Added checking for index out of range. ([429](https://github.com/databrickslabs/ucx/pull/429)).
* Added hyperlink to UCX releases in the main readme ([408](https://github.com/databrickslabs/ucx/pull/408)).
* Added integration test to check backup groups get deleted ([387](https://github.com/databrickslabs/ucx/pull/387)).
* Added logging of errors during threadpool operations. ([376](https://github.com/databrickslabs/ucx/pull/376)).
* Added recovery mode for workspace-local groups from temporary groups ([435](https://github.com/databrickslabs/ucx/pull/435)).
* Added support for migrating Legacy Table ACLs from workspace-local to account-level groups ([412](https://github.com/databrickslabs/ucx/pull/412)).
* Added detection for installations of unreleased versions ([399](https://github.com/databrickslabs/ucx/pull/399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit` ([407](https://github.com/databrickslabs/ucx/pull/407)).
* Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI ([426](https://github.com/databrickslabs/ucx/pull/426)).
* Ensured that table exists, even when crawlers produce zero records ([373](https://github.com/databrickslabs/ucx/pull/373)).
* Extended test suite for HMS->HMS TACL migration ([439](https://github.com/databrickslabs/ucx/pull/439)).
* Fixed handling of secret scope responses ([431](https://github.com/databrickslabs/ucx/pull/431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path' config ([444](https://github.com/databrickslabs/ucx/pull/444)).
* Fixed broken logic in `parallel` module and applied hardened error handling design for parallel code ([405](https://github.com/databrickslabs/ucx/pull/405)).
* Fixed codecov.io reporting ([403](https://github.com/databrickslabs/ucx/pull/403)).
* Fixed integration tests for crawlers ([379](https://github.com/databrickslabs/ucx/pull/379)).
* Improved README.py and logging messages ([433](https://github.com/databrickslabs/ucx/pull/433)).
* Improved cleanup for workspace backup groups by adding more retries on errors ([375](https://github.com/databrickslabs/ucx/pull/375)).
* Improved dashboard queries to show unsupported storage types. ([398](https://github.com/databrickslabs/ucx/pull/398)).
* Improved documentation for readme notebook ([257](https://github.com/databrickslabs/ucx/pull/257)).
* Improved test coverage for installer ([371](https://github.com/databrickslabs/ucx/pull/371)).
* Introduced deterministic `env_or_skip` fixture for integration tests ([396](https://github.com/databrickslabs/ucx/pull/396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and `TableInfo` ([409](https://github.com/databrickslabs/ucx/pull/409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier` interfaces to `workspace_access.AclSupport` ([436](https://github.com/databrickslabs/ucx/pull/436)).
* Moved examples to docs ([404](https://github.com/databrickslabs/ucx/pull/404)).
* Properly isolated integration testing for workflows on an existing shared cluster ([414](https://github.com/databrickslabs/ucx/pull/414)).
* Removed thread pool for any IAM Group removals and additions ([394](https://github.com/databrickslabs/ucx/pull/394)).
* Replace plus char with minus in version tag for GCP dev installation of UCX ([420](https://github.com/databrickslabs/ucx/pull/420)).
* Run integration tests on shared clusters for a faster devloop ([397](https://github.com/databrickslabs/ucx/pull/397)).
* Show difference between serverless and PRO warehouses during installation ([385](https://github.com/databrickslabs/ucx/pull/385)).
* Split `migrate-groups` workflow into three different stages for reliability ([442](https://github.com/databrickslabs/ucx/pull/442)).
* Use groups instead of usernames in code owners file ([389](https://github.com/databrickslabs/ucx/pull/389)).

* Added `inventory_database` name check during installation ([275](https://github.com/databrickslabs/ucx/pull/275)).
* Added a column to `$inventory.tables` to specify if a table might have been synchronised to Unity Catalog already or not ([306](https://github.com/databrickslabs/ucx/pull/306)).
* Added a migration state to skip already migrated tables ([325](https://github.com/databrickslabs/ucx/pull/325)).
* Fixed appending to tables by adding filtering of `None` rows ([356](https://github.com/databrickslabs/ucx/pull/356)).
* Fixed handling of missing but linked cluster policies. ([361](https://github.com/databrickslabs/ucx/pull/361)).
* Ignore errors for Redash widgets and queries redeployment during installation ([367](https://github.com/databrickslabs/ucx/pull/367)).
* Remove exception and added proper logging for groups in the list that… ([357](https://github.com/databrickslabs/ucx/pull/357)).
* Skip group migration when no groups are available after preparation step. ([363](https://github.com/databrickslabs/ucx/pull/363)).
* Update databricks-sdk requirement from ~=0.9.0 to ~=0.10.0 ([362](https://github.com/databrickslabs/ucx/pull/362)).

* Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([277](https://github.com/databrickslabs/ucx/pull/277)).
* Added crawler for Azure Service principals used for direct storage access ([305](https://github.com/databrickslabs/ucx/pull/305)).
* Added more SQL queries to the assessment step dashboard ([269](https://github.com/databrickslabs/ucx/pull/269)).
* Added filtering out for job clusters in the clusters crawler ([298](https://github.com/databrickslabs/ucx/pull/298)).
* Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([300](https://github.com/databrickslabs/ucx/pull/300)).
* Added comprehensive introduction user manual ([273](https://github.com/databrickslabs/ucx/pull/273)).
* Added interactive tutorial for local group migration readme ([291](https://github.com/databrickslabs/ucx/pull/291)).
* Added tutorial links to the landing page of documentation ([290](https://github.com/databrickslabs/ucx/pull/290)).
* Added (internal) support for account-level configuration and multi-cloud workspace list ([264](https://github.com/databrickslabs/ucx/pull/264)).
* Improved order of tasks in the README notebook ([286](https://github.com/databrickslabs/ucx/pull/286)).
* Improved installation script to run in a Windows Git Bash terminal ([282](https://github.com/databrickslabs/ucx/pull/282)).
* Improved installation script by setting log level to uppercase by default ([271](https://github.com/databrickslabs/ucx/pull/271)).
* Improved installation finish messages within installer script ([267](https://github.com/databrickslabs/ucx/pull/267)).
* Improved automation for `MANAGED` table migration and continued building tables migration component ([295](https://github.com/databrickslabs/ucx/pull/295)).
* Fixed debug notebook code with refactored package structure ([250](https://github.com/databrickslabs/ucx/pull/250)) ([#265](https://github.com/databrickslabs/ucx/pull/265)).
* Fixed replacement of custom configured database to replicate in the report for external locations ([296](https://github.com/databrickslabs/ucx/pull/296)).
* Removed redundant `notebooks` top-level folder ([263](https://github.com/databrickslabs/ucx/pull/263)).
* Split checking for test failures and linting errors into independent GitHub Actions checks ([287](https://github.com/databrickslabs/ucx/pull/287)).
* Verify query metadata for assessment dashboards during unit tests ([294](https://github.com/databrickslabs/ucx/pull/294)).