Metricflow

Features

- Add dbt-duckdb as a supported adapter and remove legacy DuckDB sql client ([583](https://github.com/dbt-labs/metricflow/issues/583))

Fixes

- Remove barrier to partition pruning certain time partition filter predicates ([712](https://github.com/dbt-labs/metricflow/issues/712))

Under the Hood

- Make duckdb the standard for all dev-env environment runs, including `make test` ([723](https://github.com/dbt-labs/metricflow/issues/723))
- ([728](https://github.com/dbt-labs/metricflow/issues/728))

Dependencies

- Update pandas to 1.5.x ([719](https://github.com/dbt-labs/metricflow/pull/719))
- Relax version pins for MetricFlow dependencies ([720](https://github.com/dbt-labs/metricflow/pull/720))

Contributors
- [courtneyholcomb](https://github.com/courtneyholcomb) ([#728](https://github.com/dbt-labs/metricflow/issues/728))
- [jwills](https://github.com/jwills) ([#583](https://github.com/dbt-labs/metricflow/issues/583))
- [tlento](https://github.com/tlento) ([#712](https://github.com/dbt-labs/metricflow/issues/712), [#723](https://github.com/dbt-labs/metricflow/issues/723), [#719](https://github.com/dbt-labs/metricflow/pull/719), [#720](https://github.com/dbt-labs/metricflow/pull/720))

Breaking Changes

- License Change - Version 0 to 0.140.0 was covered by the Affero GPL license. Version 0.150.0 and greater is covered by the BSL license. ([465](https://github.com/dbt-labs/metricflow/issues/465))
- Removing time_format from DimensionTypeParams ([494](https://github.com/dbt-labs/metricflow/issues/494))
- Use Templates For Defining Metric Filters ([505](https://github.com/dbt-labs/metricflow/issues/505))
- Rename Metric.constraint to Metric.filter ([511](https://github.com/dbt-labs/metricflow/issues/511))
- Deprecate and refactor CLI commands
- Removes async query and query cancel methods from SqlClient protocols ([577](https://github.com/dbt-labs/metricflow/issues/577))
- Remove time spine introspection and table creation, which may break cumulative metric queries ([592](https://github.com/dbt-labs/metricflow/issues/592))
- Remove SqlEngineAttributes construct from SqlClient interface in favor of dialect rendering and engine type properties ([577](https://github.com/dbt-labs/metricflow/issues/577))
- CLI needs to be ran in a dbt project root directory
- Remove expr & ratio metrics and bundle with derived metrics. ([504](https://github.com/dbt-labs/metricflow/issues/504))
- `explain_get_dimension_values` & `get_dimension_values` take a list of metrics parameters
- Remove MetricFlow config file - all future configuration must originate with the dbt project ([624](https://github.com/dbt-labs/metricflow/issues/624))
- Update to dbt-semantic-interfaces==0.1.0.dev8. ([634](https://github.com/dbt-labs/metricflow/issues/634))
- Changed the --group-bys option in mf query to be --group-by
- Add Support for `primary_entity` in Semantic Models ([694](https://github.com/dbt-labs/metricflow/issues/694))

Features

- Added new entity calls to CLI/MetricFlowEngine
- Script to Generate Snapshots for Supported SQL Engines. ([609](https://github.com/dbt-labs/metricflow/issues/609))
- Add dbt adapter support for postgres and enable it for tests ([578](https://github.com/dbt-labs/metricflow/issues/578))
- Use dbt adapter to run queries and warehouse validations from MetricFlow CLI ([624](https://github.com/dbt-labs/metricflow/issues/624))
- Enable Snowflake queries in dbt <-> MetricFlow CLI integration ([579](https://github.com/dbt-labs/metricflow/issues/579))
- Refactor `mf tutorial` to work alongside a dbt project.
- New package `dbt-metricflow` which bundles dbt-core and metricflow and dbt-adapters
- Add Support for Python 3.10 / 3.11 ([659](https://github.com/dbt-labs/metricflow/issues/659))
- Include metric_time in List Dimensions Output Where Appropriate ([673](https://github.com/dbt-labs/metricflow/issues/673))
- Enable support for Redshift queries in dbt-metricflow integration ([582](https://github.com/dbt-labs/metricflow/issues/582))
- Enable Databricks support for the dbt-metricflow integration ([580](https://github.com/dbt-labs/metricflow/issues/580))
- Enable support for BigQuery for dbt metricflow integration users ([581](https://github.com/dbt-labs/metricflow/issues/581))

Fixes

- Removes MySQL from SqlEngine and SqlDialect options since it is not supported. ([0](https://github.com/dbt-labs/metricflow/issues/0))
- Derived metrics were not respecting the constraint defined in the original input metric's definition.
- Fixes type error in BigQuerySqlExpressionRenderer ([536](https://github.com/dbt-labs/metricflow/issues/536))
- Fix broken type signature for log_call decorator
- Apply transformations to dbt-generated serialized model to fix issue with query generation ([624](https://github.com/dbt-labs/metricflow/issues/624))
- Improve error message rendering in MetricFlow CLI ([646](https://github.com/dbt-labs/metricflow/issues/646))
- Added --version and fix manifest transformer rules for dbt-core-=1.6.0b8 ([650](https://github.com/dbt-labs/metricflow/issues/650))
- Include granularity suffix on time dimension name rendering for all time dimension granularities
- Clean up list dimensions outputs

Under the Hood

- Adding Changie ([457](https://github.com/dbt-labs/metricflow/issues/457))
- Ensure use of ValidationIssue instead of ValidationIssueType. ValidationIssueType was from a time before ValidationIssue classes had proper inheritance, and it's continued use was become problematic for typing.
- Removing `model` from the `ModelValidator.validate` return type. The model isn't altered, and thus doesn't need to be returned.
- Moving AggregationType enum into dbt-semantic-interfaces
- Moving errors relevant to dbt_semantic_interfaces to dbt_semantic_interfaces
- Migrating to RapidFuzz ([470](https://github.com/dbt-labs/metricflow/issues/470))
- Matching dbt-core issue templates ([457](https://github.com/dbt-labs/metricflow/issues/457))
- Removing the transform CLA ([450](https://github.com/dbt-labs/metricflow/issues/450))
- Pinning dbt-core to 1.4 ([475](https://github.com/dbt-labs/metricflow/issues/475))
- Removing YamlLint ([472](https://github.com/dbt-labs/metricflow/issues/472))
- Add ObjectToReference class in preparation of removing the .reference calls ([463,](https://github.com/dbt-labs/metricflow/issues/#463,), [464](https://github.com/dbt-labs/metricflow/issues/#464))
- Moving all *Reference objects to dbt-semantic-interfaces.
- Add pytest flag to use a persistent source schema for faster repeat testing. ([482](https://github.com/dbt-labs/metricflow/issues/482))
- Renamed instances of and related to Identifiers to Entities. ([dbt-semantic-interfaces9](https://github.com/dbt-labs/metricflow/issues/dbt-semantic-interfaces#9))
- Improves typechecking coverage by updgrading to MyPy 0.942 and removing blanket ignore all imports setting ([536](https://github.com/dbt-labs/metricflow/issues/536))
- Push mypy to run using local environment packages in pre-commit. Developers should always use a clean virtual environment to ensure consistency with CI. ([530](https://github.com/dbt-labs/metricflow/issues/530), [#536](https://github.com/dbt-labs/metricflow/issues/536))
- Update mypy to 1.3.0 ([546](https://github.com/dbt-labs/metricflow/issues/546))
- Migrate from Poetry -> Hatch for Project / Package Management ([549](https://github.com/dbt-labs/metricflow/issues/549))
- Enable the ability to return only dimensions requested in the query, specifically used for dimension values queries.
- Raising a UnsupportedEngineFeatureError instead of a generic RuntimeError when a data platform doesn't support a feature
- Remove SqlIsolationLevel constructs and other vestigial remnants of defunct SqlClient features ([577](https://github.com/dbt-labs/metricflow/issues/577))
- Raise a more specific exception when a Metric isn't found during linking.
- Update test environment configuration to allow for more streamlined dependencies
- Remove DDL and other unused methods from SqlClient protocol

Dependencies

- Switches MetricFlow SemanticManifest dependencies from the local dbt semantic interfaces package to the initial dev release ([540](https://github.com/dbt-labs/metricflow/pull/540))
- Clean up unused dependencies, relax tabulate version pin ([545](https://github.com/dbt-labs/metricflow/pull/545))
- Update dbt dependencies to support development on the new integration ([571](https://github.com/dbt-labs/metricflow/pull/571))
- Move SQLAlchemy and SQL engine dependencies out of the production package ([672](https://github.com/dbt-labs/metricflow/pull/672))
- Update dependencies and attribution file in preparation for 0.200.0 release ([703](https://github.com/dbt-labs/metricflow/pull/703))

Contributors
- [DevonFulcher](https://github.com/DevonFulcher)
- [QMalcolm](https://github.com/QMalcolm)
- [WilliamDee](https://github.com/WilliamDee)
- [callum-mcdata](https://github.com/callum-mcdata)
- [courtneyholcomb](https://github.com/courtneyholcomb)
- [nhandel](https://github.com/nhandel)
- [plypaul](https://github.com/plypaul)
- [tlento](https://github.com/tlento)

Breaking Changes

- Result layout is changing from one row per metric/null dimension valued pair to one row per null dimension value regardless of number of metrics in the query. This only affects queries for multiple metrics where the requested dimensions contain null values. See the description on the [relevant PR](https://github.com/transform-data/metricflow/pull/366) for more detailed information and an example illustrating how the output will change.
- Updates to the required SqlClient protocol could cause typechecking failures for users injecting a custom SqlClient implementation into the MetricFlowClient
- Version minimum changes in SQLAlchemy and snowflake-sqlalchemy could cause dependency conflicts when installed in python environments with libraries requiring an older version of either of these dependencies.

Added

- Support for derived metrics - users can now define metrics based on expressions that use other metrics in the model as inputs. For example usage, see the description in the [initial PR](https://github.com/transform-data/metricflow/pull/281) (WilliamDee)
- Support for versioned dimension joins against a standard Slowly Changing Dimension snapshot table (SCD Type II dataset) - users can now define a data source representing a dimension-only data set with start and end time columns representing validitiy windows for the dimension values, and MetricFlow will automatically join measures to the valid row entry. For details on limitations and future steps please refer to the [initial PR](https://github.com/transform-data/metricflow/pull/276). Note we no longer use the `primary` key type, as support for a `natural` key type was added in a [follow-up PR](https://github.com/transform-data/metricflow/pull/358). (tlento)
- Support for percentile measure definitions (kyleli626)
- Support for querying metrics without grouping by dimensions (WilliamDee)
- Support for generating a MetricFlow model from a dbt cloud metric model (QMalcolm)
- A `cancel_request` API in the SQL client for canceling running queries, with the necessary support for SQL isolation levels and asynchronous query submission (plypaul)
- Support for passing in query tags for Snowflake queries (plypaul)
- DataFlowPlan optimization to reduce source table scans (plypaul)
- Internal API to enable developers to fetch joinable data source targets from an input data source (courtneyholcomb)

Updated

- Improved readability of validation error messages (QMalcolm)
- Made Postgres engine tests merge-blocking in CI to reduce cycle time on detecting engine-specific errors (tlento)
- Updated poetry and python versions in CI to align with our build process and verify all supported Python versions (tlento)
- Eliminated data source level primary time dimension requirement in cases where all measures have an aggregation time dimension set (QMalcolm)
- Extended support for typed values for bind parameters (courtneyholcolm)
- Removed the optional Python levenshtein package from build dependencies in order to streamline package version requirements (plypaul)
- Consolidated join validation logic to eliminate code duplication and speed development (plypaul)
- Factored join building logic out of DataflowToSqlQueryPlanBuilder to streamline development (tlento)
- Improved visibility on underlying errors thrown by sql client requests (courtneyholcomb)
- Updated SQLAlchemy and snowflake-sqlalchemy minimum version requirements to resolve a version incompatibility introduced with SQLAlchemy 1.4.42 (tlento)
- Added CI coverage for Databricks SQL Warehouse execution environments (tlento)

Fixed

- Resolved error encountered in Databricks whenever table rename methods were invoked (courtneyholcomb)
- Fixed bug with warehouse measure validation where an error would be inappropriately thrown when users with measure-specific agg_time_dimension configurations attempted to run the full validation suite (WilliamDee)
- Issue with parsing `explain` output for Databricks SQL warehouse configurations (courtneyholcomb)
- Floating point comparison errors in CI tests (tlento)
- Issue with incomplete time range constraint validation that could result in invalid queries(plypaul)
- Resolved GitHub upgrade warnings on use of deprecated APIs and node.js build versions (tlento)
- Resolved python-levenshtein optimization warning on CLI startup (jzhu13)
- Resolved SQLAlchemy warning about the impending deprecation of the `engine.table_names` method (Jstein77)
- Improved error message for queries with time range constraints which were too narrow for the chosen time granularity (kyleli626)
- Eliminate SQL rendering error in BigQuery which would intermittently produce invalid GROUP BY specifications (tlento)

Added
- Support for overriding dbt `profile` and `targets` attributes when querying dbt models (QMalcolm)
- Validation to block use of `DISTINCT` keyword in `COUNT` aggregation expressions, as this can lead to incorrect results if optimized queries relying on partial aggregation attempt to do something like `SUM(counts)` to retrieve a less granular total value. (tlento)

Updated
- Made minor improvements to safeguards for internal development (tlento)

Added
- [292](https://github.com/transform-data/metricflow/pull/292) This is the initial implementation of the MetricFlow dbt metrics integration. Allows MetricFlow to bootstrap itself natively from a dbt project that has metrics 🎉

Updated
- Refactored the semantic layer to use the MetricReference object as opposed to MetricSpec (WilliamDee)

Breaking Changes
- Minor API change on SqlClient protocol could break existing API users writing custom SqlClient classes. MyPy should detect this issue. See [the relevant PR](https://github.com/transform-data/metricflow/pull/282) for details.

Added
- Support for Databricks! Now you can use Metricflow with your Databricks-backed warehouse! (courtneyholcomb)
- The ability to define constraints on input measures for ratio and expr metric types. This is a temporary quality of life improvement until full-featured derived metrics (i.e., metrics based on other metrics) are available. (tlento)
- Support for different time granularities, improved group by expression handling, and corrected constraint handling on semi-additive measures (WilliamDee)
- Support for `count` as a measure aggregation type. Note this is implemented as an alias around `sum`, so use of the `DISTINCT` keyword in expressions is not supported, and will be blocked via validation in a separate update. Users wishing for a `COUNT(DISTINCT thing)` equivalent should continue to use the `count_distinct` aggregation type. (WilliamDee)

Fixed
- Resolved incorrect constraint handling with semi-additive measures (WilliamDee)
- Eliminated Dataclass deserialization errors on default parameters (tlento)

Updated
- Optimized multi-hop join candidate selection (plypaul)
- Improved error handling and error messages on config validation (QMalcolm, tlento)
- Streamlined our project README (nhandel)
- CLI now accepts metric-only queries in limited circumstances. See the assertion checks [in the PR](https://github.com/transform-data/metricflow/pull/256) for details. We will announce this as a new feature when it is complete, but for the time being users may try it out to see if it meets their needs in its current form. (WilliamDee)