Dagster

New

- Added `run_id` to the `run_tags` index to improve database performance. Run `dagster instance migrate` to update the index. (Thanks, [HynekBlaha](https://github.com/HynekBlaha)!)
- Added icons for `kind` tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, [swrookie](https://github.com/swrookie)!)
- Added a new icon for the Denodo kind tag. (Thanks, [tintamarre](https://github.com/tintamarre)!)
- Errors raised from defining more than one `Definitions` object at module scope now include the object names so that the source of the error is easier to determine.
- [ui] Asset metadata entries like `dagster/row_count` now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
- [dagster-aws] `PipesGlueClient` now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
- [dagster-aws] `PipesEMRServerlessClient` now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
- [dagster-aws] `PipesECSClient` now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
- [dagster-aws] `PipesEMRClient` now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
- [dagster-databricks] `PipesDatabricksClient` now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
- [dagster-fivetran] Added `load_fivetran_asset_specs` function. It can be used with the `FivetranWorkspace` resource and `DagsterFivetranTranslator` translator to load your Fivetran connector tables as external assets in Dagster.
- [dagster-looker] Errors are now handled more gracefully when parsing derived tables.
- [dagster-sigma] Sigma assets now contain extra metadata and kind tags.
- [dagster-sigma] Added support for direct workbook to warehouse table dependencies.
- [dagster-sigma] Added `include_unused_datasets` field to `SigmaFilter` to disable pulling datasets that aren't used by a downstream workbook.
- [dagster-sigma] Added `skip_fetch_column_data` option to skip loading Sigma column lineage. This can speed up loading large instances.
- [dagster-sigma] Introduced an experimental `dagster-sigma snapshot` command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

Introducing: `dagster-airlift` (experimental)

`dagster-airlift` is coming out of stealth. See the initial Airlift RFC [here](https://github.com/dagster-io/dagster/discussions/25279), and the following documentation to learn more:

- A full [Airflow migration tutorial](https://docs.dagster.io/integrations/airlift/tutorial/overview).
- A tutorial on [federating between Airflow instances](https://docs.dagster.io/integrations/airlift/federation-tutorial/overview).

More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the `airflow-migration` channel in the Dagster slack.

Bugfixes

- Fixed a bug in run status sensors where setting incompatible arguments `monitor_all_code_locations` and `monitored_jobs` did not raise the expected error. (Thanks, [apetryla](https://github.com/apetryla)!)
- Fixed an issue that would cause the label for `AutomationCondition.any_deps_match()` and `AutomationCondition.all_deps_match()` to render incorrectly when `allow_selection` or `ignore_selection` were set.
- Fixed a bug which could cause code location load errors when using `CacheableAssetsDefinitions` in code locations that contained `AutomationConditions`
- Fixed an issue where the default multiprocess executor kept holding onto subprocesses after their step completed, potentially causing `Too many open files` errors for jobs with many steps.
- [ui] Fixed an issue introduced in 1.9.2 where the backfill overview page would sometimes display extra assets that were targeted by the backfill.
- [ui] Fixed "Open in Launchpad" button when testing a schedule or sensor by ensuring that it opens to the correct deployment.
- [ui] Fixed an issue where switching a user setting was immediately saved, rather than waiting for the change to be confirmed.
- [dagster-looker] Unions without unique/distinct criteria are now properly handled.
- [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.
- [dagster-sigma] Fixed an issue where API fetches did not paginate properly.

Documentation

- Added an [Airflow Federation Tutorial](https://docs.dagster.io/integrations/airlift/federation-tutorial/overview).
- Added `dagster-dingtalk` to the list of [community supported libraries](https://docs.dagster.io/integrations#community-supported-libraries).
- Fixed typos in the `dagster-wandb` (Weights and Biases) documentation. (Thanks, [matt-weingarten](https://github.com/matt-weingarten)!)
- Updated the [Role-based Access Control (RBAC)](https://docs.dagster.io/dagster-plus/account/managing-users/managing-user-roles-permissions) documentation.
- Added additional information about filtering to the [`dagster-sigma`](https://docs.dagster.io/integrations/sigma) documentation.

Dagster Plus

- [ui] Fixed an issue with filtering and catalog search in branch deployments.
- [ui] Fixed an issue where the asset graph would reload unexpectedly.

New

- Introduced a new constructor, `AssetOut.from_spec`, that will construct an `AssetOut` from an `AssetSpec`.
- [ui] Column tags are now displayed in the `Column name` section of the asset overview page.
- [ui] Introduced an icon for the `gcs` (Google Cloud Storage) kind tag.
- [ui] Introduced icons for `report` and `semanticmodel` kind tags.
- [ui] The tooltip for a tag containing a cron expression now shows a human-readable, timezone-aware cron string.
- [ui] Asset check descriptions are now sourced from docstrings and rendered in the UI. (Thanks, [marijncv](https://github.com/marijncv)!)
- [dagster-aws] Added option to propagate tags to ECS tasks when using the `EcsRunLauncher`. (Thanks, [zyd14](https://github.com/zyd14)!)
- [dagster-dbt] You can now implement `DagsterDbtTranslator.get_code_version` to customize the code version for your dbt assets. (Thanks, [Grzyblon](https://github.com/Grzyblon)!)
- [dagster-pipes] Added the ability to pass arbitrary metadata to `PipesClientCompletedInvocation`. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.
- [dagster-powerbi] During a full workspace scan, owner and column metadata is now automatically attached to assets.

Bugfixes

- Fixed an issue with `AutomationCondition.execution_in_progress` which would cause it to evaluate to `True` for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.
- Fixed an issue with `AutomationCondition.run_in_progress` that would cause it to ignore queued runs.
- Fixed an issue that would cause a `default_automation_condition_sensor` to be constructed for user code servers running on dagster version `< 1.9.0` even if the legacy `auto_materialize: use_sensors` configuration setting was set to `False`.
- [ui] Fixed an issue when executing asset checks where the wrong job name was used in some situations. The correct job name is now used.
- [ui] Selecting assets with 100k+ partitions no longer causes the asset graph to temporarily freeze.
- [ui] Fixed an issue that could cause a GraphQL error on certain pages after removing an asset.
- [ui] The asset events page no longer truncates event history in cases where both materialization and observation events are present.
- [ui] The backfill coordinator logs tab no longer sits in a loading state when no logs are available to display.
- [ui] Fixed issue which would cause the "Partitions evaluated" label on an asset's automation history page to incorrectly display `0` in cases where all partitions were evaluated.
- [ui] Fix "Open in Playground" link when testing a schedule or sensor by ensuring that it opens to the correct deployment.
- [ui] Fixed an issue where the asset graph would reload unexpectedly.
- [dagster-dbt] Fixed an issue where the SQL filepath for a dbt model was incorrectly resolved when the dbt manifest file was built on a Windows machine, but executed on a Unix machine.
- [dagster-pipes] Asset keys containing embedded `/` characters now work correctly with Dagster Pipes.

Documentation

- Community-hosted integrations are now listed on the [Integrations](https://docs.dagster.io/integrations) page.
- Added a [tutorial](https://docs.dagster.io/integrations/airlift/tutorial/overview), [reference page](https://docs.dagster.io/integrations/airlift/reference) and [API docs](https://docs.dagster.io/_apidocs/libraries/dagster-airlift) for `dagster-airlift`.
- Fixed a typo in the label for superseded APIs. (Thanks, [matt-weingarten](https://github.com/matt-weingarten)!)

Deprecations

- The `types-sqlalchemy` package is no longer included in the `dagster[pyright]` extra package.

Dagster Plus

- [ui] The Environment Variables table can now be sorted by name and update time.
- [ui] The code location configuration dialog now contains more metadata about the code location.
- [ui] Fixed an issue where the incorrect user icons were shown in the Users table when a search filter had been applied.

New

- `dagster project scaffold` now has an option to create dagster projects from templates with excluded files/filepaths.
- [ui] Filters in the asset catalog now persist when navigating subdirectories.
- [ui] The Run page now displays the partition(s) a run was for.
- [ui] Filtering on owners/groups/tags is now case-insensitive.
- [dagster-tableau] the helper function `parse_tableau_external_and_materializable_asset_specs` is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.
- [dagster-looker] Looker assets now by default have owner and URL metadata.
- [dagster-k8s] Added a per_step_k8s_config configuration option to the k8s_job_executor, allowing the k8s configuration of individual steps to be configured at run launch time (thanks Kuhlwein!)
- [dagster-fivetran] Introduced `DagsterFivetranTranslator` to customize assets loaded from Fivetran.
- [dagster-snowflake] `dagster_snowflake.fetch_last_updated_timestamps` now supports ignoring tables not found in Snowflake instead of raising an error.

Bugfixes

- Fixed issue which would cause a `default_automation_condition_sensor` to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy `auto_materialize: use_sensors` configuration setting was set to `False`.
- Fixed an issue where running `dagster instance migrate` on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.
- Fixed an issue where wiping a dynamically partitioned asset causes an error.
- [dagster-polars] `ImportError`s are no longer raised when bigquery libraries are not installed [25708]

Documentation

- [dagster-dbt] A guide on how to use dbt defer with Dagster branch deployments has been added to the dbt reference.

Major changes since 1.8.0 (core) / 0.24.0 (libraries)

Automation

- Declarative Automation, the system which enables setting per-asset `AutomationConditions`, is no longer experimental. We now recommend using this system in all cases where asset-centric orchestration is desired. A suite of built-in static constructors have been added for common usecases, such as `AutomationCondition.on_missing()` (which can fill in missing partitions of assets as soon as upstream data is available), and `AutomationCondition.all_deps_blocking_checks_passed()` (which can prevent materialization of assets until all upstream blocking checks have passed).
- You can now assign `AutomationConditions` to asset checks, via the `automation_condition` parameter on `asset_check` or `AssetCheckSpec`.
- You can now assign `AutomationConditions` to observable source assets, via the `automation_condition` parameter on `observable_source_asset`.
- [experimental] You can now define custom subclasses of `AutomationCondition` to execute arbitrary Python code in the context of a broader expression. This allows you to compose built-in conditions with custom business logic.
- The `target` arguments on schedules and sensors are now marked stable, allowing a stable way for schedules and sensors to target asset selections without needing to define a job.

Integrations

- Introduced a slate of integrations with business intelligence (BI) tools, enabling dashboards, views, and reports to be represented in the Dagster asset graph.
- [Looker](https://docs.dagster.io/integrations/looker)
- [Power BI](https://docs.dagster.io/integrations/powerbi)
- [Sigma](https://docs.dagster.io/integrations/sigma)
- [Tableau](https://docs.dagster.io/integrations/tableau)
- A rich set of metadata is now automatically collected by our suite of ELT integrations.
- The `dagster/table_name` metadata tag, containing the fully-qualified name of the destination model, has been added for Airbyte, dlt, Fivetran and Sling assets.
- The `dagster/row_count` metadata tag, containing the number of records loaded in the corresponding run, has been added for dlt and Sling assets.
- The `dagster/column_schema` metadata tag, containing column schema information of the destination tables, has been added for Fivetran assets.
- Column lineage information is now collected for Sling assets.
- [dagster-pipes](https://docs.dagster.io/concepts/dagster-pipes) are replacing the now deprecated Step Launchers as the new recommended approach for executing remote Spark jobs. Three new [Pipes clients](https://docs.dagster.io/_apidocs/libraries/dagster-aws#clients) for running Spark applications on Amazon Web Services have been added:
- `dagster_aws.pipes.PipesGlueClient`
- `dagster_aws.pipes.PipesEMRServerlessClient`
- `dagster_aws.pipes.PipesEMRClient`

UI

- Several changes have been made to the information architecture to make it easier to find what you’re looking for:
- Backfills have been moved from their own tab underneath the Overview page to entries within the table on the Runs page. This reflects the fact that backfills and runs are similar entities that share most properties. You can continue to use the legacy Runs page with the “Revert to legacy Runs page” user setting. ([GitHub Discussion](https://github.com/dagster-io/dagster/discussions/24898))
- “Jobs” is now a page reachable from the top-level navigation pane. It replaces the Jobs tab within the Overview page.
- “Automations” is now a page reachable from the top-level navigation pane. It replaces the schedule and sensor tabs within the Overview page.
- `asset` and `AssetSpec` now have a `kinds` attribute that enables specifying labels that show up on asset nodes in the asset graph in the UI. This supersedes the `compute_kind` attribute.

Changes since 1.8.13 (core) / 0.24.13 (libraries)

New

- The `tags` parameter to `asset` and `AssetSpec` is no longer marked as experimental.
- The `observable_source_asset` decorator now supports an `automation_condition` argument.
- `AutomationCondition` and associated APIs are no longer marked as experimental.
- Added a new `use_user_code_server` parameter to `AutomationConditionSensorDefinition`. If set, the sensor will be evaluated in the user code server (as traditional sensors are), allowing custom `AutomationCondition` subclasses to be evaluated.
- Added a new column to the BulkActions table, a new column to the Runs table, and a new BackfillTags table to improve the performance of the Runs page. To take advantage of these performance improvements, run `dagster instance migrate`. This migration involves a schema migration to add the new columns and table, and a data migration to populate the new columns for historical backfills and runs.
- Performance improvements when loading definitions with multi-assets with many asset keys.
- [ui] The previously-experimental changes to the top nav are now enabled for all users.
- [ui] Added new code location pages which provide information regarding library versions, metadata, and definitions.
- [ui] The new version of the Runs page is now enabled by default. To use the legacy version of the Runs page, toggle the "Revert to legacy Runs page" user setting.
- [ui] Clicking an asset with failed partitions on the asset health overview now takes you to a list of the failed partitions.
- [ui] The Materialize button runs pre-flight checks more efficiently, resulting in faster run launch times.
- [dagster-pipes] Added support for multi-container log streaming (thanks, [MattyKuzyk](https://github.com/MattyKuzyk)!)
- [dagster-docker] `container_kwargs.stop_timeout` can now be set when using the `DockerRunLauncher` or `docker_executor` to configure the amount of time that Docker will wait when terminating a run for it to clean up before forcibly stopping it with a SIGKILL signal.
- [dagster-dbt] Performance improvements when loading definitions using `build_dbt_asset_selection`.

Bugfixes

- [ui] Fixed redirect behavior on full pageloads of the legacy auto-materialize overview page.
- [ui] Plots for assets that emit materialization and observation events at different rates no longer display a time period missing the more frequent event type.
- [ui] Fixed issue causing scrolling to misbehave on the concurrency settings page.
- [helm] The blockOpConcurrencyLimitedRuns section of queuedRunCoordinator now correctly templates the appropriate config.
- [dagster-pipes] Fixed issue where k8s ops would fail after 4 hours (thanks, [MattyKuzyk](https://github.com/MattyKuzyk)!)

Documentation

- [dagster-dbt] Added guide for using dbt defer with Dagster branch deployments.
- [docs] Step Launchers documentation has been removed and replaced with references to Dagster Pipes.
- [docs] Fixed code example in Dagster Essentials (thanks, [aleexharris](https://github.com/aleexharris)!)

Breaking Changes

- `dagster` no longer supports Python 3.8, which hit EOL on 2024-10-07.
- `dagster` now requires `pydantic>=2`.
- By default, `AutomationConditionSensorDefinitions` will now emit backfills to handle cases where more than one partition of an asset is requested on a given tick. This allows that asset's `BackfillPolicy` to be respected. This feature can be disabled by setting `allow_backfills` to `False`.
- Passing a custom `PartitionsDefinition` subclass into a `Definitions` object now issues an error instead of a deprecation warning.
- `AssetExecutionContext` is no longer a subclass of `OpExecutionContext`. At this release, `AssetExecutionContext` and `OpExecutionContext` implement the same methods, but in the future, the methods implemented by each class may diverge. If you have written helper functions with `OpExecutionContext` type annotations, they may need to be updated to include `AssetExecutionContext` depending on your usage. Explicit calls to `isinstance(context, OpExecutionContext)` will now fail if `context` is an `AssetExecutionContext`.
- The `asset_selection` parameter on `AutomationConditionSensorDefinition` has been renamed to `target`, to align with existing sensor APIs.
- The experimental `freshness_policy_sensor` has been removed, as it relies on the long-deprecated `FreshnessPolicy` API.
- The deprecated `external_assets_from_specs` and `external_asset_from_spec` methods have been removed. Users should use `AssetsDefinition(specs=[...])`, or pass specs directly into the `Definitions` object instead.
- `AssetKey` objects can no longer be iterated over or indexed in to. This behavior was never an intended access pattern and in all observed cases was a mistake.
- The `dagster/relation_identifier` metadata key has been renamed to `dagster/table_name`.
- [dagster-ge] `dagster-ge` now only supports `great_expectations>=0.17.15`. The `ge_validation_op_factory` API has been replaced with the API previously called `ge_validation_op_factory_v3`.
- [dagster-aws] Removed deprecated parameters from `dagster_aws.pipes.PipesGlueClient.run`.
- [dagster-embedded-elt] Removed deprecated parameter `dlt_dagster_translator` from `dlt_assets`. The `dagster_dlt_translator` parameter should be used instead.
- [dagster-polars] Dropped support for saving storage-level arbitrary metadata via IOManagers.

Deprecations

- The `DataBricksPysparkStepLauncher`, `EmrPySparkStepLauncher`, and any custom subclass of `StepLauncher` have been marked as deprecated, but will not be removed from the codebase until Dagster 2.0 is released, meaning they will continue to function as they currently do for the foreseeable future. Their functionality has been superseded by the interfaces provided by `dagster-pipes`, and so future development work will be focused there.
- The experimental `multi_asset_sensor` has been marked as deprecated, as its main use cases have been superseded by the `AutomationCondition` APIs. However, it will not be removed until version 2.0.0.

New

- Performance improvements when loading code locations using multi-assets with many asset keys.
- `AutomationCondition.in_progress()` now will be true if an asset partition is part of an in-progress backfill that has not yet executed it. The prior behavior, which only considered runs, is encapsulated in `AutomationCondition.execution_in_progress()`.
- [ui] Added tag filter to the jobs page.
- [ui] Preserve user login state for a longer period of time.
- [dagster-dbt] Performance improvements when loading definitions using `build_dbt_asset_selection`.
- [dagster-docker] `container_kwargs.stop_timeout` can now be set when using the `DockerRunLauncher` or `docker_executor` to configure the amount of time that Docker will wait when terminating a run for it to clean up before forcibly stopping it with a SIGKILL signal.
- [dagster-sigma] The Sigma integration now fetches initial API responses in parallel, speeding up initial load.
- [dagster-looker] Attempt to naively render liquid templates for derived table sql.
- [dagster-looker] Added support for views and explores that rely on refinements or extends.
- [dagster-looker] When fetching explores and dashboards from the Looker API, retrieve in parallel.

Bugfixes

- Fixed an issue with `AutomationCondition.eager()` that could cause it to attempt to launch a second attempt of an asset in cases where it was skipped or failed during a run where one of its parents successfully materialized.
- Fixed an issue which would cause `AutomationConditionSensorDefinitions` to not be evaluated if the `use_user_code_server` value was toggled after the initial evaluation.
- Fixed an issue where configuration values for aliased pydantic fields would be dropped.
- [ui] Fix an issue in the code locations page where invalid query parameters could crash the page.
- [ui] Fix navigation between deployments when query parameters are present in the URL.
- [helm] the blockOpConcurrencyLimitedRuns section of queuedRunCoordinator now correctly templates the appropriate config.
- [dagster-sigma] Fixed pulling incomplete data for very large workspaces.

New

- The `AutomationCondition.eager()`, `AutomationCondition.missing()`, and `AutomationCondition.on_cron` conditions are now compatible with asset checks.
- Added `AssetSelection.materializable()`, which returns only assets that are materializable in an existing selection.
- Added a new `AutomationCondition.all_deps_blocking_checks_passed` condition, which can be used to prevent materialization when any upstream blocking checks have failed.
- Added a `code_version` parameter to the `graph_asset` decorator.
- If a `LaunchPartitionBackfill` mutation is submitted to GQL with invalid partition keys, it will now return an early `PartitionKeysNotFoundError`.
- `AssetSelection.checks_for_assets` now accepts `AssetKey`s and string asset keys, in addition to `AssetsDefinition`s.
- [ui] Added a search bar to partitions tab on the asset details page.
- [ui] Restored docked left nav behavior for wide viewports.
- [dagster-aws] `get_objects` now has a `since_last_modified` that enables only fetching objects modified after a given timestamp.
- [dagster-aws] New AWS EMR Dagster Pipes client (`dagster_aws.pipes.PipesEMRCLient` ) for running and monitoring AWS EMR jobs from Dagster.
- [dagster-looker] Pinned the looker-sdk dependency below 24.18.0 to avoid this issue: https://github.com/looker-open-source/sdk-codegen/issues/1518.

Bugfixes

- Fixed an issue which could cause incorrect evaluation results when using self-dependent partition mappings with `AutomationConditions` that operate over dependencies.
- [ui] Fixed an issue where the breadcumb on asset pages would flicker nonstop.
- [dagster-embedded-elt] Fixed extraction of metadata for dlt assets whose source and destination identifiers differ.
- [dagster-databricks] Fixed a permissioning gap that existed with the `DatabricksPySparkStepLauncher`, so that permissions are now set correctly for non-admin users.
- [dagster-dbt] Fixed an issue where column metadata generated with `fetch_column_metadata` did not work properly for models imported through dbt dependencies.

Documentation

- [dagster-k8s] `DagsterK8sPipesClient.run` now shows up in API docs.

Dagster Plus

- [ui] Fixed a bug in the catalog UI where owners filters were not applied correctly.
- [ui] Fixed width of the column lineage dropdown selector on the asset page.
- [ui] Column lineage now correctly renders when set on asset definition metadata
- [ui] Fixed Settings link on the list of deployments, for users in the legacy navigation flag.