Dagster

New

- Added a new function `load_definitions_from_module`, which can load all the assets, checks, schedules, sensors, and job objects within a module scope into a single Definitions object. Check out [the documentation to learn more](https://docs.dagster.io/_apidocs/definitions#dagster.load_definitions_from_module).
- When using the `DAGSTER_REDACT_USER_CODE_ERRORS` environment variable to mask user code errors, the unmasked log lines are now written using a `dagster.masked` Python logger instead of being written to stderr, allowing the format of those log lines to be customized.
- The `croniter` package is now vendored in dagster.
- [ui] Corrected the `minstral` typo and updated the Mistral logo for asset `kinds` tag.
- [ui] The relevant runs are now shown within the same dialog when viewing details of an automation evaluation.
- [ui] Clicking to view runs with a specific status from the backfill overview now switches to the new backfill runs tab with your filters applied, instead of the global runs page.
- [ui] In the run timeline, all run ids and timings are now shown in the hover popover.
- [ui] Added a new tab on the Runs page that shows a filterable list of recent backfills.
- [dagster-airlift] Added support for Python 3.7.
- [dagster-aws] Added a `task_definition_prefix` argument to `EcsRunLauncher`, allowing the name of the task definition families for launched runs to be customized. Previously, the task definition families always started with `run`.
- [dagster-azure] Moved azure fake implementations to its own submodule, paving the way for fake implementations to not be imported by default. (Thanks [futurwasfree](https://github.com/futurwasfree)!)
- [dagster-dlt] The `dagster-dlt` library is added. It replaces the dlt module of `dagster-embedded-elt`.
- [dagster-sling] The `dagster-sling` library is added. It replaces the Sling module of `dagster-embedded-elt`.
- [helm] Added support for sidecar containers for all Dagster pods, for versions of K8s after 1.29 ([Native Sidecars](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)). (Thanks [hom3r](https://github.com/hom3r)!)

Bugfixes

- Fixed an issue where the tick timeline wouldn't load for an automation condition sensor that emitted a backfill.
- Fixed a bug with asset checks where additional_deps/additional_ins were not being threaded through properly in certain cases, and would result in errors at job creation.
- Fixed a bug where the UI will hit an unexpected error when loading details for a run containing a step retry before the step has started.
- Fixed a bug with load_assets_from_x functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if include_specs=True.
- Fixed a bug with `load_assets_from_modules` where AssetSpec objects were being given the key_prefix instead of the source_key_prefix. Going forward, when load_specs is set to True, only the source_key_prefix will affect AssetSpec objects.
- Fixed a bug with the run queue criteria UI for branch deployments in Dagster Plus.
- [ui] Fixed the "View evaluation" links from the "Automation condition" tag popover on Runs.
- [dagster-aws] Fixed an issue with the EcsRunLauncher where it would sometimes create a new task definition revision for each run if the "task_role_arn" or "execution_role_arn" parameters were specified without the `arn:aws:iam:` prefix.
- [dagster-aws] Fixed a bug with `PipesEMRServerlessClient` trying to get the dashboard URL for a run before it transitions to RUNNING state.
- [dagster-dbt] Fixed an issue where group names set on partitioned dbt assets created using the `dbt_assets` decorator would be ignored.
- [dagster-azure] Fixed the default configuration for the `show_url_only` parameter on the `AzureBlobComputeLogManager`. (Thanks [ion-elgreco](https://github.com/ion-elgreco)!)
- [dagster-aws] Fixed an issue handling null `networkConfiguration` parameters for the ECS run launcher. (Thanks [markgrin](https://github.com/markgrin)!)

Documentation

- Added example potential use cases for sensors. (Thanks [gianfrancodemarco](https://github.com/gianfrancodemarco)!)
- Updated the tutorial to match the outlined structure. (Thanks [vincent0426](https://github.com/vincent0426)!)

Deprecations

- [dagster-embedded-elt] the `dagster-embedded-elt` library is deprecated in favor of `dagster-dlt` and `dagster-sling`.

Dagster Plus

- The Alert Policies page will now show a warning if a slack channel for a policy no longer exists.

Bugfixes

- Fixed a bug with `load_assets_from_x` functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if `include_specs=True`.
- [dagster-azure] Fixed a bug in 1.9.6 and 1.9.7 where the default behavior of the compute log manager switched from showing logs in the UI to showing a URL. You can toggle the `show_url_only` option to `True` to enable the URL showing behavior.
- [dagster-dbt] Fixed an issue where group names set on partitioned dbt assets created using the `dbt_assets` decorator would be ignored

New

- Added new function `load_definitions_from_module`, which can load all the assets, checks, schedules, sensors, and job objects within a module scope into a single Definitions object. Check out the documentation to learn more: https://docs.dagster.io/_apidocs/definitions#dagster.load_definitions_from_module.
- Previously, asset backfills could only target selections of assets in which all assets had a `BackfillPolicy`, or none of them did. Mixed selections are now supported.
- `AssetSpecs` may now contain a `partitions_def`. Different `AssetSpecs` passed to the same invocation of `multi_asset` can now have different `PartitionsDefinitions`, as long as `can_subset=True`.
- Added the option to use a thread pool to process backfills in parallel.
- Exceptions that are raised when a schedule or sensor is writing to logs will now write an error message to stdout instead of failing the tick.
- Added validation of `title` for asset backfills (not just for job backfills).
- [ui] Design tweaks to the asset Automations tab.
- [ui] Asset selection filtering is now case insensitive.
- [ui] Add Teradata icon for kind tags.
- [ui] When creating and editing alerts, when the form is in an invalid state, display the reason on the disabled buttons.
- [ui] Add Automation history to asset checks.
- [ui] Improve performance of Run page for very long-running runs.
- [dagster-airbyte] The `airbyte_assets` decorator has been added. It can be used with the `AirbyteCloudWorkspace` resource and `DagsterAirbyteTranslator` translator to load Airbyte tables for a given connection as assets in Dagster. The `build_airbyte_assets_definitions` factory can be used to create assets for all the connections in your Airbyte workspace.
- [dagster-airbyte] Airbyte Cloud assets can now be materialized using the `AirbyteCloudWorkspace.sync_and_poll(…)` method in the definition of a `airbyte_assets` decorator.
- [dagster-airlift] Airflow imports are now compatible with Airflow 1.
- [dagster-aws] new `ecs_executor` which executes Dagster steps via AWS ECS tasks. This can be used in conjunction with `ECSRunLauncher`.
- [dagster-dbt] `dbt-core>=1.9` is now supported.
- [dagster-dbt] Adds SQL syntax highlighting to raw sql code in dbt asset descriptions.
- [dagster-looker] `load_looker_asset_specs` and `build_looker_pdt_assets_definitions` are updated to accept an instance of `DagsterLookerApiTranslator` or custom subclass.
- [dagster-looker] Type hints in the signature of `DagsterLookerApiTranslator.get_asset_spec` have been updated - the parameter `looker_structure` is now of type `LookerApiTranslatorStructureData` instead of `LookerStructureData`. Custom Looker API translators should be updated.
- [dagster-powerbi] `load_powerbi_asset_specs` has been updated to accept an instance of `DagsterPowerBITranslator` or custom subclass.
- [dagster-powerbi] Type hints in the signature of `DagsterPowerBITranslator.get_asset_spec` have been updated - the parameter `data` is now of type `PowerBITranslatorData` instead of `PowerBIContentData`. Custom Power BI translators should be updated.
- [dagster-sigma] `load_sigma_asset_specs` has been updated to accept an instance of `DagsterSigmaTranslator` or a custom subclass.
- [dagster-sigma] Type hints in the signature of `DagsterLookerApiTranslator.get_asset_spec` have been updated - the parameter `data` is now of type `Union[SigmaDatasetTranslatorData, SigmaWorkbookTranslatorData]` instead of `Union[SigmaDataset, SigmaWorkbook]`. Custom Looker API translators should be updated.
- [dagster-sigma] Added the option to filter to specific workbooks in addition to folders.
- [dagster-sigma] Added the option to skip fetching lineage for workbooks in cases where users want to build this information themselves.
- [dagster-tableau] `load_tableau_asset_specs` has been updated to accept an instance of `DagsterTableauTranslator` or custom subclass.
- [dagster-tableau] Type hints in the signature of `DagsterTableauTranslator.get_asset_spec` have been updated - the parameter `data` is now of type `TableauTranslatorData` instead of `TableauContentData`. Custom Tableau translators should be updated.

Bugfixes

- Fixed an issue where sensor and schedule tick logs would accumulate disk over time on Dagster code servers.
- [ui] Fixed an issue where the app sometimes loads with styles missing.
- [ui] Fix search string highlighting in global search results.
- Fixed a race condition where immediately after adding a new asset to the graph, a freshness check sensor targeting that asset might raise an InvalidSubsetError in its first one.
- [ui] Fixed a bug where backfills launched by Declarative Automation were not being shown in the table of launched runs.
- The `dagster-airlift` package erroneously introduced a dependency on `dagster`. This has been rectified - `dagster` is only required for the `dagster-airlift[core]` submodule.

Deprecations

- Deprecation of `multi_asset_sensor` has been rolled back.

Dagster Plus

- Introduce the Catalog Viewer role for Users and Teams.
- Slack, MS Teams, and email alerts for run failures will list the steps that were successful or not executed.
- [experimental] The option `blobStorageSnapshotUploads` has been added which enables a new process for how definition snapshots are uploaded to Dagster Cloud.
- Fixed a catalog search issue where exact prefix matches are not prioritized in the search results.
- Fixed a bug with Insights metric customization.

New

- Updated `cronitor` pin to allow versions `>= 5.0.1` to enable use of `DayOfWeek` as 7. Cronitor `4.0.0` is still disallowed. (Thanks, [joshuataylor](https://github.com/joshuataylor)!)
- Added flag `checkDbReadyInitContainer` to optionally disable db check initContainer.
- Added job name filtering to increase the throughput for run status sensors that target jobs.
- [ui] Added Google Drive icon for `kind` tags. (Thanks, [dragos-pop](https://github.com/dragos-pop)!)
- [ui] Renamed the run lineage sidebar on the Run details page to `Re-executions`.
- [ui] Sensors and schedules that appear in the Runs page are now clickable.
- [ui] Runs targeting assets now show more of the assets in the Runs page.
- [dagster-airbyte] The destination type for an Airbyte asset is now added as a `kind` tag for display in the UI.
- [dagster-gcp] `DataprocResource` now receives an optional parameter `labels` to be attached to Dataproc clusters. (Thanks, [thiagoazcampos](https://github.com/thiagoazcampos)!)
- [dagster-k8s] Added a `checkDbReadyInitContainer` flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, [easontm](https://github.com/easontm)!)
- [dagster-k8s] K8s pod logs are now logged when a pod fails. (Thanks, [apetryla](https://github.com/apetryla)!)
- [dagster-sigma] Introduced `build_materialize_workbook_assets_definition` which can be used to build assets that run materialize schedules for a Sigma workbook.
- [dagster-snowflake] `SnowflakeResource` and `SnowflakeIOManager` both accept `additional_snowflake_connection_args` config. This dictionary of arguments will be passed to the `snowflake.connector.connect` method. This config will be ignored if you are using the `sqlalchemy` connector.
- [helm] Added the ability to set user-deployments labels on k8s deployments as well as pods.

Bugfixes

- Assets with self dependencies and `BackfillPolicy` are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.
- Previously, the freshness check sensor would not re-evaluate freshness checks if an in-flight run was planning on evaluating that check. Now, the freshness check sensor will kick off an independent run of the check, even if there's already an in flight run, as long as the freshness check can potentially fail.
- Previously, if the freshness check was in a failing state, the sensor would wait for a run to update the freshness check before re-evaluating. Now, if there's a materialization later than the last evaluation of the freshness check and no planned evaluation, we will re-evaluate the freshness check automatically.
- [ui] Fixed run log streaming for runs with a large volume of logs.
- [ui] Fixed a bug in the Backfill Preview where a loading spinner would spin forever if an asset had no valid partitions targeted by the backfill.
- [dagster-aws] `PipesCloudWatchMessageReader` correctly identifies streams which are not ready yet and doesn't fail on `ThrottlingException`. (Thanks, [jenkoian](https://github.com/jenkoian)!)
- [dagster-fivetran] Column metadata can now be fetched for Fivetran assets using `FivetranWorkspace.sync_and_poll(...).fetch_column_metadata()`.
- [dagster-k8s] The k8s client now waits for the main container to be ready instead of only waiting for sidecar init containers. (Thanks, [OrenLederman](https://github.com/OrenLederman)!)

Documentation

- Fixed a typo in the `dlt_assets` API docs. (Thanks, [zilto](https://github.com/zilto)!)

New

- The automatic run retry daemon has been updated so that there is a single source of truth for if a run will be retried and if the retry has been launched. Tags are now added to run at failure time indicating if the run will be retried by the automatic retry system. Once the automatic retry has been launched, the run ID of the retry is added to the original run.
- When canceling a backfill of a job, the backfill daemon will now cancel all runs launched by that backfill before marking the backfill as canceled.
- Dagster execution info (tags such as `dagster/run-id`, `dagster/code-location`, `dagster/user` and Dagster Cloud environment variables) typically attached to external resources are now available under `DagsterRun.dagster_execution_info`.
- `SensorReturnTypesUnion` is now exported for typing the output of sensor functions.
- [dagster-dbt] dbt seeds now get a valid code version (Thanks [marijncv](https://github.com/marijncv)!).
- Manual and automatic retries of runs launched by backfills that occur while the backfill is still in progress are now incorporated into the backfill's status.
- Manual retries of runs launched by backfills are no longer considered part of the backfill if the backfill is complete when the retry is launched.
- [dagster-fivetran] Fivetran assets can now be materialized using the FivetranWorkspace.sync_and_poll(…) method in the definition of a `fivetran_assets` decorator.
- [dagster-fivetran] `load_fivetran_asset_specs` has been updated to accept an instance of `DagsterFivetranTranslator` or custom subclass.
- [dagster-fivetran] The `fivetran_assets` decorator was added. It can be used with the `FivetranWorkspace` resource and `DagsterFivetranTranslator` translator to load Fivetran tables for a given connector as assets in Dagster. The `build_fivetran_assets_definitions` factory can be used to create assets for all the connectors in your Fivetran workspace.
- [dagster-aws] `ECSPipesClient.run` now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks [jenkoian](https://github.com/jenkoian)!)
- [dagster-dbt] Update dagster-dbt scaffold template to be compatible with uv (Thanks [wingyplus](https://github.com/wingyplus)!).
- [dagster-airbyte] A `load_airbyte_cloud_asset_specs` function has
been added. It can be used with the `AirbyteCloudWorkspace` resource and `DagsterAirbyteTranslator` translator to load your Airbyte Cloud connection streams as external assets in Dagster.
- [ui] Add an icon for the `icechunk` kind.
- [ui] Improved ui for manual sensor/schedule evaluation.

Bugfixes

- Fixed database locking bug for the `ConsolidatedSqliteEventLogStorage`, which is mostly used for tests.
- [dagster-aws] Fixed a bug in the ECSRunLauncher that prevented it from accepting a user-provided task definition when DAGSTER_CURRENT_IMAGE was not set in the code location.
- [ui] Fixed an issue that would sometimes cause the asset graph to fail to render on initial load.
- [ui] Fix global auto-materialize tick timeline when paginating.

New

- Global op concurrency is now enabled on the default SQLite storage. Deployments that have not been migrated since `1.6.0` may need to run `dagster instance migrate` to enable.
- Introduced `map_asset_specs` to enable modifying `AssetSpec`s and `AssetsDefinition`s in bulk.
- Introduced `AssetSpec.replace_attributes` and `AssetSpec.merge_attributes` to easily alter properties of an asset spec.
- [ui] Add a "View logs" button to open tick logs in the sensor tick history table.
- [ui] Add Spanner kind icon.
- [ui] The asset catalog now supports filtering using the asset selection syntax.
- [dagster-pipes, dagster-aws] `PipesS3MessageReader` now has a new parameter `include_stdio_in_messages` which enables log forwarding to Dagster via Pipes messages.
- [dagster-pipes] Experimental: A new Dagster Pipes message type `log_external_stream` has been added. It can be used to forward external logs to Dagster via Pipes messages.
- [dagster-powerbi] Opts in to using admin scan APIs to pull data from a Power BI instance. This can be disabled by passing `load_powerbi_asset_specs(..., use_workspace_scan=False)`.
- [dagster-sigma] Introduced an experimental `dagster-sigma snapshot` command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

Bugfixes

- Fixed a bug that caused `DagsterExecutionStepNotFoundError` errors when trying to execute an asset check step of a run launched by a backfill.
- Fixed an issue where invalid cron strings like "0 0 30 2 \*" that represented invalid dates in February were still allowed as Dagster cron strings, but then failed during schedule execution. Now, these invalid cronstrings will raise an exception when they are first loaded.
- Fixed a bug where `owners` added to `AssetOut`s when defining a `graph_multi_asset` were not added to the underlying `AssetsDefinition`.
- Fixed a bug where using the `&` or `|` operators on `AutomationCondition`s with labels would cause that label to be erased.
- [ui] Launching partitioned asset jobs from the launchpad now warns if no partition is selected.
- [ui] Fixed unnecessary middle truncation occurring in dialogs.
- [ui] Fixed timestamp labels and "Now" line rendering bugs on the sensor tick timeline.
- [ui] Opening Dagster's UI with a single job defined takes you to the Overview page rather than the Job page.
- [ui] Fix stretched tags in backfill table view for non-partitioned assets.
- [ui] Open automation sensor evaluation details in a dialog instead of navigating away.
- [ui] Fix scrollbars in dark mode.
- [dagster-sigma] Workbooks filtered using a `SigmaFilter` no longer fetch lineage information.
- [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.

Deprecations

- [dagster-powerbi] `DagsterPowerBITranslator.get_asset_key` is deprecated in favor of `DagsterPowerBITranslator.get_asset_spec().key`
- [dagster-looker] `DagsterLookerApiTranslator.get_asset_key` is deprecated in favor of `DagsterLookerApiTranslator.get_asset_spec().key`
- [dagster-sigma] `DagsterSigmaTranslator.get_asset_key` is deprecated in favor of `DagsterSigmaTranslator.get_asset_spec().key`
- [dagster-tableau] `DagsterTableauTranslator.get_asset_key` is deprecated in favor of `DagsterTableauTranslator.get_asset_spec().key`