Dagster

New

- Applying changes from sensor test results now also applies changes from dynamic partition requests.
- When merging assets from multiple code locations, autogenerated specs are now prioritized lower than customized external asset specs.
- [ui] Allowed using command-click to view a run from the toast message that appears when starting a materialization of an asset.
- [ui] Asset graph can now zoom out a lot more.
- [ui] Added a kind tag for dbt Cloud.
- [dagster-dlt] Added backfill policy to dlt_assets, defaulting to single-run. (Thanks [neuromantik33](https://github.com/neuromantik33)!)

Bugfixes

- Updated `AutomationCondition.initial_evaluation` condition to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.
- Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected `dagster-airbyte`, `dagster-dlift`, `dagster-dbt`, `dagster-fivetran`, `dagster-looker`, `dagster-powerbi`, `dagster-sigma`, and `dagster-tableau`.
- [ui] Fixed an issue with the Run step selection input autocomplete where it would suggest `key:"*substring*"` instead of `name:"*substring*"`.
- [ui] Fixed the "View run" link shown when launching runs.
- [ui] Fixed an issue where updating a catalog view caused an infinite loading state.
- Fixed an issue which could cause asset check evaluations emitted from the body of the op to not impact the check status of an asset in the UI.
- Fixed an issue that could cause an asset backfill created by re-executing another backfill from the point of failure to error on the first tick in rare cases.
- Fixed an issue that could cause automation condition evaluations to fail to render in the UI in rare cases.
- [ui] Fixed a regression in the "Cancel Backfill" option for job backfills that have finished queuing runs.
- [ui] Fixed overflow of long runs feed table on backfill page.
- [dagster-dbt] Replaced `validator` with `field_validator` in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks [tintamarre](https://github.com/tintamarre)!)

Documentation

- Updated the "Asset versioning and caching" guide to reflect the current Dagster UI and "Unsynced" labels.
- Removed a few repeated lines in documentation on customizing automation conditions. (Thanks [zero-stroke](https://github.com/zero-stroke)!)
- Fixed example in TableRecord documentation to use the new input format.

Configuration

- [dagster-gcp] Updated Dataproc configuration to the latest version. If necessary, consider pinning your `dagster-gcp` version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/b4d562e04fe571e40037a344b7a9937d
- [dagster-aws][dagster-spark] Updated Spark configuration to the latest version (3.5.5). If necessary, consider pinning your `dagster-aws` and/or `dagster-spark` version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/f358f5a70fea28d5f164aca8da3dee04

Dagster Plus

- [ui] Fixed filtering for multiple tags on list view pages, including Automations.
- [ui] Fixed an issue where the urls generated by catalog filtering would remove all filters if loaded directly.
- [ui] Added a warning on the sign-in page indicating that the sign-in and signup flows will be changing soon.
- [ui] Require confirmation when rolling back a code location to a previous version.

dg & Components (Preview)

- Virtual environment detection settings for projects have changed. Previously, the global settings `use_dg_managed_environment` and `require_local_venv` controlled the environment used when launching project subprocesses. This is now configured at the project level. The `tool.dg.project.python_environment` setting takes a value of either `persistent_uv` or `active`. `persistent_uv` will be used by default in new scaffolded projects and uses a uv-managed `.venv` in the project root. `active` is the default if no `tool.dg.project.python_environment` is set, and just uses the active python environment and opts out of `dg` management of the environment.
- A new base class, `Resolvable`, has been added. This can be used to simplify the process of defining a yaml schema for your components. Instead of manually defining a manual `ResolvedFrom[...]` and `ResolvableModel`, the framework will automatically derive the model schema for you based off of the annotations of your class.
- Python files with Pythonic Components (i.e. defined with `component`) can now contain relative imports.
- The `dg init` command now accepts optional `--workspace-name` and `--project-name` options to allow scaffolding an initial workspace and project via CLI options instead of prompts.
- Added a new `dagster_components.dagster.DefsFolderComponent` that can be used at any level of your `defs/` folder to apply asset attributes to the definitions at or below that level. This was previously named `dagster_components.dagster.DefsModuleComponent`.

New

- Added a new `AutomationCondition.executed_with_tags()` condition that makes it possible to filter for updates from runs with particular tags.
- `AssetCheckEvaluation` can now be yielded from Dagster ops to log an evaluation of an asset check outside of an asset context.
- Added the `kinds` argument to `dagster.AssetOut`, allowing kinds to be specified in `multi_asset`.
- [dagster-dbt] `AssetCheckEvaluations` are now yielded from `ops` leveraging `DbtCliResource.cli(...)` when asset checks are included in the dbt asset lineage.
- [dagster-sling] The `SlingResource.replicate()` method now takes an option `stream` parameter, which allows events to be streamed as the command executes, instead of waiting until it completes (thanks, natpatchara-w!).
- [dagster-graphql] The `DagsterGraphQLClient` now supports an `auth` keyword argument, which is passed to the underlying `RequestsHTTPTransport` constructor.
- [ui] The asset selection syntax input now allows slashes "/" in the freeform search.
- [ui] The backfill pages now show summary information on all tabs for easier backfill monitoring.

Bugfixes

- Fixed issue with `AutomationCondition.newly_requested()` which could cause it to fail when nested within `AutomationCondition.any_deps_match()` or `AutomationCondition.all_deps_match()`.
- Fixed a bug with `AutomationCondition.replace()` that would cause it to not effect `AutomationCondition.since()` conditions.
- Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected: `dagster-airbyte`, `dagster-dlift`, `dagster-dbt`, `dagster-fivetran`, `dagster-looker`, `dagster-powerbi`, `dagster-sigma`, and `dagster-tableau`.
- Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
- [dagster-fivetran] Loading assets for a Fivetran workspace containing incomplete and broken connectors now no longer raises an exception.
- [ui] Fixed the colorblind (no red/green) theme behavior when in dark mode.
- [ui] The Asset > Partitions page no longer displays an error in some cases when creating dynamic partitions.
- [ui] The Launch and Report Events buttons no longer error if you click it immediately after creating a new dynamic partition.

dg & Components (Preview)

- `__pycache__` files are no longer included in the output of `dg list component`. (Thanks [stevenayers](https://github.com/stevenayers)!)
- When resolving the `deps` of an `AssetSpec` from yaml, multi-part asset keys are now correctly parsed. (Thanks [stevenayers](https://github.com/stevenayers)!)
- The entrypoint group for dg projects has been renamed from `dagster.components` to `dagster_dg.library`.
- `dg check yaml` is now run by default before `dg dev` and `dg check defs`.

New

- `async def yield_for_execution` is now supported on `ConfigurableResource`. An `event_loop` argument has been added to context builders to support direct execution.
- `dagster dev` deduplicates stacktraces when code locations fail to load, and will by default truncate them to highlight only user code frames.
- Improved error message experience for resources expecting an env var which was not provided.
- [ui] An updated asset selection syntax is now available in the asset graph, insights, and alerts. The new syntax allows combining logical operators, lineage operators, and attribute filters.
- [dagster-polars] The minimal compatible `deltalake` version has been bumped to `0.25.0`; the `PolarsDeltaIOManager` is now using the `rust` engine for writing DeltaLake tables by default.

Bugfixes

- Fixed a bug with AutomationCondition.replace() that would cause it to not effect `AutomationCondition.since()` conditions.
- Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
- Fixed an issue in Dagster OSS where failed runs of multiple partitions didn't update those partitions as failed in the Dagster UI or trigger failure automation conditions.
- Fixed an issue where `dagster dev` would fail to load code that took more than 45 seconds to import unless the `--use-legacy-code-server-behavior` flag was used.
- [dagster-airbyte] Fixed an issue that caused the group name of assets created using `build_airbyte_assets_definitions` function to error when attempting to modify the default group name.
- [dagster-fivetran] Fixed an issue that caused the group name of assets created using `build_fivetran_assets_definitions` function to error when attempting to modify the default group name.

New

- [ui] The asset overview tab for a partitioned asset now shows metadata and schema of the most recent materialization, not today's partition.
- [ui] In run logs, asset materialization and observation events now show the output partition as well as the asset key.
- [ui] The backfills view has moved to Runs > Backfills and is no longer available on the Overview tab.
- [ui] Pool event information from a run now links to the pool configuration page.
- Added support for passing `tags` to the created `RunRequest` when using `build_sensor_for_freshness_checks()`.
- [dagster-gcp] The `PickledObjectGCSIOManager` now replaces the underlying blob when the same asset is materialized multiple times, instead of deleting and then re-uploading the blob.
- [docs] Added docs covering run-scoped op concurrency.
- [dagster-fivetran] Fivetran connectors fetched in Dagster can now be filtered and selected using the ConnectorSelectorFn.

Bugfixes

- Fixed a bug where if a run was deleted while the re-execution system was determining whether the run should be retried an error was raised. Now, if a run is deleted while the re-execution system is determining whether the run should be retried, the run will not be retried.
- [ui] Fixed an issue where assets with automation conditions wouldn't show the jobs/sensors/schedules targeting them.
- [ui] Steps properly transition to failed in the Run gantt chart when resource initialization fails.

New

- Added links from pool info in run event logs to the respective pool configuration pages.
- Added queued run information on the pool info page, even if the pool granularity is set to `run`.
- [ui] Added information about asset partitions that fail to materialize due to run cancellations to the asset partition detail page.
- [ui] Added two new themes for users with reduced sensitivity to red and green light.
- [ui] Added Not Diamond icon for asset `kinds` tag. (Thanks [dragos-pop](https://github.com/dragos-pop)!)
- [ui] Added Weaviate icon for asset `kinds` tag. (Thanks [jjyeo](https://github.com/jjyeo)!)
- [ui] Made Alerts page visible to users with Viewer roles.
- [dagster-postgres] Removed the cap on `PostgresEventLogStorage` `QueuePool` by setting [`max_overflow`](https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.create_engine.params.max_overflow) to `-1`. (Thanks [axelwas](https://github.com/axelwas)!)

Bugfixes

- Fixed a bug where a sensor emitting multiple `RunRequests` with the same run key within a single tick would cause two runs with the same key to be executed. Now, only the first run will be executed. (Thanks [Gw1p](https://github.com/Gw1p)!)
- Fixed a bug where run step selections were not affecting which pools limit a given run.
- Fixed an issue where seeding the random number generator during code import or when initializing a resource could cause every step to write to the same stdout or stderr key.
- [ui] Fixed an issue where certain jobs weren't showing the assets they targeted.
- Asset backfills will now move into a `CANCELED` state instead of a `FAILURE` state when not every requested partition has been marked as materialized or failed by the backfill.
- [dagster-dbt] Fixed a bug breaking `packaged_project_dir` since supporting `profiles_dir` in `DbtProject`.
- Fixed an issue with `DbIOManagers` being unable to process subclasses of handled types.
- [ui] Preserved asset selection filters when navigating folders in the asset catalog.
- [ui] Corrected PostgreSQL SVG icon for asset `kinds` tag. (Thanks [dragos-pop](https://github.com/dragos-pop)!)
- [ui] Fixed an issue that caused Markdown with code blocks in languages not supported for syntax highlighting to crash the page.
- Fixed an issue where asset backfills included failed partitions in the in-progress list in logging output.

Documentation

- Fixed broken image links in quickstart examples. (Thanks [stevenayers](https://github.com/stevenayers)!)
- [dagster-dbt] Made several fixes to the "Using dbt with Dagster" page. (Thanks [jjyeo](https://github.com/jjyeo)!)
- Fixed broken link in defining-assets.md. (Thanks [Exlll](https://github.com/Exlll)!)
- Fixed link in CONTRIBUTING.md leading to a 404. (Thanks [Exlll](https://github.com/Exlll)!)
- Fixed typo in managing-code-locations-with-definitions.md. (Thanks [kgeis](https://github.com/kgeis)!)
- Fixed typo in asset-versioning-and-caching.md. (Thanks [petrusek](https://github.com/petrusek)!)

Dagster Plus

- [ui] Enabled setting long-running job alerts in minutes instead of hours.
- [dagster-insights] Fix links to branch deployments in the deployment list UI.
- [dagster-insights] Adjusted the way batching runs from the `create_snowflake_insights_asset_and_schedule` sensor using the `schedule_batch_size_hrs` parameter works to yield a single partition range run instead of individual runs per partition.

New

- Turned on run-blocking for concurrency keys / pools by default. For op granularity, runs are dequeued if there exists at least one op that can execute once the run has started. For run granularity, runs are dequeued if all pools have available slots.
- Performance improvements for backfills of large partition sets.
- The prefix of temporary directories created when running a temporary Dagster instance (as with `dagster dev`) has been changed from `tmp` to `.tmp_dagster_home_`. (Thanks [chazmo03](https://github.com/chazmo03)!)
- Added sanitation checks on valid pool names.
- [dagster-aws] Added sample Terraform modules for Dagster deployment on AWS ECS.
- [dagster-dbt] Added pool support for dbt integrations.
- [dagster-dlt] Added pool support for dlt integrations.
- [dagster-sling] Added pool support for sling integrations.
- [dagster-aws] Added AWS RDSResource. (Thanks [shimon-cherrypick](https://github.com/shimon-cherrypick)!)
- [dagster-mysql] Added MySQLResource. (Thanks [shimon-cherrypick](https://github.com/shimon-cherrypick)!)
- [dagster-azure] Added Azure Blob Storage Resource. (Thanks [shimon-cherrypick](https://github.com/shimon-cherrypick)!)
- [ui] Expanding/collapsing groups in the Asset Graph will no longer reset your zoom.
- [ui] Changed the queue criteria dialog to reference pools instead of concurrency keys.
- [ui] The Instance Backfills page is being removed in the upcoming March 6 release in favor of the new Runs > Backfills view.
- [ui] When re-executing a run that is part of a backfill that has completed, Dagster UI notifies you that the re-execution will not update the backfill stats.
- [ui] The backfill actions menu now includes "Re-execute" and "Re-execute from failure", which create new backfills targeting the same partitions, and the partitions which failed to materialize, respectively.
- [ui] The latest asset check evaluation is shown in the Evaluation History tab, and `AssetCheckResult` descriptions are rendered in the table making it easier to publish a summary of check evaluation.
- [ui] The Materialize button appears more quickly on asset pages in the Dagster UI.
- [ui] The queue details modal for a run no longer closes as new runs arrive and links to the correct concurrency page.

Bugfixes

- Fixed an issue where if two separate code locations defined the same asset key with an automation condition, duplicate runs could be created by Declarative Automation.
- Fixed the `psycopg2.errors.UndefinedColumn` database error when trying to set a concurrency key without first having run `dagster instance migrate`.
- Fixed an issue where Declarative Automation sensors in code locations that included source assets referencing assets with automation conditions in other code locations would sometimes cause duplicate runs to be created.
- Fixed a bug in the enforcement of global op concurrency limits.
- Fixed an issue where when using `dagster dev`, some changes were not reflected in the UI after pressing the "Reload Definitions" button.
- Fixed the issue where a resource initialization error within a sensor definition test incorrectly recommended using `build_schedule_context` instead of `build_sensor_context`.
- Fixed migration issue where `dagster instance migrate` was failing for instances with non-empty concurrency limits tables.
- [ui] Fixed an issue where a "Message: Cannot return null for non-nullable field PartitionKeys.partitionKeys." error was raised in the launchpad for jobs with unpartitioned assets.
- [ui] Fixed concurrency link escaping in the `View queue criteria` dialog.
- [ui] Fixed an issue where the deployment switcher can become permanently "unset" when navigating from Org Settings back to a deployment.
- [ui] Fixed an issue with the traversal operators on the asset graph (`asset++`) not including assets connected to the target asset by paths of varying distance.

Dagster Plus

- A setting is available in agent configuration `direct_snapshot_uploads` (`directSnapshotUploads` in helm) which opts in to a new more efficient scheme for how definitions are handled during code location updates.
- Introduced new test utilities `event_log` and `dagster_event` in `dagster-cloud-test-infra` to facilitate the creation of test data with sensible defaults for EventLogEntry and DagsterEvent objects.
- [bigquery-insights][bugfix] Support querying for insights from the configured `execution_project` if defined.
- [bigquery-insights][bugfix] When `execution_project` is defined in the dbt profile, fall back to fetching the dataset from the dbt profile's `project` if the dataset cannot be found in the `execution_project`.