Dagster

Bugfixes

- Fixes a bug in `dagster project from-example` that was preventing it from downloading examples correctly.

New

- The `--name` argument is now optional when running `dagster project from-example`.
- An asset key can now be directly specified via the asset decorator: `asset(key=...)`.
- `AssetKey` now has a `with_prefix` method.
- Significant performance improvements when using `AutoMaterializePolicy`s with large numbers of partitions.
- `dagster instance migrate` now prints information about changes to the instance database schema.
- The [`dagster-cloud-agent` helm chart](https://artifacthub.io/packages/helm/dagster-cloud/dagster-cloud-agent) now supports setting K8s labels on the agent deployment.
- [ui] Step compute logs are shown under “Last Materialization” in the asset sidebar.
- [ui] Truncated asset names now show a tooltip when hovered in the asset graph.
- [ui] The “Propagate changes” button has been removed and replaced with “Materialize Stale and Missing” (which was the “Propagate changes” predecessor).

Bugfixes

- [ui] Fixed an issue that prevented filtering by date on the job-specific runs tab.
- [ui] “F” key with modifiers (alt, ctrl, cmd, shift) no longer toggles the filter menu on pages that support filtering.
- [ui] Fix empty states on Runs table view for individual jobs, to provide links to materialize an asset or launch a run for the specific job, instead of linking to global pages.
- [ui] When a run is launched from the Launchpad editor while an editor hint popover is open, the popover remained on the page even after navigation. This has been fixed.

- [ui] Fixed an issue where clicking on the zoom controls on a DAG view would close the right detail panel for selected nodes.
- [ui] Fixed an issue shift-selecting assets with multi-component asset keys.
- [ui] Fixed an issue with the truncation of the asset stale causes popover.
- When using a `TimeWindowPartitionMapping` with a `start_offset` or `end_offset` specified, requesting the downstream partitions of a given upstream partition would yield incorrect results. This has been fixed.
- When using `AutoMaterializePolicy`s with observable source assets, in rare cases, a second run could be launched in response to the same version being observed twice. This has been fixed.

- When passing in `hook_defs` to `define_asset_job`, if any of those hooks had required resource keys, a missing resource error would surface when the hook was executed. This has been fixed.
- Fixed a typo in a documentation URL in `dagster-duckdb-polars` tests. The URL now works correctly.

Experimental

- [dagster-dbt] Added methods to `DbtManifest` to fetch asset keys of sources and models: `DbtManifest.get_asset_key_for_model`, `DbtManifest.get_asset_key_for_source`. These methods are utilities for defining python assets as dependencies of dbt assets via `asset(key=manifest.get_asset_key_for_model(...)`.
- [dagster-dbt] The use of the `state_path` parameter with `DbtManifestAssetSelection` has been deprecated, and will be removed in the next minor release.
- Added experimental support for limiting global op/asset concurrency across runs.

Dependencies

- Upper bound on the `grpcio` package (for `dagster`) has been removed.

Breaking Changes

- Legacy methods of `PartitionMapping` have been removed. Defining custom partition mappings has been unsupported since 1.1.7.

Community Contributions

- [dagster-airbyte] Added the ability to specify asset groups to `build_airbyte_assets`. Thanks [guy-rvvup](https://github.com/guy-rvvup)!

Documentation

- For Dagster Cloud Serverless users, we’ve added our static IP addresses to [the Serverless docs](https://docs.dagster.io/dagster-cloud/deployment/serverless#whitelisting-dagsters-ip-addresses).

New

- Assets with lazy auto-materialize policies are no longer auto-materialized if they are missing but don’t need to be materialized in order to help downstream assets meet their freshness policies.
- [ui] The descriptions of auto-materialize policies in the UI now include their skip conditions along with their materialization conditions.
- [dagster-dbt] Customized asset keys can now be specified for nodes in the dbt project, using `meta.dagster.asset_key`. This field takes in a list of strings that are used as the components of the generated `AssetKey`.

yaml
version: 2

models:
- name: users
config:
meta:
dagster:
asset_key: ["my", "custom", "asset_key"]


- [dagster-dbt] Customized groups can now be specified for models in the dbt project, using `meta.dagster.group`. This field takes in a string that is used as the Dagster group for the generated software-defined asset corresponding to the dbt model.

yaml
version: 2

models:
- name: users
config:
meta:
dagster:
group: "my_group"


Bugfixes

- Fixed an issue where the `dagster-msteams` and `dagster-mlflow` packages could be installed with incompatible versions of the `dagster` package due to a missing pin.
- Fixed an issue where the `dagster-daemon run` command sometimes kept code server subprocesses open longer than it needed to, making the process use more memory.
- Previously, when using `observable_source_asset`s with AutoMaterializePolicies, it was possible for downstream assets to get “stuck”, not getting materialized when other upstream assets changed, or for multiple down materializations to be kicked off in response to the same version being observed multiple times. This has been fixed.
- Fixed a case where the materialization count for partitioned assets could be wrong.
- Fixed an error which arose when trying to request resources within run failure sensors.
- [dagster-wandb] Fixed handling for multi-dimensional partitions. Thanks chrishiste

Experimental

- [dagster-dbt] improvements to `dbt_assets`
- `project_dir` and `target_path` in `DbtCliTask` are converted from type `str` to type `pathlib.Path`.
- In the case that dbt logs are not emitted as json, the log will still be redirected to be printed in the Dagster compute logs, under `stdout`.

Documentation

- Fixed a typo in dagster_aws S3 resources. Thanks akan72
- Fixed a typo in link on the Dagster Instance page. Thanks PeterJCLaw

New

- [dagster-dbt] By default, freshness policies and auto materialize policies on dbt assets can now be specified using the `dagster` field under `+meta` configuration. The following are equivalent:

Before:

yaml
version: 2

models:
- name: users
config:
dagster_freshness_policy:
maximum_lag_minutes: 60
cron_schedule: "0 9 * * *"
dagster_auto_materialize_policy:
type: "lazy"


After:

yaml
version: 2

models:
- name: users
config:
meta:
dagster:
freshness_policy:
maximum_lag_minutes: 60
cron_schedule: "0 9 * * *"
auto_materialize_policy:
type: "lazy"


- Added support for Pythonic Config classes to the `configured` API, which makes reusing op and asset definitions easier:

python
class GreetingConfig(Config):
message: str

op
def greeting_op(config: GreetingConfig):
print(config.message)

class HelloConfig(Config):
name: str

configured(greeting_op)
def hello_op(config: HelloConfig):
return GreetingConfig(message=f"Hello, {config.name}!")


- Added `AssetExecutionContext` to replace `OpExecutionContext` as the context object passed in to `asset` functions.
- `TimeWindowPartitionMapping` now contains an `allow_nonexistent_upstream_partitions` argument that, when set to `True`, allows a downstream partition subset to have nonexistent upstream parents.
- Unpinned the `alembic` dependency in the `dagster` package.
- [ui] A new “Assets” tab is available from the Overview page.
- [ui] The Backfills table now includes links to the assets that were targeted by the backfill.

Bugfixes

- Dagster is now compatible with a breaking change introduced in `croniter==1.4.0`. Users of earlier versions of Dagster can pin `croniter<1.4`.
- Fixed an issue introduced in 1.3.8 which prevented resources from being bound to sensors when the specified job required late-bound resources.
- Fixed an issue which prevented specifying resource requirements on a `run_failure_sensor`.
- Fixed an issue where the asset reconciliation sensor failed with a “invalid upstream partitions” error when evaluating time partitions definitions with different start times.
- [dagster-k8s] Fixed an issue where annotations are not included in the Dagster Helm chart for the pod that is created when configuring the Helm chart to run database migrations.
- [ui] Fixed an issue with filtering runs by created date on the Runs page.
- [ui] The “upstream partitions missing” warning no longer appears in the asset backfill dialog if the upstream partitioned asset is a source asset.
- [dagster-dbt] Fixed an issue where asset dependencies for dbt models with ephemeral models in between them would sometimes be improperly rendered.

Community Contributions

- Added support for setting resources in asset and multi_asset sensors. Thanks [plaflamme](https://github.com/plaflamme)!
- Fixed an issue where `py.typed` was missing in the `dagster-graphql` package. Thanks [Tanguy-LeFloch](https://github.com/Tanguy-LeFloch)!

Experimental

- Evaluation history for `AutoMaterializePolicy`s will now be cleared after 1 week.
- [dagster-dbt] Several improvements to `dbt_assets`:
- `profile` and `target` can now be customized on the `DbtCli` resource.
- If a `partial_parse.msgpack` is detected in the target directory of your dbt project, it is now copied into the target directories created by `DbtCli` to take advantage of [partial parsing](https://docs.getdbt.com/reference/parsing).
- The metadata of assets generated by `dbt_assets` can now be customized by overriding `DbtManifest.node_info_to_metadata`.
- Execution duration of dbt models is now added as default metadata to `AssetMaterialization`s.

Documentation

- Added [a new tutorial section](https://docs.dagster.io/tutorial/connecting-to-external-services) about using resources.

Dagster Cloud

- Fixed an issue where overriding the container name of a code server pod using `serverK8sConfig.containerConfig.name` did not actually change the container name.

Dagster Cloud

- Fixed an issue in the `1.3.8` release where the Dagster Cloud agent would sometimes fail to start up with an import error.

New

- Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
- `define_asset_job` now accepts a `hooks` argument.
- Added support for `sqlalchemy==2.x`
- [ui] The Runs page has been revamped with better filtering support
- [ui] The automaterialize policy page for SDA’s using the experimental AutomaterializePolicy feature now indicates time periods where no materializations happened due to no materialization conditions being met
- [dagster-k8s] The Dagster Helm chart now includes an `additionalInstanceConfig` key that allows you to supply additional configuration to the [Dagster instance](https://docs.dagster.io/deployment/dagster-instance#dagster-instance).
- [dagster-aws] The `EcsRunLauncher` now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.
- [dagster-aws] The `EcsRunLauncher` now includes a `run_ecs_tags` config key that lets you configure tags on the launched ECS task for each run.

Bugfixes

- When a sensor had yield statement and also returned a `SkipReason`, the `SkipReason` would be ignored. This has been fixed.
- [dagster-cloud] Fixed a bug in the docker user code launcher that was preventing code location containers from being properly cleaned up.
- Fixed an issue where the Dagster UI would sometimes raise a ``RuntimeError: dictionary changed size during iteration` exception while code servers were being reloaded.
- Fixed an issue where the Dagster daemon reloaded your code server every 60 seconds when using the new experimental `dagster code-server start` CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.
- Fixed a GraphQL error which would occur when loading the default config for jobs without config.
- [dagster-dbt] Fixed an error which would arise when trying to load assets from a DBT Cloud instance using the Pythonic-style resource.

Community Contributions

- Added the ability to specify metadata on asset jobs, by adding the `metadata` parameter to `define_asset_job` (Thanks **[Elliot2718](https://github.com/Elliot2718)!)**
- [dagster-databricks] Connected databricks stdout to local stdout, to be handled by the compute log manager (Thanks **[loerinczy](https://github.com/loerinczy)!)**
- [dagster-census] Fixed `poll_sync_run` to handle the “preparing” status from the Census API (Thanks **[ldnicolasmay](https://github.com/ldnicolasmay)!)**

Experimental

- `observable_source_asset`-decorated functions can now return a `DataVersionsByPartition` to record versions for partitions.
- `dbt_assets`
- `DbtCliTask`'s created by invoking `DbtCli.cli(...)` now have a method `.is_successful()`, which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.
- Descriptions of assets generated by `dbt_assets` can now be customized by overriding `DbtManifest.node_info_to_description`.
- IO Managers can now be configured on `dbt_assets`.

Documentation

- New guide on using Dagster to manage machine learning pipelines

Dagster Cloud

- Added support for streaming upload of compute logs to Dagster Cloud
- The ECS agent now supports setting `server_ecs_tags` and `run_ecs_tags` that apply to each service or task created by the agent. See [the docs](https://docs.dagster.io/dagster-cloud/deployment/agents/amazon-ecs/configuration-reference#amazon-ecs-agent-configuration-reference) for more information.
- Fixed run filtering for calls to `instance.get_run_partition_data` in Dagster Cloud.