**Major Changes since 1.4.0 (core) / 0.20.0 (libraries)**
Core
- **Improved ergonomics for execution dependencies in assets** - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager system for handling data between assets. I/O manager workflows will not be affected.
- `AssetDep` type allows you to specify upstream dependencies with partition mappings when using the `deps` parameter of `asset` and `AssetSpec`.
- `MaterializeResult` can be optionally returned from an asset to report metadata about the asset when the asset handles any storage requirements within the function body and does not use an I/O manager.
- `AssetSpec` has been added as a new way to declare the assets produced by `multi_asset`. When using `AssetSpec`, the multi_asset does not need to return any values to be stored by the I/O manager. Instead, the multi_asset should handle any storage requirements in the body of the function.
- **Asset checks (experimental)** - You can now define, execute, and monitor data quality checks in Dagster [[docs](https://docs.dagster.io/concepts/assets/asset-checks)].
- The `asset_check` decorator, as well as the `check_specs` argument to `asset` and `multi_asset` enable defining asset checks.
- Materializing assets from the UI will default to executing their asset checks. You can also execute individual checks.
- When viewing an asset in the asset graph or the asset details page, you can see whether its checks have passed, failed, or haven’t run successfully.
- **Auto materialize customization (experimental)** - `AutoMaterializePolicies` can now be customized [[docs](https://docs.dagster.io/concepts/assets/asset-auto-execution#auto-materialize-policies)].
- All policies are composed of a set of `AutoMaterializeRule`s which determine if an asset should be materialized or skipped.
- To modify the default behavior, rules can be added to or removed from a policy to change the conditions under which assets will be materialized.
dagster-pipes
- Dagster pipes is a new library that implements a protocol for launching compute into external execution environments and consuming streaming logs and Dagster metadata from those environments. See https://github.com/dagster-io/dagster/discussions/16319 for more details on the motivation and vision behind Pipes.
- Out-the-box integrations
- Clients: local subprocess, Docker containers, Kubernetes, and Databricks
- `PipesSubprocessClient`, `PipesDocketClient`, `PipesK8sClient`, `PipesDatabricksClient`
- Transport: Unix pipes, Filesystem, s3, dbfs
- Languages: Python
- Dagster pipes is composable with existing launching infrastructure via `open_pipes_session`. One can augment existing invocations rather than replacing them wholesale.
**Since 1.4.17 (core) / 0.20.17 (libraries)**
New
- [ui] Global Asset Graph performance improvement - the first time you load the graph it will be cached to disk and any subsequent load of the graph should load instantly.
Bugfixes
- Fixed a bug where deleted runs could retain instance-wide op concurrency slots.
Breaking Changes
- `AssetExecutionContext` is now a subclass of `OpExecutionContext`, not a type alias. The code
python
def my_helper_function(context: AssetExecutionContext):
...
op
def my_op(context: OpExecutionContext):
my_helper_function(context)
will cause type checking errors. To migrate, update type hints to respect the new subclassing.
- `AssetExecutionContext` cannot be used as the type annotation for `op`s run in `jobs`. To migrate, update the type hint in `op` to `OpExecutionContext`. `op`s that are used in `graph_assets` may still use the `AssetExecutionContext` type hint.
python
old
op
def my_op(context: AssetExecutionContext):
...
correct
op
def my_op(context: OpExecutionContext):
...
- [ui] We have removed the option to launch an asset backfill as a single run. To achieve this behavior, add `backfill_policy=BackfillPolicy.single_run()` to your assets.
Community Contributions
- `has_dynamic_partition` implementation has been optimized. Thanks edvardlindelof!
- [dagster-airbyte] Added an optional `stream_to_asset_map` argument to `build_airbyte_assets` to support the Airbyte prefix setting with special characters. Thanks chollinger93!
- [dagster-k8s] Moved “labels” to a lower precedence. Thanks jrouly!
- [dagster-k8s] Improved handling of failed jobs. Thanks Milias!
- [dagster-databricks] Fixed an issue where `DatabricksPysparkStepLauncher` fails to get logs when `job_run` doesn’t have `cluster_id` at root level. Thanks PadenZach!
- Docs type fix from sethusabarish, thank you!
Documentation
- Our Partitions documentation has gotten a facelift! We’ve split the original page into several smaller pages, as follows:
- [Partitions](https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions) - An overview of what a partition is, benefits, and how to use it
- [Partitioning assets](https://docs.dagster.io/concepts/partitions-schedules-sensors/partitioning-assets) - Details about partitioning assets
- [Partitioning ops](https://docs.dagster.io/concepts/partitions-schedules-sensors/partitioning-ops) - Details about partitioning ops
- [Testing partitions](https://docs.dagster.io/concepts/partitions-schedules-sensors/testing-partitions) - As described
Dagster Cloud
- **New dagster-insights sub-module** - We have released an experimental `dagster_cloud.dagster_insights` module that contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an api. Dagster Cloud Insights is a soon-to-be released feature that shows improves visibility into usage and cost metrics such as run duration and Snowflake credits in the Cloud UI.