Dagster

New

- When Dagster loads an event from the event log of a type that it doesn’t recognize (for example, because it was created by a newer version of Dagster) it will now return a placeholder event rather than raising an exception.
- AssetsDefinition.from_graph() now accepts a group_name parameter. All assets created by from_graph are assigned to this group.
- You can define an asset from an op via a new utility method `AssetsDefinition.from_op`. Dagster will infer asset inputs and outputs from the ins/outs defined on the `op` in the same way as `graphs`.
- A default executor definition can be defined on a repository using the `default_executor_def` argument. The default executor definition will be used for all op/asset jobs that don’t explicitly define their own executor.
- `JobDefinition.run_request_for_partition` now accepts a `tags` argument (Thanks jburnich!)
- In Dagit, the graph canvas now has a dotted background to help it stand out from the reset of the UI.
- `multi_asset` now accepts a resource_defs argument. The provided resources can be either used on the context, or satisfy the io manager requirements of the outs on the asset.
- In Dagit, show execution timezone on cron strings, and use 12-hour or 24-hour time format depending on the user’s locale.
- In Dagit, when viewing a run and selecting a specific step in the Gantt chart, the compute log selection state will now update to that step as well.
- `define_asset_job` and `to_job` now can now accept a `partitions_def` argument and a `config` argument at the same time, as long as the value for the `config` argument is a hardcoded config dictionary (not a `PartitionedConfig` or `ConfigMapping`)

Bugfixes

- Fixed an issue where entering a string in the launchpad that is valid YAML but invalid JSON would render incorrectly in Dagit.
- Fixed an issue where steps using the `k8s_job_executor` and `docker_executor` would sometimes return the same event lines twice in the command-line output for the step.
- Fixed type annotations on the `op` decorator (Thanks Milos Tomic!)
- Fixed an issue where job backfills were not displayed correctly on the Partition view in Dagit.
- `UnresolvedAssetJobDefinition` now supports the `run_request_for_partition` method.
- Fixed an issue in Dagit where the Instance Overview page would briefly flash a loading state while loading fresh data.

Breaking Changes

- Runs that were executed in newer versions of Dagster may produce errors when their event logs are loaded in older versions of Dagit, due to new event types that were recently added. Going forward, Dagit has been made more resilient to handling new events.

Deprecations

- Updated deprecation warnings to clarify that the deprecated metadata APIs will be removed in 0.16.0, not 0.15.0.

Experimental

- If two assets are in the same group and the upstream asset has a multi-segment asset key, the downstream asset doesn’t need to specify the full asset key when declaring its dependency on the upstream asset - just the last segment.

Documentation

- Added dedicated sections for op, graph, and job Concept docs in the sidenav
- Moved graph documentation from the jobs docs into its [own page](https://docs.dagster.io/concepts/ops-jobs-graphs/graphs)
- Added documentation for assigning [asset groups](https://docs.dagster.io/concepts/assets/software-defined-assets#grouping-assets) and viewing them in Dagit
- Added apidoc for `AssetOut` and `AssetIn`
- Fixed a typo on the Run Configuration concept page (Thanks Wenshuai Hou!)
- Updated screenshots in the software-defined assets tutorial to match the new Dagit UI
- Fixed a typo in the **Defining an asset** section of the software-defined assets tutorial (Thanks Daniel Kim!)

Major Changes

- Software-defined assets are now marked fully stable and are ready for prime time - we recommend using them whenever your goal using Dagster is to build and maintain data assets.
- You can now organize software defined assets into groups by providing a group_name on your asset definition. These assets will be grouped together in Dagit.
- Software-defined assets now accept configuration, similar to ops. E.g.


from dagster import asset

asset(config_schema={"iterations": int})
def my_asset(context):
for i in range(context.op_config["iterations"]):
...


- Asset definitions can now be created from graphs via `AssetsDefinition.from_graph`:


graph(out={"asset_one": GraphOut(), "asset_two": GraphOut()})
def my_graph(input_asset):
...

graph_asset = AssetsDefinition.from_graph(my_graph)


- `execute_in_process` and `GraphDefinition.to_job` now both accept an `input_values` argument, so you can pass arbitrary Python objects to the root inputs of your graphs and jobs.
- Ops that return Outputs and DynamicOutputs now work well with Python type annotations. You no longer need to sacrifice static type checking just because you want to include metadata on an output. E.g.


from dagster import Output, op

op
def my_op() -> Output[int]:
return Output(5, metadata={"a": "b"})


- You can now automatically re-execute runs from failure. This is analogous to op-level retries, except at the job level.
- You can now supply arbitrary structured metadata on jobs, which will be displayed in Dagit.
- The partitions and backfills pages in Dagit have been redesigned to be faster and show the status of all partitions, instead of just the last 30 or so.
- The left navigation pane in Dagit is now grouped by repository, which makes it easier to work with when you have large numbers of jobs, especially when jobs in different repositories have the same name.
- The Asset Details page for a software-defined asset now includes a Lineage tab, which makes it easy to see all the assets that are upstream or downstream of an asset.

Breaking Changes and Deprecations

Software-defined assets

This release marks the official transition of software-defined assets from experimental to stable. We made some final changes to incorporate feedback and make the APIs as consistent as possible:

- Support for adding tags to asset materializations, which was previously marked as experimental, has been removed.
- Some of the properties of the previously-experimental AssetsDefinition class have been renamed. group_names is now group_names_by_key, asset_keys_by_input_name is now keys_by_input_name, and asset_keys_by_output_name is now keys_by_output_name, asset_key is now key, and asset_keys is now keys.
- Removes previously experimental IO manager `fs_asset_io_manager` in favor of merging its functionality with `fs_io_manager`. `fs_io_manager` is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key. Similarly, removed `adls2_pickle_asset_io_manager`, `gcs_pickle_asset_io_manager` , and `s3_pickle_asset_io_manager`. Instead, `adls2_pickle_io_manager`, `gcs_pickle_io_manager`, and `s3_pickle_io_manager` now support software-defined assets.
- _(deprecation)_ The namespace argument on the `asset` decorator and AssetIn has been deprecated. Users should use key_prefix instead.
- _(deprecation)_ AssetGroup has been deprecated. Users should instead place assets directly on repositories, optionally attaching resources using with_resources. Asset jobs should be defined using `define_asset_job` (replacing `AssetGroup.build_job`), and arbitrary sets of assets can be materialized using the standalone function materialize (replacing `AssetGroup.materialize`).
- _(deprecation)_ The `outs` property of the previously-experimental `multi_asset` decorator now prefers a dictionary whose values are `AssetOut` objects instead of a dictionary whose values are `Out` objects. The latter still works, but is deprecated.
- The previously-experimental property on `OpExecutionContext` called `output_asset_partition_key` is now deprecated in favor of `asset_partition_key_for_output`

Event records

- The `get_event_records` method on DagsterInstance now requires a non-None argument `event_records_filter`. Passing a `None` value for the `event_records_filter` argument will now raise an exception where previously it generated a deprecation warning.
- Removed methods `events_for_asset_key` and `get_asset_events`, which have been deprecated since 0.12.0.

Extension libraries

- [dagster-dbt] (breaks previously-experimental API) When using the load_assets_from_dbt_project or load_assets_from_dbt_manifest , the AssetKeys generated for dbt sources are now the union of the source name and the table name, and the AssetKeys generated for models are now the union of the configured schema name for a given model (if any), and the model name. To revert to the old behavior: `dbt_assets = load_assets_from_dbt_project(..., node_info_to_asset_key=lambda node_info: AssetKey(node_info["name"])`.
- [dagster-k8s] In the Dagster Helm chart, user code deployment configuration (like secrets, configmaps, or volumes) is now automatically included in any runs launched from that code. Previously, this behavior was opt-in. In most cases, this will not be a breaking change, but in less common cases where a user code deployment was running in a different kubernetes namespace or using a different service account, this could result in missing secrets or configmaps in a launched run that previously worked. You can return to the previous behavior where config on the user code deployment was not applied to any runs by setting the includeConfigInLaunchedRuns.enabled field to false for the user code deployment. See the [Kubernetes Deployment docs](https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment) for more details.
- [dagster-snowflake] dagster-snowflake has dropped support for python 3.6. The library it is currently built on, snowflake-connector-python, dropped 3.6 support in their recent 2.7.5 release.

Other

- The `prior_attempts_count` parameter is now removed from step-launching APIs. This parameter was not being used, as the information it held was stored elsewhere in all cases. It can safely be removed from invocations without changing behavior.
- The `FileCache` class has been removed.
- Previously, when schedules/sensors targeted jobs with the same name as other jobs in the repo, the jobs on the sensor/schedule would silently overwrite the other jobs. Now, this will cause an error.

New

- [dagster-aws] Added an `env_vars` field to the EcsRunLauncher that allows you to configure environment variables in the ECS task for launched runs.
- [dagster-k8s] The `env_vars` field on `K8sRunLauncher` and `k8s_job_executor` can now except input of the form ENV_VAR_NAME=ENV_VAR_VALUE, and will set the value of ENV_VAR_NAME to ENV_VAR_VALUE. Previously, it only accepted input of the form ENV_VAR_NAME, and the environment variable had to be available in the pod launching the job.
- [dagster-k8s] setting ‘includeConfigInLaunchedRuns’ on a user code deployment will now also include any image pull secrets from the user code deployment in the pod for the launched runs.

Bugfixes

- A recent change had made it so that, when `IOManager.load_input` was called to load an asset that was not being materialized as part of the run, the provided context would not include the metadata for that asset. `context.upstream_output.metadata` now correctly returns the metadata on the upstream asset.
- Fixed an issue where using generic type aliases introduced in Python 3.9 (like `list[str]`) as the type of an input would raise an exception.
- [dagster-k8s] Fixed an issue where upgrading the Helm chart version without upgrading your user code deployment version would result in an “Received unexpected config entry "scheme" at path root:postgres_db" error.

New

- Metadata can now be added to jobs (via the `metadata` parameter) and viewed in dagit. You can use it to track code owners, link to docs, or add other useful information.
- In the Dagit launchpad, the panel below the config editor now shows more detailed information about the state of the config, including error state and whether the config requires further scaffolding or the removal of extra config.
- FileCache is now marked for deprecation in 0.15.0.
- In Dagit, the asset catalog now shows the last materialization for each asset and links to the latest run.
- Assets can now have a `config_schema`. If you attempt to materialize an asset with a config schema in Dagit, you'll be able to enter the required config via a modal.

Bugfixes

- [helm] Fixed an issue where string floats and integers were not properly templated as image tags.
- [dagster-k8s] Fixed an issue when using the `k8s_job_executor` where ops with long names sometimes failed to create a pod due to a validation error with the label names automatically generated by Dagster.
- [dagster-aws] Fixed an issue where ECS tasks with large container contexts would sometimes fail to launch because their request to the ECS RunTask API was too large.

Breaking Changes

- `fs_asset_io_manager` has been removed in favor of merging its functionality with `fs_io_manager`. `fs_io_manager` is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key.

Community Contributions

- Fixed a bug that broke the `k8s_job_executor`’s `max_conccurent` configuration. Thanks fahadkh!
- Fixed a bug that caused the `fs_io_manager` to incorrectly handle assets associated with upstream assets. Thanks aroig!

Documentation

- [helm] Add documentation for code server image pull secrets in the main chart.
- The Dagster README has been revamped with documentation and community links.

New

- Added a pin to `protobuf` version 3 due to a backwards incompatible change in the `probobuf` version 4 release.
- [helm] The name of the Dagit deployment can now be overridden in the Dagster Helm chart.
- [dagit] The left navigation now shows jobs as expandable lists grouped by repository. You can opt out of this change using the feature flag in User Settings.
- [dagit] In the left navigation, when a job has more than one schedule or sensor, clicking the schedule/sensor icon will now display a dialog containing the full list of schedules and sensors for that job.
- [dagit] Assets on the runs page are now shown in more scenarios.
- [dagster-dbt] dbt assets now support subsetting! In dagit, you can launch off a dbt command which will only refresh the selected models, and when you’re building jobs using `AssetGroup.build_job()`, you can define selections which select subsets of the loaded dbt project.
- [dagster-dbt] [experimental] The `load_assets_from_dbt_manifest` function now supports an experimental `select` parameter. This allows you to use dbt selection syntax to select from an existing manifest.json file, rather than having Dagster re-compile the project on demand.
- For software-defined assets, `OpExecutionContext` now exposes an `asset_key_for_output` method, which returns the asset key that one of the op’s outputs corresponds too.
- The Backfills tab in Dagit loads much faster when there have been backfills that produced large numbers of runs.
- Added the ability to run the Dagster Daemon as a Python module, by running `python -m dagster.daemon`.
- The `non_argument_deps` parameter for the `asset` and `multi_asset` decorators can now be a set of strings in addition to a set of `AssetKey`.

Bugfixes

- [dagit] In cases where Dagit is unable to make successful WebSocket connections, run logs could become stuck in a loading state. Dagit will now time out on the WebSocket connection attempt after a brief period of time. This allows run logs to fall back to http requests and move past the loading state.
- In version 0.14.16, launching an asset materialization run with source assets would error with an `InvalidSubsetError`. This is now fixed.
- Empty strings are no longer allowed as `AssetKey`s.
- Fixed an issue where schedules built from partitioned job config always ran at midnight, ignoring any hour or minute offset that was specified on the config.
- Fixed an issue where if the scheduler was interrupted and resumed in the middle of running a schedule tick that produced multiple RunRequests, it would show the same run ID multiple times on the list of runs for the schedule tick.
- Fixed an issue where Dagit would raise a GraphQL error when a non-dictionary YAML string was entered into the Launchpad.
- Fixed an issue where Dagster gRPC servers would sometimes raise an exception when loading repositories with many partition sets.
- Fixed an issue where the `snowflake_io_manager` would sometimes raise an error with `pandas` 1.4 or later installed.
- Fixed an issue where re-executing an entire set of dynamic steps together with their upstream step resulted in `DagsterExecutionStepNotFoundError`. This is now fixed.
- [dagit] Added loading indicator for job-scoped partition backfills.
- Fixed an issue that made it impossible to have graph-backed assets with upstream SourceAssets.

Community Contributions

- `AssetIn` can now accept a string that will be coerced to an `AssetKey`. Thanks [aroig](https://github.com/aroig)!
- Runtime type checks improved for some asset-related functions. Thanks [aroig](https://github.com/aroig)!
- Docs grammar fixes. Thanks [dwinston](https://github.com/dwinston)!
- Dataproc ops for `dagster-gcp` now have user-configurable timeout length. Thanks [3cham](https://github.com/3cham)!

New

- `AssetsDefinition.from_graph` now accepts a `partitions_def` argument.
- `asset`-decorated functions can now accept variable keyword arguments.
- Jobs executed in ECS tasks now report the health status of the ECS task
- The CLI command `dagster instance info` now prints the current schema migration state for the configured instance storage.
- [dagster-dbt] You can now configure a `docs_url` on the `dbt_cli_resource`. If this value is set, AssetMaterializations associated with each dbt model will contain a link to the dbt docs for that model.
- [dagster-dbt] You can now configure a `dbt_cloud_host` on the `dbt_cloud_resource`, in the case that your dbt cloud instance is under a custom domain.

Bugfixes

- Fixed a bug where `InputContext.upstream_output` was missing the `asset_key` when it referred to an asset outside the run.
- When specifying a `selection` parameter in `AssetGroup.build_job()`, the generated job would include an incorrect set of assets in certain situations. This has been fixed.
- Previously, a set of database operational exceptions were masked with a `DagsterInstanceSchemaOutdated` exception if the instance storage was not up to date with the latest schema. We no longer wrap these exceptions, allowing the underlying exceptions to bubble up.
- [dagster-airbyte] Fixed issue where successfully completed Airbyte syncs would send a cancellation request on completion. While this did not impact the sync itself, if alerts were set up on that connection, they would get triggered regardless of if the sync was successful or not.
- [dagster-azure] Fixed an issue where the Azure Data Lake Storage `adls2_pickle_io_manager` would sometimes fail to recursively delete a folder when cleaning up an output.
- Previously, if two different jobs with the same name were provided to the same repo, and one was targeted by a sensor/schedule, the job provided by the sensor/schedule would silently overwrite the other job instead of failing. In this release, a warning is fired when this case is hit, which will turn into an error in 0.15.0.
- Dagit will now display workspace errors after reloading all repositories.

Breaking Changes

- Calls to `instance.get_event_records` without an event type filter is now deprecated and will generate a warning. These calls will raise an exception starting in `0.15.0`.

Community Contributions

- `multi_asset` now supports partitioning. Thanks aroig!
- Orphaned process detection now works correctly across a broader set of platforms. Thanks aroig!
- [K8s] Added a new `max_concurrent` field to the `k8s_job_executor` that limits the number of concurrent Ops that will execute per run. Since this executor launches a Kubernetes Job per Op, this also limits the number of concurrent Kuberenetes Jobs. Note that this limit is per run, not global. Thanks kervel!
- [Helm] Added a new `externalConfigmap` field as an alternative to `dagit.workspace.servers` when running the user deployments chart in a separate release. This allows the workspace to be managed outside of the main Helm chart. Thanks peay!
- Removed the pin on `markupsafe<=2.0.1`. Thanks [bollwyvl](https://github.com/dagster-io/dagster/commits?author=bollwyvl)!