Dagster

New

- Added a pin to `protobuf` version 3 due to a backwards incompatible change in the `probobuf` version 4 release.
- [helm] The name of the Dagit deployment can now be overridden in the Dagster Helm chart.
- [dagit] The left navigation now shows jobs as expandable lists grouped by repository. You can opt out of this change using the feature flag in User Settings.
- [dagit] In the left navigation, when a job has more than one schedule or sensor, clicking the schedule/sensor icon will now display a dialog containing the full list of schedules and sensors for that job.
- [dagit] Assets on the runs page are now shown in more scenarios.
- [dagster-dbt] dbt assets now support subsetting! In dagit, you can launch off a dbt command which will only refresh the selected models, and when you’re building jobs using `AssetGroup.build_job()`, you can define selections which select subsets of the loaded dbt project.
- [dagster-dbt] [experimental] The `load_assets_from_dbt_manifest` function now supports an experimental `select` parameter. This allows you to use dbt selection syntax to select from an existing manifest.json file, rather than having Dagster re-compile the project on demand.
- For software-defined assets, `OpExecutionContext` now exposes an `asset_key_for_output` method, which returns the asset key that one of the op’s outputs corresponds too.
- The Backfills tab in Dagit loads much faster when there have been backfills that produced large numbers of runs.
- Added the ability to run the Dagster Daemon as a Python module, by running `python -m dagster.daemon`.
- The `non_argument_deps` parameter for the `asset` and `multi_asset` decorators can now be a set of strings in addition to a set of `AssetKey`.

Bugfixes

- [dagit] In cases where Dagit is unable to make successful WebSocket connections, run logs could become stuck in a loading state. Dagit will now time out on the WebSocket connection attempt after a brief period of time. This allows run logs to fall back to http requests and move past the loading state.
- In version 0.14.16, launching an asset materialization run with source assets would error with an `InvalidSubsetError`. This is now fixed.
- Empty strings are no longer allowed as `AssetKey`s.
- Fixed an issue where schedules built from partitioned job config always ran at midnight, ignoring any hour or minute offset that was specified on the config.
- Fixed an issue where if the scheduler was interrupted and resumed in the middle of running a schedule tick that produced multiple RunRequests, it would show the same run ID multiple times on the list of runs for the schedule tick.
- Fixed an issue where Dagit would raise a GraphQL error when a non-dictionary YAML string was entered into the Launchpad.
- Fixed an issue where Dagster gRPC servers would sometimes raise an exception when loading repositories with many partition sets.
- Fixed an issue where the `snowflake_io_manager` would sometimes raise an error with `pandas` 1.4 or later installed.
- Fixed an issue where re-executing an entire set of dynamic steps together with their upstream step resulted in `DagsterExecutionStepNotFoundError`. This is now fixed.
- [dagit] Added loading indicator for job-scoped partition backfills.
- Fixed an issue that made it impossible to have graph-backed assets with upstream SourceAssets.

Community Contributions

- `AssetIn` can now accept a string that will be coerced to an `AssetKey`. Thanks [aroig](https://github.com/aroig)!
- Runtime type checks improved for some asset-related functions. Thanks [aroig](https://github.com/aroig)!
- Docs grammar fixes. Thanks [dwinston](https://github.com/dwinston)!
- Dataproc ops for `dagster-gcp` now have user-configurable timeout length. Thanks [3cham](https://github.com/3cham)!

New

- `AssetsDefinition.from_graph` now accepts a `partitions_def` argument.
- `asset`-decorated functions can now accept variable keyword arguments.
- Jobs executed in ECS tasks now report the health status of the ECS task
- The CLI command `dagster instance info` now prints the current schema migration state for the configured instance storage.
- [dagster-dbt] You can now configure a `docs_url` on the `dbt_cli_resource`. If this value is set, AssetMaterializations associated with each dbt model will contain a link to the dbt docs for that model.
- [dagster-dbt] You can now configure a `dbt_cloud_host` on the `dbt_cloud_resource`, in the case that your dbt cloud instance is under a custom domain.

Bugfixes

- Fixed a bug where `InputContext.upstream_output` was missing the `asset_key` when it referred to an asset outside the run.
- When specifying a `selection` parameter in `AssetGroup.build_job()`, the generated job would include an incorrect set of assets in certain situations. This has been fixed.
- Previously, a set of database operational exceptions were masked with a `DagsterInstanceSchemaOutdated` exception if the instance storage was not up to date with the latest schema. We no longer wrap these exceptions, allowing the underlying exceptions to bubble up.
- [dagster-airbyte] Fixed issue where successfully completed Airbyte syncs would send a cancellation request on completion. While this did not impact the sync itself, if alerts were set up on that connection, they would get triggered regardless of if the sync was successful or not.
- [dagster-azure] Fixed an issue where the Azure Data Lake Storage `adls2_pickle_io_manager` would sometimes fail to recursively delete a folder when cleaning up an output.
- Previously, if two different jobs with the same name were provided to the same repo, and one was targeted by a sensor/schedule, the job provided by the sensor/schedule would silently overwrite the other job instead of failing. In this release, a warning is fired when this case is hit, which will turn into an error in 0.15.0.
- Dagit will now display workspace errors after reloading all repositories.

Breaking Changes

- Calls to `instance.get_event_records` without an event type filter is now deprecated and will generate a warning. These calls will raise an exception starting in `0.15.0`.

Community Contributions

- `multi_asset` now supports partitioning. Thanks aroig!
- Orphaned process detection now works correctly across a broader set of platforms. Thanks aroig!
- [K8s] Added a new `max_concurrent` field to the `k8s_job_executor` that limits the number of concurrent Ops that will execute per run. Since this executor launches a Kubernetes Job per Op, this also limits the number of concurrent Kuberenetes Jobs. Note that this limit is per run, not global. Thanks kervel!
- [Helm] Added a new `externalConfigmap` field as an alternative to `dagit.workspace.servers` when running the user deployments chart in a separate release. This allows the workspace to be managed outside of the main Helm chart. Thanks peay!
- Removed the pin on `markupsafe<=2.0.1`. Thanks [bollwyvl](https://github.com/dagster-io/dagster/commits?author=bollwyvl)!

New

- Sensors / schedules can now return a list of `RunRequest` objects instead of yielding them.
- Repositories can now contain asset definitions and source assets for the same asset key.
- `OpExecutionContext` (provided as the `context` argument to Ops) now has fields for, `run`, `job_def`, `job_name`, `op_def`, and `op_config`. These replace `pipeline_run`, `pipeline_def`, etc. (though they are still available).
- When a job is partitioned using an hourly, daily, weekly, or monthly partitions definition, `OpExecutionContext` now offers a `partition_time_window` attribute, which returns a tuple of datetime objects that mark the bounds of the partition’s time window.
- `AssetsDefinition.from_graph` now accepts a `partitions_def` argument.
- [dagster-k8s] Removed an unnecessary `dagster-test-connection` pod from the Dagster Helm chart.
- [dagster-k8s] The `k8s_job_executor` now polls the event log on a ~1 second interval (previously 0.1). Performance testing showed that this reduced DB load while not significantly impacting run time.
- [dagit] Removed package pins for `Jinja2` and `nbconvert`.
- [dagit] When viewing a list of Runs, tags with information about schedules, sensors, and backfills are now more visually prominent and are sorted to the front of the list.
- [dagit] The log view on Run pages now includes a button to clear the filter input.
- [dagit] When viewing a list of Runs, you can now hover over a tag to see a menu with an option to copy the tag, and in filtered Run views, an option to add the tag to the filter.
- [dagit] Configuration editors throughout Dagit now display clear indentation guides, and our previous whitespace indicators have been removed.
- [dagit] The Dagit Content-Security-Policy has been moved from a `<meta>` tag to a response header, and several more security and privacy related headers have been added as well.
- [dagit] Assets with multi-component key paths are always shown as `foo/bar` in dagit, rather than appearing as `foo > bar` in some contexts.
- [dagit] The Asset graph now includes a “Reload definitions” button which reloads your repositories.
- [dagit] On all DAGs, you can hold shift on the keyboard to switch from mouse wheel / touch pad zooming to panning. This makes it much easier to scroll horizontally at high speed without click-drag-click-drag-click-drag.
- [dagit] a `--log-level` flag is now available in the dagit cli for controlling the uvicorn log level.
- [dagster-dbt] The `load_assets_from_dbt_project()` and `load_assets_from_dbt_manifest()` utilities now have a `use_build_command` parameter. If this flag is set, when materializing your dbt assets, Dagster will use the `dbt build` command instead of `dbt run`. Any tests run during this process will be represented with AssetObservation events attached to the relevant assets. For more information on `dbt build`, see the [dbt docs](https://docs.getdbt.com/reference/commands/build).
- [dagster-dbt] If a dbt project successfully runs some models and then fails, AssetMaterializations will now be generated for the successful models.
- [dagster-snowflake] The new Snowflake IO manager, which you can create using `build_snowflake_io_manager` offers a way to store assets and op outputs in Snowflake. The `PandasSnowflakeTypeHandler` stores Pandas `DataFrame`s in Snowflake.
- [helm] `dagit.logLevel` has been added to values.yaml to access the newly added dagit --log-level cli option.

Bugfixes

- Fixed incorrect text in the error message that’s triggered when building a job and an asset can’t be found that corresponds to one of the asset dependencies.
- An error is no longer raised when an op/job/graph/other definition has an empty docstring.
- Fixed a bug where pipelines could not be executed if `toposort<=1.6` was installed.
- [dagit] Fixed an issue in global search where rendering and navigation broke when results included objects of different types but with identical names.
- [dagit] server errors regarding websocket send after close no longer occur.
- [dagit] Fixed an issue where software-defined assets could be rendered improperly when the dagster and dagit versions were out of sync.

Community Contributions

- [dagster-aws] `PickledObjectS3IOManager` now uses `list_objects` to check the access permission. Thanks trevenrawr!

Breaking Changes

- [dagster-dbt] The asset definitions produced by the experimental `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` functions now include the schemas of the dbt models in their asset keys. To revert to the old behavior: `dbt_assets = load_assets_from_dbt_project(..., node_info_to_asset_key=lambda node_info: AssetKey(node_info["name"])`.

Experimental

- The `TableSchema` API is no longer experimental.

Documentation

- Docs site now has a new design!
- Concepts pages now have links to code snippets in our examples that use those concepts.

New

- When viewing a config schema in the Dagit launchpad, default values are now shown. Hover over an underlined key in the schema view to see the default value for that key.
- dagster, dagit, and all extension libraries (dagster-\*) now contain py.typed files. This exposes them as typed libraries to static type checking tools like mypy. If your project is using mypy or another type checker, this may surface new type errors. For mypy, to restore the previous state and treat dagster or an extension library as untyped (i.e. ignore Dagster’s type annotations), add the following to your configuration file:


[mypy-dagster] (or e.g. mypy-dagster-dbt)
follow_imports = "skip"


- Op retries now surface the underlying exception in Dagit.
- Made some internal changes to how we store schema migrations across our different storage implementations.
- `build_output_context` now accepts an `asset_key` argument.
- They key argument to the SourceAsset constructor now accepts values that are strings or sequences of strings and coerces them to AssetKeys.
- You can now use the + operator to add two AssetGroups together, which forms an AssetGroup that contains a union of the assets in the operands.
- `AssetGroup.from_package_module`, `from_modules`, `from_package_name`, and `from_current_module` now accept an `extra_source_assets` argument that includes a set of source assets into the group in addition to the source assets scraped from modules.
- AssetsDefinition and AssetGroup now both expose a `to_source_assets` method that return SourceAsset versions of their assets, which can be used as source assets for downstream AssetGroups.
- Repositories can now include multiple AssetGroups.
- The new prefixed method on AssetGroup returns a new AssetGroup where a given prefix is prepended to the asset key of every asset in the group.
- Dagster now has a BoolMetadataValue representing boolean-type metadata. Specifying True or False values in metadata will automatically be casted to the boolean type.
- Tags on schedules can now be expressed as nested JSON dictionaries, instead of requiring that all tag values are strings.
- If an exception is raised during an op, Dagster will now always run the failure hooks for that op. Before, certain system exceptions would prevent failure hooks from being run.
- `mapping_key` can now be provided as an argument to `build_op_context`/`build_solid_context`. Doing so will allow the use of `OpExecutionContext.get_mapping_key()`.

Bugfixes

- [dagit] Previously, when viewing a list of an asset’s materializations from a specified date/time, a banner would always indicate that it was a historical view. This banner is no longer shown when viewing the most recent materialization.
- [dagit] Special cron strings like daily were treated as invalid when converting to human-readable strings. These are now handled correctly.
- The selection argument to `AssetGroup.build_job` now uses `>` instead of `.` for delimiting the components within asset keys, which is consistent with how selection works in Dagit.
- [postgres] passwords and usernames are now correctly url quoted when forming a connection string. Previously spaces were replaced with `+`.
- Fixed an issue where the `celery_docker_executor` would sometimes fail to execute with a JSON deserialization error when using Dagster resources that write to stdout.
- [dagster-k8s] Fixed an issue where the Helm chart failed to work when the user code deployment subchart was used in a different namespace than the main dagster Helm chart, due to missing configmaps.
- [dagster-airbyte] When a Dagster run is terminated while executing an Airbyte sync operation, the corresponding Airbyte sync will also be terminated.
- [dagster-dbt] Log output from dbt cli commands will no longer have distracting color-formatting characters.
- [dagit] Fixed issue where multi_assets would not show correct asset dependency information.
- Fixed an issue with the sensor daemon, where the sensor would sometimes enter a race condition and overwrite the sensor status.

Community Contributions

- [dagster-graphql] The Python DagsterGraphQLClient now supports terminating in-progress runs using `client.terminate_run(run_id)`. Thanks [Javier162380](https://github.com/Javier162380)!

Experimental

- Added an experimental view of the Partitions page / Backfill page, gated behind a feature flag in Dagit.

New

- [dagster-k8s] You can now specify resource requests and limits to the K8sRunLauncher when using the Dagster helm chart, that will apply to all runs. Before, you could only set resource configuration by tagging individual jobs. For example, you can set this config in your `values.yaml` file:


runLauncher:
type: K8sRunLauncher
config:
k8sRunLauncher:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi


- [dagster-k8s] Specifying `includeConfigInLaunchedRuns: true` in a user code deployment will now launch runs using the same namespace and service account as the user code deployment.
- The `asset` decorator now accepts an `op_tags` argument, which allows e.g. providing k8s resource requirements on the op that computes the asset.
- Added CLI output to `dagster api grpc-health-check` (previously it just returned via exit codes)
- [dagster-aws] The `emr_pyspark_step_launcher` now supports dynamic orchestration, `RetryPolicy`s defined on ops, and re-execution from failure. For failed steps, the stack trace of the root error will now be available in the event logs, as will logs generated with `context.log.info`.
- Partition sets and can now return a nested dictionary in the `tags_fn_for_partition` function, instead of requiring that the dictionary have string keys and values.
- [dagit] It is now possible to perform bulk re-execution of runs from the Runs page. Failed runs can be re-executed from failure.
- [dagit] Table headers are now sticky on Runs and Assets lists.
- [dagit] Keyboard shortcuts may now be disabled from User Settings. This allows users with certain keyboard layouts (e.g. QWERTZ) to inadvertently avoid triggering unwanted shortcuts.
- [dagit] Dagit no longer continues making some queries in the background, improving performance when many browser tabs are open.
- [dagit] On the asset graph, you can now filter for multi-component asset keys in the search bar and see the “kind” tags displayed on assets with a specified compute_kind.
- [dagit] Repositories are now displayed in a stable order each time you launch Dagster.

Bugfixes

- [dagster-k8s] Fixed an issue where the Dagster helm chart sometimes failed to parse container images with numeric tags. Thanks [jrouly](https://github.com/jrouly)!
- [dagster-aws] The `EcsRunLauncher` now registers new task definitions if the task’s execution role or task role changes.
- Dagster now correctly includes `setuptools` as a runtime dependency.
- `In` can now accept `asset_partitions` without crashing.
- [dagit] Fixed a bug in the Launchpad, where default configuration failed to load.
- [dagit] Global search now truncates the displayed list of results, which should improve rendering performance.
- [dagit] When entering an invalid search filter on Runs, the user will now see an appropriate error message instead of a spinner and an alert about a GraphQL error.

Documentation

- Added documentation for partitioned assets
- [dagster-aws] Fixed example code of a job using `secretsmanager_resource`.

Bugfixes

- Fixed an issue where the Launchpad in Dagit sometimes incorrectly launched in an empty state.