Dagster

New

- Several updates to the Dagit run timeline view: your time window preference will now be preserved locally, there is a clearer “Now” label to delineate the current time, and upcoming scheduled ticks will no longer be batched with existing runs.
- [dagster-k8s] `ingress.labels` is now available in the Helm chart. Any provided labels are appended to the default labels on each object (`helm.sh/chart`, `app.kubernetes.io/version`, and `app.kubernetes.io/managed-by`).
- [dagster-dbt] Added support for two types of dbt nodes: metrics, and ephemeral models.
- When constructing a `GraphDefinition` manually, InputMapping and OutputMapping objects should be directly constructed.

Bugfixes

- [dagster-snowflake] Pandas is no longer imported when `dagster_snowflake` is imported. Instead, it’s only imported when using functionality inside `dagster-snowflake` that depends on pandas.
- Recent changes to `run_status_sensors` caused sensors that only monitored jobs in external repositories to also monitor all jobs in the current repository. This has been fixed.
- Fixed an issue where "unhashable type" errors could be spawned from sensor executions.
- [dagit] Clicking between assets in different repositories from asset groups and asset jobs now works as expected.
- [dagit] The DAG rendering of composite ops with more than one input/output mapping has been fixed.
- [dagit] Selecting a source asset in Dagit no longer produces a GraphQL error
- [dagit] Viewing “Related Assets” for an asset run now shows the full set of assets included in the run, regardless of whether they were materialized successfully.
- [dagit] The Asset Lineage view has been simplified and lets you know if the view is being clipped and more distant upstream/downstream assets exist.
- Fixed erroneous experimental warnings being thrown when using `with_resources` alongside source assets.

Breaking Changes

- [dagit] The launchpad tab is no longer shown for Asset jobs. Asset jobs can be launched via the “Materialize All” button shown on the Overview tab. To provide optional configuration, hold shift when clicking “Materialize”.
- The arguments to `InputMapping` and `OutputMapping` APIs have changed.

Community Contributions

- The `ssh_resource` can now accept configuration from environment variables. Thanks [cbini](https://github.com/cbini)!
- Spelling corrections in `migrations.md`. Thanks [gogi2811](https://github.com/gogi2811)!

New

- [dagit] nbconvert is now installed as an extra in Dagit.
- Multiple assets can be monitored for materialization using the `multi_asset_sensor` (experimental).
- Run status sensors can now monitor jobs in external repositories.
- The `config` argument of `define_asset_job` now works if the job contains partitioned assets.
- When configuring sqlite-based storages in dagster.yaml, you can now point to environment variables.
- When emitting `RunRequests` from sensors, you can now optionally supply an `asset_selection` argument, which accepts a list of `AssetKey`s to materialize from the larger job.
- [dagster-dbt] `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` now support the `exclude` parameter, allowing you to more precisely which resources to load from your dbt project (thanks flvndh!)
- [dagster-k8s] `schedulerName` is now available for all deployments in the Helm chart for users who use a custom Kubernetes scheduler

Bugfixes

- Previously, types for multi-assets would display incorrectly in Dagit when specified. This has been fixed.
- In some circumstances, viewing nested asset paths in Dagit could lead to unexpected empty states. This was due to incorrect slicing of the asset list, and has been fixed.
- Fixed an issue in Dagit where the dialog used to wipe materializations displayed broken text for assets with long paths.
- [dagit] Fixed the Job page to change the latest run tag and the related assets to bucket repository-specific jobs. Previously, runs from jobs with the same name in different repositories would be intermingled.
- Previously, if you launched a backfill for a subset of a multi-asset (e.g. dbt assets), all assets would be executed on each run, instead of just the selected ones. This has been fixed.
- [dagster-dbt] Previously, if you configured a `select` parameter on your `dbt_cli_resource` , this would not get passed into the corresponding invocations of certain `context.resources.dbt.x()` commands. This has been fixed.

New

- Assets can now be materialized to storage conditionally by setting `output_required=False`. If this is set and no result is yielded from the asset, Dagster will not create an asset materialization event, the I/O manager will not be invoked, downstream assets will not be materialized, and asset sensors monitoring the asset will not trigger.
- `JobDefinition.run_request_for_partition` can now be used inside sensors that target multiple jobs (Thanks Metin Senturk!)
- The environment variable `DAGSTER_GRPC_TIMEOUT_SECONDS` now allows for overriding the default timeout for communications between host processes like dagit and the daemon and user code servers.
- Import time for the `dagster` module has been reduced, by approximately 50% in initial measurements.
- `AssetIn` now accepts a `dagster_type` argument, for specifying runtime checks on asset input values.
- [dagit] The column names on the Activity tab of the asset details page no longer reference the legacy term “Pipeline”.
- [dagster-snowflake] The `execute_query` method of the snowflake resource now accepts a `use_pandas_result` argument, which fetches the result of the query as a Pandas dataframe. (Thanks swotai!)
- [dagster-shell] Made the execute and execute_script_file utilities in dagster_shell part of the public API (Thanks Fahad Khan!)
- [dagster-dbt] `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` now support the `exclude` parameter. (Thanks flvndh!)

Bugfixes

- [dagit] Removed the x-frame-options response header from Dagit, allowing the Dagit UI to be rendered in an iframe.
- [fully-featured project example] Fixed the duckdb IO manager so the comment_stories step can load data successfully.
- [dagster-dbt] Previously, if a `select` parameter was configured on the `dbt_cli_resource`, it would not be passed into invocations of `context.resources.dbt.run()` (and other similar commands). This has been fixed.
- [dagster-ge] An incompatibility between `dagster_ge_validation_factory` and dagster 1.0 has been fixed.
- [dagstermill] Previously, updated arguments and properties to `DagstermillExecutionContext` were not exposed. This has since been fixed.

Documentation

- The integrations page on the docs site now has a section for links to community-hosted integrations. The first linked integration is silentsokolov’s Vault integration.

New

- `Failure` now has an `allow_retries` argument, allowing a means to manually bypass retry policies.
- `dagstermill.get_context` and `dagstermill.DagstermillExecutionContext` have been updated to reflect stable dagster-1.0 APIs. `pipeline`/`solid` referencing arguments / properties will be removed in the next major version bump of `dagstermill`.
- `TimeWindowPartitionsDefinition` now exposes a `get_cron_schedule` method.

Bugfixes

- In some situations where an asset was materialized and that asset that depended on a partitioned asset, and that upstream partitioned asset wasn’t part of the run, the partition-related methods of InputContext returned incorrect values or failed erroneously. This was fixed.
- Schedules and sensors with the same names but in different repositories no longer affect each others idempotence checks.
- In some circumstances, reloading a repository in Dagit could lead to an error that would crash the page. This has been fixed.

Community Contributions

- will-holley added an optional `key` argument to GCSFileManager methods to set the GCS blob key, thank you!
- Fix for sensors in [fully featured example](https://docs.dagster.io/guides/dagster/example_project#fully-featured-project), thanks pwachira!

Documentation

- New documentation for getting started with Dagster Cloud, including:
- [Serverless deployment documentation](https://docs.dagster.io/dagster-cloud/getting-started/getting-started-with-serverless-deployment)
- [Hybrid deployment documentation](https://docs.dagster.io/dagster-cloud/getting-started/getting-started-with-hybrid-deployment)

New

- When the workpace is updated, a notification will appear in Dagit, and the Workspace tab will automatically refresh.

Bugfixes

- Restored the correct version mismatch warnings between dagster core and dagster integration libraries
- `Field.__init__` has been typed, which resolves an error that pylance would raise about `default_value`
- Previously, `dagster_type_materializer` and `dagster_type_loader` expected functions to take a context argument from an internal dagster import. We’ve added `DagsterTypeMaterializerContext` and `DagsterTypeLoaderContext` so that functions annotated with these decorators can annotate their arguments properly.
- Previously, a single-output op with a return description would not pick up the description of the return. This has been rectified.

Community Contributions

- Fixed the `dagster_slack` documentation examples. Thanks ssingh13-rms!

Documentation

- New documentation for [Dagster Cloud environment variables](https://docs.dagster.io/dagster-cloud/developing-testing/environment-variables).
- The full list of APIs removed in 1.0 has been added to the [migration guide](https://github.com/dagster-io/dagster/blob/master/MIGRATION.md).

Bugfixes

- Fixed an issue where Dagster libraries would sometimes log warnings about mismatched versions despite having the correct version loaded.

Documentation

- The [Dagster Cloud docs](https://docs.dagster.io/dagster-cloud) now live alongside all the other Dagster docs! Check them out by nagivating to Deployment > Cloud.