Dagster

Latest version: v1.8.7

Safety actively analyzes 663899 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 3 of 49

1.7.12

Bugfixes

- [ui] fixes behavior issues with jobs and asset pages introduced in 1.7.11

1.7.11

New

- [ui] Improved performance for loading assets that are part of big asset graphs.
- [ui] Improved performance for loading job backfills that have thousands of partitions
- [ui] The code location page can now be filtered by status
- [agent] K8s and ECS agent main loop writes a sentinel file that can be used for liveness checks.
- [agent][experimental] ECS CloudFormation template with private IP addresses using NAT Gateways, security groups, IAM role separation, tighter permissions requirements, and improved documentation.
- Ephemeral asset jobs are now supported in run status sensors (thanks [the4thamigo-uk](https://github.com/the4thamigo-uk))!

Bugfixes

- In `AssetsDefinition` construction, enforce single key per output name
- Fixed a bug where freshness checks on assets with both observations and materializations would incorrectly miss a materialization if there’s no observation with `dagster/last_updated_timestamp`.
- Fixed a bug with anomaly detection freshness checks where “not enough records” result would cause the sensor to crash loop.
- Fixed a bug that could cause errors in the Asset Daemon if an asset using `AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron()` rule gained a new dependency with a different PartitionsDefinition.
- [ui] Fixed an issue that caused the backfill page not to be scrollable.
- [ui] Fixed an issue where filtering by partition on the Runs page wouldn’t work if fetching all of your partitions timed out.
- [dagster-dlt] Fixed bug with dlt integration in which partitioned assets would change the file name when using the filesystem destination.
- [ui] Fixed an issue where an erroring code location would cause multiple toast popups.
- Allow a string to be provided for `source_key_prefix` arg of `load_assets_from_modules`. (thanks [drjlin](https://github.com/drjlin))!
- Added a missing debug level log message when loading partitions with polars (thanks [Daniel Gafni](https://github.com/danielgafni))!
- Set postgres timeout via statement, which improves storage-layer compatibility with Amazon RDS (thanks [james lewis](https://github.com/jameslewisfaculty))!
- In DBT integration, quote the table identifiers to handle cases where table names require quotes due to special characters. (thanks [alex launi](https://github.com/lamalex))!
- remove deprecated param usage in dagster-wandb integration (thanks [chris histe](https://github.com/chrishiste))!
- Add missing QUEUED state to DatabricksRunLifeCycleState (thanks [gabor ratky](https://github.com/gaborratky-db))!
- Fixed a bug with dbt-cloud integration subsetting implementation (thanks [ivan tsarev](https://github.com/mudravrik))!

Breaking Changes

- [dagster-airflow] `load_assets_from_airflow_dag` no longer allows multiple tasks to materialize the same asset.

Documentation

- Added type-hinting to backfills example
- Added syntax highlighting to some examples (thanks [Niko](https://github.com/nikomancy))!
- Fixed broken link (thanks [federico caselli](https://github.com/caselit))!

Dagster Plus

- The `dagster-cloud ci init` CLI will now use the `--deployment` argument as the base deployment when creating a branch deployment. This base deployment will be used for Change Tracking.
- The BigQuery dbt insights wrapper `dbt_with_bigquery_insights` now respects CLI arguments for profile configuration and also selects location / dataset from the profile when available.
- [experimental feature] Fixes a recent regression where the UI errored upon attempting to create an insights metric alert.

1.7.10

New

- Performance improvements when rendering the asset graph while runs are in progress.
- A new API `build_freshness_checks_for_dbt_assets` which allows users to parameterize freshness checks entirely within dbt. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dbt-dagster-dbt.
- Asset search results now display compute and storage kind icons.
- Asset jobs where the underlying assets have multiple backfill policies will no longer fail at definition time. Instead, the backfill policy for the job will use the minimum `max_partitions_per_run` from the job’s constituent assets.
- [dagstermill] `asset_tags` can now be specified when building dagstermill assets
- [dagster-embedded-elt] Custom asset tags can be applied to Sling assets via the `DagsterSlingTranslator`
- [dagster-embedded-elt] dlt assets now automatically have `dagster/storage_kind` tags attached

Bugfixes

- `tags` passed to `outs` in `graph_multi_asset` now get correctly propagated to the resulting assets.
- [ui] Fixed an issue in the where when multiple runs were started at the same time to materialize the same asset, the most recent one was not always shown as in progress in the asset graph in the Dagster UI.
- The “newly updated” auto-materialize rule will now respond to either new observations or materializations for observable assets.
- `build_metadata_bounds_checks` now no longer errors when targeting metadata keys that have special characters.

Documentation

- The [Schedule concept docs](https://docs.dagster.io/concepts/automation/schedules) got a revamp! Specifically, we:
- Updated the Schedule concept page to be a “jumping off” point for all-things scheduling, including a high-level look at how schedules work, their benefits, and what you need to know before diving in
- Added some basic how-to guides for [automating assets](https://docs.dagster.io/concepts/automation/schedules/automating-assets-schedules-jobs) and [ops](https://docs.dagster.io/concepts/automation/schedules/automating-ops-schedules-jobs) using schedules
- Added a [reference of schedule-focused examples](https://docs.dagster.io/concepts/automation/schedules/examples)
- Added dedicated guides for common schedule uses, including creating p[artitioned schedules](https://docs.dagster.io/concepts/automation/schedules/partitioned-schedules), [customizing executing timezones](https://docs.dagster.io/concepts/automation/schedules/customizing-executing-timezones), [testing](https://docs.dagster.io/concepts/automation/schedules/testing), and [troubleshooting](https://docs.dagster.io/concepts/automation/schedules/troubleshooting)

Dagster Plus

- [experimental] The backfill daemon can now store logs and display them in the UI for increased visibility into the daemon’s behavior. Please contact Dagster Labs if you are interested in piloting this experimental feature.
- Added a `--read-only` flag to the `dagster-cloud ci branch-deployment` CLI command, which returns the current branch deployment name for the current code repository branch without update the status of the branch deployment.

1.7.9

New

- Dagster will now display a “storage kind” tag on assets in the UI, similar to the existing compute kind. To set storage kind for an asset, set its `dagster/storage_kind` tag.
- You can now set retry policy on dbt assets, to enable coarse-grained retries with delay and jitter. For fine-grained partial retries, we still recommend invoking `dbt retry` within a try/except block to avoid unnecessary, duplicate work.
- `AssetExecutionContext` now exposes a `has_partition_key_range` property.
- The `owners`, `metadata`, `tags`, and `deps` properties on `AssetSpec` are no longer `Optional`. The `AssetSpec` constructor still accepts `None` values, which are coerced to empty collections of the relevant type.
- The `docker_executor` and `k8s_job_executor` now consider at most 1000 events at a time when loading events from the current run to determine which steps should be launched. This value can be tuned by setting the `DAGSTER_EXECUTOR_POP_EVENTS_LIMIT` environment variable in the run process.
- Added a `dagster/retry_on_asset_or_op_failure` tag that can be added to jobs to override run retry behavior for runs of specific jobs. See [the docs](https://docs.dagster.io/deployment/run-retries#combining-op-and-run-retries) for more information.
- Improved the sensor produced by `build_sensor_for_freshness_checks` to describe when/why it skips evaluating freshness checks.
- A new “Runs” tab on the backfill details page allows you to see list and timeline views of the runs launched by the backfill.
- [dagster-dbt] dbt will now attach relation identifier metadata to asset materializations to indicate where the built model is materialized to.
- [dagster-graphql] The GraphQL Python client will now include the HTTP error code in the exception when a query fails. Thanks [yuvalgimmunai](https://github.com/yuvalgimmunai)!

Bugfixes

- Fixed sensor logging behavior with the `multi_asset_sensor`.
- `ScheduleDefinition` now properly supports being passed a `RunConfig` object.
- When an asset function returns a `MaterializeResult`, but the function has no type annotation, previously, the IO manager would still be invoked with a `None` value. Now, the IO manager is not invoked.
- The `AssetSpec` constructor now raises an error if an invalid owner string is passed to it.
- When using the `graph_multi_asset` decorator, the `code_version` property on `AssetOut`s passed in used to be ignored. Now, they no longer are.
- [dagster-deltalake] Fixed GcsConfig import error and type error for partitioned assets (Thanks [thmswt](https://github.com/thmswt))
- The asset graph and asset catalog now show the materialization status of External assets (when manually reported) rather than showing “Never observed”

Documentation

- The External Assets REST APIs now have their own [reference page](https://docs.dagster.io/apidocs/external-assets-rest)
- Added details, updated copy, and improved formatting to External Assets REST APIs

Dagster Plus

- The ability to set a custom base deployment when creating a branch deployment has been enabled for all organizations.
- When a code location fails to deploy, the Kubernetes agent now includes additional any warning messages from the underlying replicaset in the failure message to aid with troubleshooting.
- Serverless deployments now support using a requirements.txt with [hashes](https://pip.pypa.io/en/stable/topics/secure-installs/#secure-installs).
- Fixed an issue where the `dagster-cloud job launch` command did not support specifying asset keys with prefixes in the `--asset-key` argument.
- [catalog UI] Catalog search now allows filtering by type, i.e. `group:`, `code location:`, `tag:`, `owner:`.
- New dagster+ accounts will now start with two default alert policies; one to alert if the default free credit budget for your plan is exceeded, and one to alert if a single run goes over 24 hours. These alerts will be sent as emails to the email with which the account was initially created.

1.7.8

New

- Backfills created via GQL can have a custom title and description.
- `Definitions` now has a `get_all_asset_specs` method, which allows iterating over properties of the defined assets
- [ui] In filter dropdowns, it’s now possible to submit before all the suggestions have been loaded (thanks [bmalehorn](https://github.com/bmalehorn)!)
- [ui] Performance improvements when loading the Dagster UI for asset graphs with thousands of partition keys.
- [dagster-dbt] Dbt asset checks now emit execution duration and the number of failing rows as metadata
- [dagster-embedded-elt] Added support for partitioning in dlt assets (thanks [edsoncezar16](https://github.com/edsoncezar16)!)
- [dagster-embedded-elt] Added ability to set custom metadata on dlt assets (thanks [edsoncezar16](https://github.com/edsoncezar16)!)
- [dagster-graphql] Added a `terminate_runs` method to the Python GraphQL Client. (thanks [baumann-t](https://github.com/baumann-t)!)
- [dagster-polars] dagster-polars IO managers now emit dagster/row_count metadata (thanks [danielgafni](https://github.com/danielgafni)!)
- [dagster-dbt] `DbtCliInvocation` now has a `.get_error()` method that can be useful when using `dbt.cli(..., raise_on_error=False)`.

Bugfixes

- Fix a bug with legacy `DynamicPartitionsDefinition` (using `partitions_fn`) that caused a crash during job backfills.
- [ui] On the asset graph, filtering to one or more code locations via the Filter dropdown now works as expected.
- [ui] On the asset overview page, viewing an asset with no definition in a loaded code location no longer renders a clipped empty state.

Experimental

- The new `build_metadata_bounds_checks` API creates asset checks which verify that numeric metadata values on asset materializations fall within min or max values. See the [documentation](https://docs.dagster.io/_apidocs/asset-checks#dagster.build_metadata_bounds_checks) for more information.

Documentation

- Added details and links to the [Schedules and Sensors API documentation](https://docs.dagster.io/_apidocs/schedules-sensors)
- Removed leftover mention of Dagster Cloud from the [Dagster+ Hybrid architecture documentation](https://docs.dagster.io/dagster-plus/deployment/hybrid)

Dagster Plus

- Fixed an incompatibility between `build_sensor_for_freshness_checks` and Dagster Plus. This API should now work when used with Dagster Plus.
- [ui] Billing / usage charts no longer appear black-on-black in Dagster’s dark mode.
- [ui] The asset catalog is now available for teams plans.
- [ui] Fixed a bug where the alert policy editor would misinterpret the threshold on a long-running job alert.
- [kubernetes] Added a `dagsterCloudAgent.additionalPodSpecConfig` to the Kubernetes agent Helm chart allowing arbitrary pod configuration to be applied to the agent pod.
- [ECS] Fixed an issue where the ECS agent would sometimes raise a “Too many concurrent attempts to create a new revision of the specified family” exception when using agent replicas.

1.7.7

New

- [ui] Command clicking on nodes in the asset lineage tab will now open them in a separate tab. Same with external asset links in the asset graph.
- Added support for setting a custom job namespace in user code deployments. (thanks [tmatthews0020](https://github.com/tmatthews0020)!)
- Removed warnings due to use of `datetime.utcfromtimestamp` (thanks [dbrtly](https://github.com/dbrtly)!)
- Custom smtp user can now be used for e-mail alerts (thanks [edsoncezar16](https://github.com/edsoncezar16)!)
- [dagster-dbt] Added support for `dbt-core==1.8.*`.
- [dagster-embedded-elt] Failed dlt pipelines are now accurately reflected on the asset materialization (thanks [edsoncezar16](https://github.com/edsoncezar16)!)

Bugfixes

- Fixed spurious errors in logs due to module shadowing.
- Fixed an issue in the Backfill Daemon where if the assets to be materialized had different `BackfillPolicy`s, each asset would get materialized in its own run, rather than grouping assets together into single run.
- Fixed an issue that could cause the Asset Daemon to lose information in its cursor about an asset if that asset’s code location was temporarily unavailable.
- [dagster-dbt] Mitigated issues with cli length limits by only listing specific dbt tests as needed when the tests aren’t included via indirect selection, rather than listing all tests.

Documentation

- Markdoc tags can now be used in place of MDX components (thanks [nikomancy](https://github.com/nikomancy))

Page 3 of 49

Links

Releases

Has known vulnerabilities

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.