Dagster-cloud

New

- [K8s] The agent now monitors active runs. Failures in the underlying Kubernetes Job (e.g. an out of memory error) will now be reported in Dagit.

Bugfixes

- Fixed an issue where the favicon didn’t update to reflect success, pending, or failure status when looking at a job’s run page.

New

- Individual `context.log` messages which appear in the Dagster event log will now be truncated after 50,000 characters. The full contents of these messages remain available in the compute logs tab. For large logs, we recommend logging straight to stdout or stderr rather than using `context.log`.

Bugfixes

- Added a missing IAM permission to the ECS Agent Cloudformation template that was preventing the ECS agent from being able to terminate runs.

Documentation

- Added a new [guide](https://docs.dagster.cloud/guides/adding-code) to the Dagster Cloud docs covering how to add and update code.

Bugfixes

- Sensors that have a default status can now be manually started. Previously, this would fail with an invariant exception.

New

- Added a button to test out Dagster Cloud with a sample code location when you go to the Workspace tab on an empty Deployment.
- Added quick links to log in to your organizations from https://dagster.cloud (must be signed in)
- [K8s] Added an `imagePullGracePeriod` field to the helm chart that tells the agent how long to allow errors while pulling an image before failing. For example:

yaml
values.yaml
workspace:
imagePullGracePeriod: 60


- [K8s] The agent container can now run as a non-root user. The `docker.io/dagster/dagster-cloud-agent` image includes a dagster user with ID `1001` that can be assumed by setting `podSecurityContext` in your Helm values:

yaml
values.yaml
podSecurityContext:
runAsUser: 1001


Bugfixes

- Fixed an issue where run status sensors sometimes failed to trigger.
- [ECS] Fixed an issue where the Cloudformation template for setting up an ECS agent would sometimes fail to spin up tasks with an AWS Secretsmanager permission error.
- Fixed an issue where the agent sometimes left old user code servers running after the server failed to start up.
- Fixed an issue where the agent sometimes stopped sending heartbeats while it was in the middle of starting up a new user code server.
- Fixed an issue where an erroring code location would not show as updating when redeploying.

Documentation

- Added a new guide to configuring the Kubernetes agent (https://docs.dagster.cloud/agents/kubernetes/configuring).
- Updated documentation to show adding new code locations through Dagit.

New

- The Dagster Cloud workspace page now allows creating, deleting, and updating code locations from Cloud Dagit in addition to via the CLI.
- The ECS agent can now override the `secrets_tag` parameter to None, which will cause it to not look for any secrets to be included in the tasks that the agent creates. This can be useful in situations where the agent does not have permissions to query AWS Secretsmanager
- Added a `dagit_url` property to the DagsterInstance in Dagster Cloud that can be used to reference the Dagster Cloud Dagit URL within ops and sensors.
- Introduced a path to run the local Dagster Cloud agent ephemerally without specifying a `dagster.yaml`, by using CLI arguments.
- Added the ability to configure [Python logging](https://docs.dagster.io/concepts/logging/python-logging#python-logging) in the Kubernetes agent helm chart. For example:


pythonLogs:
The names of python loggers that will be captured as Dagster logs
managedPythonLoggers:
- foo_logger
The log level for the instance. Logs emitted below this severity will be ignored.
One of [NOTSET, DEBUG, INFO, WARNING, WARN, ERROR, FATAL, CRITICAL]
pythonLogLevel: INFO
Python log handlers that will be applied to all Dagster logs
dagsterHandlerConfig:
handlers:
myHandler:
class: logging.FileHandler
filename: "/logs/my_dagster_logs.log"
mode: "a"


Bugfixes

- Added a missing AWS secretsmanager permission to the example CloudFormation template for creating an ECS agent.
- Improved dagster-cloud CLI web authentication error messages on the client and server, including troubleshooting steps and instructions on alternative token authentication.
- Fixed an issue where deleting a deployment from Dagit would sometimes fail.

Documentation

- Added a setup guide for SAML SSO using PingOne.
- Added a documentation page detailing switching, creating, and deleting deployments.

New

- Secrets in the ECS Agent: When using Dagster Cloud in ECS, you can specify a list of AWS Secrets Manager ARNs to include in all tasks that the agent spins up. Any secrets that are tagged with the key “dagster” in AWS Secrets Manager (or a custom key that you specify) will also be included in all tasks. You can customize the secrets in your ECS Agent in your `dagster.yaml` file as follows:


user_code_launcher:
module: dagster_cloud.workspace.ecs
class: EcsUserCodeLauncher
config:
cluster: your-cluster-name
subnets:
- your-subnet-name
service_discovery_namespace_id: your-service-discovery-namespace-id
execution_role_arn: your-execution-role-arn
log_group: your-log-group
secrets_tag: "my-tag-name"
secrets:- "arn:aws:secretsmanager:us-east-1:1234567890:secret:MY_SECRET"`


- Added support in Dagster Cloud for disabling compute logs, which display the stdout/stderr output from your Dagster jobs within Dagit. When using the Dagster Cloud helm chart, by setting `Values.computeLogs.enabled` to false, users can prevent compute logs from being forwarded into Dagster Cloud. Logging configured separately from Dagster on Kubernetes will continue to work, but won’t be viewable in the Dagster Cloud UI.

Bugfixes

- Fixed an issue where omitting the Dagster Cloud agent endpoint when installing the Kubernetes agent using the helm chart would sometimes cause the agent to fail to start. The agent endpoint is no longer a required field on the helm chart.
- Fixed an issue with dagster-cloud CLI web authentication where users who did not have an available user token could not be authenticated.

Documentation

- Added a “Customizing your agent” section to the docs, including documentation on how to disable compute logging when manually authoring an agent’s `dagster.yaml`.