Prefect

🎉 Support for access block fields in `prefect.yaml` templating

You can now access fields on blocks used in your `prefect.yaml` files. This enables you to use values stored in blocks to provide dynamic configuration for attributes like your `work_pool_name` and `job_variables`.

Here's what it looks like in action:

yaml
deployments:
- name: test
version: 0.1
tags: []
description: "Example flow"
schedule: {}
entrypoint: "flow.py:example_flow"
parameters: {}
work_pool:
name: "{{ prefect.blocks.json.default-config.value.work_pool }}"
work_queue: "{{ prefect.blocks.json.default-config.value.work_queue }}"


In the above example, we use fields from a `JSON` block to configure which work pool and queue we deploy our flow to. We can update where our flow is deployed to by updating the referenced block without needing to change our `prefect.yaml` at all!

Many thanks to bjarneschroeder for contributing this functionality! Check out this PR for implementation details: https://github.com/PrefectHQ/prefect/pull/10938


👍🏼 Enhancements
- Add the `wait_for_flow_run` method to `PrefectClient` to allow waiting for a flow run to complete — https://github.com/PrefectHQ/prefect/pull/11305
- Add a provisioner for `Modal` push work pools — https://github.com/PrefectHQ/prefect/pull/11665
- Expose the `limit` kwarg in `serve` to increase its visibility — https://github.com/PrefectHQ/prefect/pull/11645
- Add methods supporting modification and suppression of flow run notification policies — https://github.com/PrefectHQ/prefect/pull/11163
- Enhancements to sending and receiving flow run inputs by automatically converting types to `RunInput` subclasses — https://github.com/PrefectHQ/prefect/pull/11636

⛑️ Fixes
- Fix block reference resolution in `prefect.yaml` and support accessing block fields using keypaths — https://github.com/PrefectHQ/prefect/pull/10938
- Avoid rerunning task runs forced to `COMPLETED` state — https://github.com/PrefectHQ/prefect/pull/11385
- Add a new UI setting to customize the served static directory — https://github.com/PrefectHQ/prefect/pull/11648

✏️ Documentation
- Fix retry handler example code in task concept docs — https://github.com/PrefectHQ/prefect/pull/11633
- Fix docstring example in `from_source` — https://github.com/PrefectHQ/prefect/pull/11634
- Add an active incident screenshot to the documentation — https://github.com/PrefectHQ/prefect/pull/11647
- Add clarification on work queues being a feature of hybrid work pools only — https://github.com/PrefectHQ/prefect/pull/11651
- Update interactive workflow guide description and heading — https://github.com/PrefectHQ/prefect/pull/11663
- Add API reference documentation for `wait_for_flow_run` — https://github.com/PrefectHQ/prefect/pull/11668
- Remove duplicate line in `prefect deploy` docs — https://github.com/PrefectHQ/prefect/pull/11644
- Update README to clearly mention running the Python file before starting server — https://github.com/PrefectHQ/prefect/pull/11643
- Fix typo in `Modal` infrastructure documentation — https://github.com/PrefectHQ/prefect/pull/11676

🥇 New Contributors
* N-Demir made their first contribution in https://github.com/PrefectHQ/prefect/pull/11633
* sgbaird made their first contribution in https://github.com/PrefectHQ/prefect/pull/11644
* bjarneschroeder made their first contribution in https://github.com/PrefectHQ/prefect/pull/10938
* Fizzizist made their first contribution in https://github.com/PrefectHQ/prefect/pull/11305
* NeodarZ made their first contribution in https://github.com/PrefectHQ/prefect/pull/11163

What's Changed
Hotfix
* Bugfix: Missing `UI_SERVE_BASE_URL` lead to incorrect asset urls by znicholasbrown in https://github.com/PrefectHQ/prefect/pull/11628


**Full Changelog**: https://github.com/PrefectHQ/prefect/compare/2.14.14...2.14.15

✨ Support for custom prefect.yaml deployment configuration files

You can now specify a `prefect.yaml` deployment configuration file while running `prefect deploy` by using the
`--prefect-file` command line argument. This means that your configuration files can be in any directory
and can follow your own naming conventions. Using this feature provides more flexibility in defining
and managing your deployments.

✅ Toggle Deployment Schedule Status via `prefect.yaml`

You can now toggle your deployment schedules between `active` and `inactive` in your `prefect.yaml` configuration file. This enables you to create deployments with initially _inactive_ schedules, allowing for thorough testing or staged rollouts!

🐍 Support for Python 3.12

You can now install `prefect` using Python 3.12! This support is experimental and will be hardened in future releases.

🧑‍🤝‍🧑 Contributors
A big thanks to brett-koonce, jitvimol, oz-elhassid, Zyntogz, and Andrew-S-Rosen for making their first contributions in this release.

See the [release notes](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-21414) for more details.

This release includes a few very exciting enhancements! 🪄

Access default work pool configurations in an air-gapped environment 🌐
Those who run Prefect server in an environment where arbitrary outbound internet traffic is not allowed were previously unable to retrieve up-to-date default work pool configurations (via the UI or otherwise). You can now access the worker metadata needed to access the corresponding work pool configurations in your server even in such an air-gapped environment. Upon each release of `prefect`, the most recent version of this worker metadata will be embedded in the `prefect` package so that it can be used as a fallback if the outbound call to retrieve the real-time metadata fails.

Introducing conditional task retries for enhanced workflow control 🔁
In this release, we're excited to introduce the ability to conditionally retry tasks by passing in an argument to `retry_condition_fn` in your task decorator, enabling more nuanced and flexible retry mechanisms. This adds a significant level of control and efficiency, particularly in handling complex or unpredictable task outcomes. For more information on usage, check out our [docs](https://docs.prefect.io/latest/concepts/tasks/#custom-retry-behavior)!

Other enhancements and fixes 🛠️
We've also added the ability to open your current Prefect Cloud workspace in the browser from the CLI, display work queue status details via CLI, and much more! See the [release notes](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-21413) for details.

Contributors 🧑‍🤝‍🧑
A big thanks to dominictarro and ConstantinoSchillebeeckx for contributing enhancements to this release, as well as to yifanmai, who made their first contribution.

This release brings some exciting enhancements and fixes!

Enhanced UI Customization 📆

- Now featuring relative time spans like "Past 7 days" for better insight into current and upcoming flow runs across the UI.
- Flow Runs Page: Enhanced with customizable date and time filters. Save custom filters with specific date ranges for tailored views
- Flows Page: Updated with similar date and time filtering capabilities

See a demonstration here!
[](https://www.loom.com/share/95113969257d4cffa48ad13f943f950f?sid=b20bc27c-0dc2-40be-a627-a2148942c427)

Get type-checked input from humans in the loop

Human-in-the-loop flows just got an upgrade. You can now pause or suspend a flow and wait for type-checked input. To get started, declare the structure of the input data using a Pydantic model, and Prefect will render a form dynamically in the UI when a human resumes the flow. Form validation will ensure that the data conforms to your Pydantic model, and your flow will receive tbe input.

<img width="472" alt="image" src="https://github.com/PrefectHQ/prefect/assets/97182/ac743557-e872-4b48-a61e-c74c95e076f0">

Prefect's new `RunInput` class powers this experience. `RunInput` is a subclass of Pydantic's `BaseModel`. Here's an example of a `RunInput` that uses dates, literals, and nested Pydantic models to show you what's possible:

python
class Person(RunInput):
first_name: str
last_name: str
birthday: datetime.date
likes_tofu: bool
age: int = Field(gt=0, lt=150)
shirt_size: Literal[ShirtSize.SMALL, ShirtSize.MEDIUM, ShirtSize.LARGE,
ShirtSize.XLARGE]
shirt_color: Literal["red", "blue", "green"]
preferred_delivery_time: datetime.datetime
shipping_address: ShippingAddress
billing_address: BillingAddress | SameAsShipping = Field(
title="", default_factory=SameAsShipping
)


Check out our [guide on how to create human-in-the-loop flows](https://docs.prefect.io/latest/guides/creating-human-in-the-loop-workflows/) to learn more!

Enhancements and Fixes 🛠️
- Fixed environment variable parsing during deployment and prompt and role assignment in the `ContainerInstanceProvisioner`
- Enhanced dashboard header to adapt to various tag and date input sizes
- Addressed an error encountered when deploying flows with options

See the [release notes](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-21412) for details!

🪄 ✨ Customize resource names when provisioning infrastructure for push work pools

In the past few releases, we've added the ability to provision infrastructure for push work pools via the CLI. This release adds the ability to customize the name of the resources created in your cloud environment when provisioning infrastructure for push work pools so you can follow your organization's naming conventions.

To customize your resource names when provisioning infrastructure for a push work pool, follow the interactive prompts:

bash
? Proceed with infrastructure provisioning with default resource names? [Use arrows to move; enter to select]
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Options: ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ │ Yes, proceed with infrastructure provisioning with default resource names │
│ > │ Customize resource names │
│ │ Do not proceed with infrastructure provisioning │
└────┴───────────────────────────────────────────────────────────────────────────┘
? Please enter a name for the resource group (prefect-aci-push-pool-rg): new-rg
? Please enter a name for the app registration (prefect-aci-push-pool-app): new-app
? Please enter a prefix for the Azure Container Registry (prefect): newregistry
? Please enter a name for the identity (used for ACR access) (prefect-acr-identity): new-identity
? Please enter a name for the ACI credentials block (new-work-pool-push-pool-credentials): new-aci-block
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool new-work-pool will require: │
│ │
│ Updates in subscription: Azure subscription 1 │
│ │
│ - Create a resource group in location: eastus │
│ - Create an app registration in Azure AD: new-app │
│ - Create/use a service principal for app registration │
│ - Generate a secret for app registration │
│ - Create an Azure Container Registry with prefix newregistry │
│ - Create an identity new-identity to allow access to the created registry │
│ - Assign Contributor role to service account │
│ - Create an ACR registry for image hosting │
│ - Create an identity for Azure Container Instance to allow access to the registry │
│ │
│ Updates in Prefect workspace │
│ │
│ - Create Azure Container Instance credentials block: new-aci-block │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
Creating resource group
Resource group 'new-rg' created successfully
Creating app registration
App registration 'new-app' created successfully
Generating secret for app registration
Secret generated for app registration with client ID '03923189-3151-4acd-8d59-76483752cd39'
Creating ACI credentials block
ACI credentials block 'new-aci-block' created in Prefect Cloud
Assigning Contributor role to service account
Service principal created for app ID '25329389-3151-4acd-8d59-71835252cd39'
Contributor role assigned to service principal with object ID '483h4c85-4a8f-4fdb-0394-bd0f0b1202d0'
Creating Azure Container Registry
Registry created
Logged into registry newregistry1702538242q2z2.azurecr.io
Creating identity
Identity 'new-identity' created
Provisioning infrastructure. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Your default Docker build namespace has been set to 'newregistry1702538242q2z2.azurecr.io'.
Use any image name to build and push to this registry by default:

╭─────────────────────────── example_deploy_script.py ───────────────────────────╮
│ from prefect import flow │
│ from prefect.deployments import DeploymentImage │
│ │
│ │
│ flow(log_prints=True) │
│ def my_flow(name: str = "world"): │
│ print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") │
│ │
│ │
│ if __name__ == "__main__": │
│ my_flow.deploy( │
│ name="my-deployment", │
│ work_pool_name="my-work-pool", │
│ image=DeploymentImage( │
│ name="my-image:latest", │
│ platform="linux/amd64", │
│ ) │
│ ) │
╰────────────────────────────────────────────────────────────────────────────────╯
Infrastructure successfully provisioned for 'new-work-pool' work pool!
Created work pool 'new-work-pool'!


Using a push work pool with automatic infrastructure provisioning is a great way to get started with a production-level Prefect set up in minutes! Check out our [push work pool guide](https://docs.prefect.io/latest/guides/deployment/push-work-pools/) for step-by-step instructions on how to get started!

See the following pull requests for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11407
- https://github.com/PrefectHQ/prefect/pull/11381
- https://github.com/PrefectHQ/prefect/pull/11412

📆 🕐 An updated date time input on the workspace dashboard

We've added a new date and time filter to the workspace dashboard that gives greater control over the dashboard. You can now filter by days, hours, and even minutes. You can also specify a specific date and time range to filter by. You can also go backwards and forwards in time using that time window, for example, you can scroll through by hour.

See it in action!
[
](https://www.loom.com/share/ca099d3792d146d08df6fcd506ff9eb2?sid=70797dda-6dc6-4fe6-bf4a-a9df2a0bf230)

See the following pull requests for implementation details:
- https://github.com/PrefectHQ/prefect-ui-library/pull/1937
- https://github.com/PrefectHQ/prefect-design/pull/1048


💥 Enhancements
- Add the ability to publish `KubernetesJob` blocks as work pools — https://github.com/PrefectHQ/prefect/pull/11347
- Add setting to configure a default Docker namespace for image builds — https://github.com/PrefectHQ/prefect/pull/11378
- Add the ability to provision an ECR repository for ECS push work pools — https://github.com/PrefectHQ/prefect/pull/11382
- Add ability to provision an Artifact Registry repository for Cloud Run push work pools — https://github.com/PrefectHQ/prefect/pull/11399
- Add ability to provision an Azure Container Registry for Azure Container Instance push work pools — https://github.com/PrefectHQ/prefect/pull/11387
- Add support for `is_schedule_active` to `flow.deploy` and `flow.serve` — https://github.com/PrefectHQ/prefect/pull/11375
- Allow users to select relative and fixed date ranges to filter the dashboard — https://github.com/PrefectHQ/prefect/pull/11406
- Add support for arbitrary sink types to `prefect.utilities.processutils.stream_text` — https://github.com/PrefectHQ/prefect/pull/11298
- Update the Prefect UI deployments page to add run activity and separate out the deployment and flow names — https://github.com/PrefectHQ/prefect/pull/11394
- Update Prefect UI workspace dashboard filters to use new date range - https://github.com/PrefectHQ/prefect-ui-library/pull/1937

🧰 Fixes
- Fix bug where a pause state reused an existing state ID — https://github.com/PrefectHQ/prefect/pull/11405

🧪 Experimental
- Build out API for creating/reading/deleting flow run inputs — https://github.com/PrefectHQ/prefect/pull/11363
- Integrate flow run input and schema/response mechanics into pause/suspend — https://github.com/PrefectHQ/prefect/pull/11376
- Add typing overloads for pause/suspend methods — https://github.com/PrefectHQ/prefect/pull/11403
- Use bytes for `value` in `create_flow_run_input` — https://github.com/PrefectHQ/prefect/pull/11421
- Validate run input when resuming flow runs — https://github.com/PrefectHQ/prefect/pull/11396
- Run existing deployments via the `Runner` webserver — https://github.com/PrefectHQ/prefect/pull/11333

🧠 Documentation
- Add instructions for automatic infrastructure provisioning to the push work pools guide — https://github.com/PrefectHQ/prefect/pull/11316
- Fix broken links in states concept doc and daemonize guide — https://github.com/PrefectHQ/prefect/pull/11374
- Update agent upgrade guide to include `flow.deploy` and examples — https://github.com/PrefectHQ/prefect/pull/11373
- Update block document names in Moving Data guide — https://github.com/PrefectHQ/prefect/pull/11386
- Rename `Guides` to` How-to Guides` — https://github.com/PrefectHQ/prefect/pull/11388
- Add guide to provision infrastructure for existing push work pools — https://github.com/PrefectHQ/prefect/pull/11365
- Add documentation for required permissions for infrastructure provisioning — https://github.com/PrefectHQ/prefect/pull/11417
- Add docs for managed execution open beta — https://github.com/PrefectHQ/prefect/pull/11397, https://github.com/PrefectHQ/prefect/pull/11426, and https://github.com/PrefectHQ/prefect/pull/11425