Prefect

Released on August 6, 2020.

Features

- Support cancellation of active flow runs - [2942](https://github.com/PrefectHQ/prefect/pull/2942)
- Add Webhook storage - [3000](https://github.com/PrefectHQ/prefect/pull/3000)

Enhancements

- Only supply versions when setting `SUBMITTED` and `RUNNING` states - [2730](https://github.com/PrefectHQ/prefect/issues/2730)
- Gracefully recover from version lock errors - [2731](https://github.com/PrefectHQ/prefect/issues/2731)
- Add `--ui-version` server start CLI option to run a specific UI image - [3087](https://github.com/PrefectHQ/prefect/pull/3087)
- Agent querying of flow runs now passes active tenant ID - [3087](https://github.com/PrefectHQ/prefect/pull/3087)
- Ignore calls to flow.register when parsing a flow using file based storage - [3051](https://github.com/PrefectHQ/prefect/issues/3051)

Task Library

- Allow idempotency keys in `FlowRunTask` when using server backend - [3006](https://github.com/PrefectHQ/prefect/issues/3006)
- Require project name in `FlowRunTask` when using server backend - [3006](https://github.com/PrefectHQ/prefect/issues/3006)

Fixes

- Fix use of absolute path in Docker storage on Windows - [3044](https://github.com/PrefectHQ/prefect/pull/3044)
- Determine if checkpointing is enabled from config set in the flow-runner process - [3085](https://github.com/PrefectHQ/prefect/pull/3085)
- Fix `--no-ui` server start CLI option still attempting to pull UI image - [3087](https://github.com/PrefectHQ/prefect/pull/3087)

Deprecations

- Deprecate `execute cloud-flow` CLI command in favor of `execute flow-run` - [3087](https://github.com/PrefectHQ/prefect/pull/3087)
- Deprecate `run server/cloud` CLI commands in favor of `run flow` - [3087](https://github.com/PrefectHQ/prefect/pull/3087)

Breaking Changes

- Move server and UI code out into separate repositories - [3087](https://github.com/PrefectHQ/prefect/pull/3087)
- Project names are now required when managing flows with the core server - [3087](https://github.com/PrefectHQ/prefect/pull/3087)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [Pravin Dahal](https://github.com/pravindahal)
- [Panagiotis Simakis](https://github.com/sp1thas)

Released on July 28, 2020.

Features

- Add `flatten` operator for unnesting and flat-maps - [2898](https://github.com/PrefectHQ/prefect/pull/2898)

Enhancements

- Add `retry_on_api_error` flag to client methods - [3012](https://github.com/PrefectHQ/prefect/pull/3012)
- Add `reg_allow_list` option for Docker Agent - [3026](https://github.com/PrefectHQ/prefect/pull/3026#issuecomment-663078217)
- Update FargateTaskEnvironment to throw if task definition is inconsistent with existing task definition - [3031](https://github.com/PrefectHQ/prefect/pull/3031)

Fixes

- Cleanup to ShellTask to close open stdout file which was observable in some cases - [3002](https://github.com/PrefectHQ/prefect/issues/3002)
- Fix check of flow existence in storage object `get_flow` to only occur when provided - [3027](https://github.com/PrefectHQ/prefect/issues/3027)
- Use full name and tag when Docker Storage determines if build was successful - [3029](https://github.com/PrefectHQ/prefect/pull/3029)
- Prevent duplicated agent labels - [3029](https://github.com/PrefectHQ/prefect/pull/3042)

Deprecations

- `prefect.utilities.tasks.unmapped` moved to `prefect.utilities.edges.unmapped` - [2898](https://github.com/PrefectHQ/prefect/pull/2898)

Breaking Changes

- Remove `dbt` extra from dependencies - [3018](https://github.com/PrefectHQ/prefect/pull/3018)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [Spencer Ellinor](https://github.com/zpencerq)
- [Thomas Frederik Hoeck](https://github.com/thomasfrederikhoeck)
- [berosen](https://github.com/berosen)

Changelog

Released on July 21, 2020.

Features

- Add `resource_manager` api for cleaner setup/cleanup of temporary resources - [2913](https://github.com/PrefectHQ/prefect/pull/2913)

Enhancements

- Add `new_flow_context` to FlowRunTask for configurable context - [2941](https://github.com/PrefectHQ/prefect/issues/2941)
- All storage types now support file-based storage - [2944](https://github.com/PrefectHQ/prefect/pull/2944)
- Turn work stealing ON by default on Dask K8s environment - [2973](https://github.com/PrefectHQ/prefect/pull/2973)
- Send regular heartbeats while waiting to retry / dequeue - [2977](https://github.com/PrefectHQ/prefect/pull/2977)
- Cached states now validate based on `hashed_inputs` for more efficient storage - [2984](https://github.com/PrefectHQ/prefect/pull/2984)
- Simplify creation of optional parameters with default of `None` - [2995](https://github.com/PrefectHQ/prefect/pull/2995)

Task Library

- Implement AWSSecretsManager task - [2069](https://github.com/PrefectHQ/prefect/issues/2069)
- Update return value and config for DbtShellTask - [2980](https://github.com/PrefectHQ/prefect/pull/2980)

Fixes

- Don't send idempotency key when running against a local backend - [3001](https://github.com/PrefectHQ/prefect/issues/3001)
- Fix bug in `DaskExecutor` when running with external cluster where dask clients could potentially be leaked - [3009](https://github.com/PrefectHQ/prefect/pull/3009)

Deprecations

- All states have deprecated the usage of `cached_inputs` - [2984](https://github.com/PrefectHQ/prefect/pull/2984)

Breaking Changes

- Remove password from Postgres tasks' initialization methods for security - [1345](https://github.com/PrefectHQ/prefect/issues/1345)

Contributors

- [Robin Beer](https://github.com/Zaubeerer)

Released on July 14, 2020.

Enhancements

- Improve output formatting of `prefect describe` CLI - [2934](https://github.com/PrefectHQ/prefect/pull/2934)
- Add new `wait` kwarg to Flow Run Task for reflecting the flow run state in the task - [2935](https://github.com/PrefectHQ/prefect/pull/2935)
- Separate build-time and run-time job spec details in KubernetesJobEnvironment - [2950](https://github.com/PrefectHQ/prefect/pull/2950)

Task Library

- Implement RunNamespacedJob task for Kubernetes - [2916](https://github.com/PrefectHQ/prefect/pull/2916)
- Add `log_stderr` option to `ShellTask` and `DbtShellTask` for logging the full output from stderr - [2961](https://github.com/PrefectHQ/prefect/pull/2961)

Fixes

- Ensure `is_serializable` always uses same executable for subprocess. - [1262](https://github.com/PrefectHQ/prefect/issues/1262)
- Fix issue with Mapped tasks not always reloading child state results on reruns - [2656](https://github.com/PrefectHQ/prefect/issues/2656)
- Fix `FargateTaskEnvironment` attempting to retrieve authorization token when not present - [2940](https://github.com/PrefectHQ/prefect/pull/2940)
- Fix issue with Metastates compounding - [2965](https://github.com/PrefectHQ/prefect/pull/2965)

Contributors

- [Chris Bowdon](https://github.com/cbowdon)
- [James Lamb](https://github.com/jameslamb)
- [Paweł Cieśliński](https://github.com/pcieslinski)

Released on July 8, 2020.

Enhancements

- Update `flow.slugs` during `flow.replace` - [2919](https://github.com/PrefectHQ/prefect/issues/2919)
- `flow.update` accepts the optional kwarg `merge_parameters` that allows flows to be updated with common `Parameters` - [2501](https://github.com/PrefectHQ/prefect/issues/2501)
- Added poke handler to notify agent process of available flow runs - [2914](https://github.com/PrefectHQ/prefect/pull/2914)
- Add `Cancelling` state for indicating a flow-run that is being cancelled, but may still have tasks running - [2923](https://github.com/PrefectHQ/prefect/pull/2923)

Task Library

- Add `ReadAirtableRow` task - [2843](https://github.com/PrefectHQ/prefect/pull/2843)
- Add `container_name` kwarg to `CreateContainer` Docker task - [2904](https://github.com/PrefectHQ/prefect/pull/2904)
- Adds an `extra_docker_kwargs` argument to `CreateContainer` Docker task - [2915](https://github.com/PrefectHQ/prefect/pull/2915)

Fixes

- Fix issue with short-interval IntervalClocks that had a start_date far in the past - [2906](https://github.com/PrefectHQ/prefect/pull/2906)
- When terminating early, executors ensure all pending work is cancelled/completed before returning, ensuring no lingering background processing - [2920](https://github.com/PrefectHQ/prefect/pull/2920)

Contributors

- [Bradley McElroy](https://github.com/limx0)
- [Itay Livni](https://github.com/gryBox)
- [Matthew Alhonte](https://github.com/mattalhonte)
- [Panagiotis Simakis](https://github.com/sp1thas)
- [Sandeep Aggarwal](https://github.com/asandeep)

Released on June 30, 2020.

Features

- Add `apply_map`, a function to simplify creating complex mapped pipelines - [2846](https://github.com/PrefectHQ/prefect/pull/2846)

Enhancements

- Make storage location inside Docker storage configurable - [2865](https://github.com/PrefectHQ/prefect/pull/2865)
- Send heartbeats on each iteration of the Cloud task runner's retry loop - [2893](https://github.com/PrefectHQ/prefect/pull/2893)

Task Library

- Add option to BigQueryTask to return query as dataframe - [2862](https://github.com/PrefectHQ/prefect/pull/2862)

Fixes

- Add more context keys when running locally so that templating is consistent between local and Cloud runs - [2662](https://github.com/PrefectHQ/prefect/issues/2662)
- Fix Fargate agent not parsing string provided containerDefinitions - [2875](https://github.com/PrefectHQ/prefect/issues/2875)
- Fix Fargate agent providing empty parameters if not set - [2878](https://github.com/PrefectHQ/prefect/issues/2878)
- Fix issue with Queued task runs flooding agents with work - [2884](https://github.com/PrefectHQ/prefect/issues/2884)
- Add missing `prefect register flow` to CLI help text - [2895](https://github.com/PrefectHQ/prefect/pull/2895)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [nathaniel-md](https://github.com/nathaniel-md)