Prefect

Released on July 8, 2020.

Enhancements

- Update `flow.slugs` during `flow.replace` - [2919](https://github.com/PrefectHQ/prefect/issues/2919)
- `flow.update` accepts the optional kwarg `merge_parameters` that allows flows to be updated with common `Parameters` - [2501](https://github.com/PrefectHQ/prefect/issues/2501)
- Added poke handler to notify agent process of available flow runs - [2914](https://github.com/PrefectHQ/prefect/pull/2914)
- Add `Cancelling` state for indicating a flow-run that is being cancelled, but may still have tasks running - [2923](https://github.com/PrefectHQ/prefect/pull/2923)

Task Library

- Add `ReadAirtableRow` task - [2843](https://github.com/PrefectHQ/prefect/pull/2843)
- Add `container_name` kwarg to `CreateContainer` Docker task - [2904](https://github.com/PrefectHQ/prefect/pull/2904)
- Adds an `extra_docker_kwargs` argument to `CreateContainer` Docker task - [2915](https://github.com/PrefectHQ/prefect/pull/2915)

Fixes

- Fix issue with short-interval IntervalClocks that had a start_date far in the past - [2906](https://github.com/PrefectHQ/prefect/pull/2906)
- When terminating early, executors ensure all pending work is cancelled/completed before returning, ensuring no lingering background processing - [2920](https://github.com/PrefectHQ/prefect/pull/2920)

Contributors

- [Bradley McElroy](https://github.com/limx0)
- [Itay Livni](https://github.com/gryBox)
- [Matthew Alhonte](https://github.com/mattalhonte)
- [Panagiotis Simakis](https://github.com/sp1thas)
- [Sandeep Aggarwal](https://github.com/asandeep)

Released on June 30, 2020.

Features

- Add `apply_map`, a function to simplify creating complex mapped pipelines - [2846](https://github.com/PrefectHQ/prefect/pull/2846)

Enhancements

- Make storage location inside Docker storage configurable - [2865](https://github.com/PrefectHQ/prefect/pull/2865)
- Send heartbeats on each iteration of the Cloud task runner's retry loop - [2893](https://github.com/PrefectHQ/prefect/pull/2893)

Task Library

- Add option to BigQueryTask to return query as dataframe - [2862](https://github.com/PrefectHQ/prefect/pull/2862)

Fixes

- Add more context keys when running locally so that templating is consistent between local and Cloud runs - [2662](https://github.com/PrefectHQ/prefect/issues/2662)
- Fix Fargate agent not parsing string provided containerDefinitions - [2875](https://github.com/PrefectHQ/prefect/issues/2875)
- Fix Fargate agent providing empty parameters if not set - [2878](https://github.com/PrefectHQ/prefect/issues/2878)
- Fix issue with Queued task runs flooding agents with work - [2884](https://github.com/PrefectHQ/prefect/issues/2884)
- Add missing `prefect register flow` to CLI help text - [2895](https://github.com/PrefectHQ/prefect/pull/2895)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [nathaniel-md](https://github.com/nathaniel-md)

Released on June 25, 2020.

Features

- Task slugs are now stable across rebuilds of the same Flow - [2531](https://github.com/PrefectHQ/prefect/pull/2531)
- Support configuring executors for `LocalEnvironment`, `KubernetesJobEnvironment`, and `FargateTaskEnvironment` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Flows can now be stored and executed using file-based storage - [2840](https://github.com/PrefectHQ/prefect/pull/2840)

Enhancements

- Add option to set `repositoryCredentials` on Fargate Agent `containerDefinitions` - [2822](https://github.com/PrefectHQ/prefect/issues/2822)
- Update GraphQL endpoint to `/graphql` - [2669](https://github.com/PrefectHQ/prefect/pull/2669)
- Allow Cloud Flow Runners to interact properly with Queued runs - [2741](https://github.com/PrefectHQ/prefect/pull/2741)
- Add `Result` serializers - [2755](https://github.com/PrefectHQ/prefect/pull/2755)
- Simplify `DaskExecutor` internals - [2817](https://github.com/PrefectHQ/prefect/pull/2817)
- Set task names in `LocalDaskExecutor` - [2819](https://github.com/PrefectHQ/prefect/pull/2819)
- Flows registered without an image set will default to `all_extras` - [2828](https://github.com/PrefectHQ/prefect/pull/2828)
- Improve error message when sending unauthorized requests to Cloud - [2810](https://github.com/PrefectHQ/prefect/issues/2810)
- Forward state change status back to core - [2839](https://github.com/PrefectHQ/prefect/pull/2839)
- Add GitHub storage for storing flows as files in a GitHub repo - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Add `prefect register flow` CLI command for registering flows from files - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Add default `GITHUB_ACCESS_TOKEN` secret - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Create utility function for getting Kubernetes client - [2845](https://github.com/PrefectHQ/prefect/pull/2845)

Task Library

- Adds a MySQL task using pymysql driver - [2124](https://github.com/PrefectHQ/prefect/issues/2124)
- Add some tasks for working with Google Sheets - [2614](https://github.com/PrefectHQ/prefect/pull/2614)
- Add support for HTML content in the EmailTask - [2811](https://github.com/PrefectHQ/prefect/pull/2811)

Server

- Failing to set a state raises errors more aggressively - [2708](https://github.com/PrefectHQ/prefect/pull/2708)

Fixes

- Fix `all_extras` tag not being set during CI job to build image - [2801](https://github.com/PrefectHQ/prefect/issues/2801)
- Quiet *no candidate Cached states were valid* debug logging - [2815](https://github.com/PrefectHQ/prefect/issues/2815)
- Fix `LocalEnvironment` execute function's use of the flow object - [2804](https://github.com/PrefectHQ/prefect/pull/2804)
- Properly set task names when using `DaskExecutor` - [2814](https://github.com/PrefectHQ/prefect/issues/2814)
- Fix the `LocalDaskExecutor` to only compute tasks once, not multiple times - [2819](https://github.com/PrefectHQ/prefect/pull/2819)
- Generate key names for mapped tasks that work better with Dask's dashboard - [2831](https://github.com/PrefectHQ/prefect/pull/2831)
- Fix FlowRunTask when running against locally deployed Server - [2832](https://github.com/PrefectHQ/prefect/pull/2832)
- Make sure image from Docker storage is always used with KubernetesJobEnvironment - [2838](https://github.com/PrefectHQ/prefect/pull/2838)
- Change Environment.run_flow() to prefer executor from flow's environment - [2849](https://github.com/PrefectHQ/prefect/pull/2849)

Deprecations

- Deprecate `RemoteEnvironment` in favor of `LocalEnvironment` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Deprecate `RemoteDaskEnvironment` in favor of `LocalEnvironment` with a `DaskExecutor` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Deprecate `executor_kwargs` in `KubernetesJobEnvironment` and `FargateTaskEnvironment` in favor of `executor` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)

Breaking Changes

- Remove previously deprecated `SynchronousExecutor` - [2826](https://github.com/PrefectHQ/prefect/pull/2826)

Contributors

- [manesioz](https://github.com/manesioz)
- [Alex Cano](https://github.com/alexisprince1994)
- [James Lamb](https://github.com/jameslamb)
- [Matthew Alhonte](https://github.com/mattalhonte/)
- [Paweł Cieśliński](https://github.com/pcieslinski)

Released on June 17, 2020.

Features

- Depth First Execution with Mapping on Dask - [2646](https://github.com/PrefectHQ/prefect/pull/2646)
- Support use of cloud storage with containerized environments - [2517](https://github.com/PrefectHQ/prefect/issues/2517),[#2796](https://github.com/PrefectHQ/prefect/pull/2796)

Enhancements

- Add flag to include hostname on local storage - [2653](https://github.com/PrefectHQ/prefect/issues/2653)
- Add option to set `image_pull_secret` directly on `DaskKubernetesEnvironment` - [2657](https://github.com/PrefectHQ/prefect/pull/2657)
- Allow for custom callables for Result locations - [2577](https://github.com/PrefectHQ/prefect/issues/2577)
- Ensure all Parameter values, included non-required defaults, are present in context - [2698](https://github.com/PrefectHQ/prefect/pull/2698)
- Use absolute path for `LocalResult` location for disambiguation - [2698](https://github.com/PrefectHQ/prefect/pull/2698)
- Retry client requests when receiving an `API_ERROR` code in the response - [2705](https://github.com/PrefectHQ/prefect/pull/2705)
- Reduce size of serialized tasks when running on Dask - [2707](https://github.com/PrefectHQ/prefect/pull/2707)
- Extend run state signatures for future development - [2718](https://github.com/PrefectHQ/prefect/pull/2718)
- Update set_flow_run_state for future meta state use - [2725](https://github.com/PrefectHQ/prefect/pull/2725)
- Add an optional `flow` argument to `merge` to support using it when not inside a flow context - [2727](https://github.com/PrefectHQ/prefect/pull/2727)
- Add option to set service account name on Prefect jobs created by Kubernetes agent - [2547](https://github.com/PrefectHQ/prefect/issues/2547)
- Add option to set imagePullPolicy on Prefect jobs created by Kubernetes agent - [2721](https://github.com/PrefectHQ/prefect/issues/2721)
- Add option to set API url on agent start CLI command - [2633](https://github.com/PrefectHQ/prefect/issues/2633)
- Add CI step to build `prefecthq/prefect:all_extras` Docker image for bundling all Prefect dependencies - [2745](https://github.com/PrefectHQ/prefect/pull/2745)
- Move `Parameter` to a standalone module - [2758](https://github.com/PrefectHQ/prefect/pull/2758)
- Validate Cached states based on hashed inputs - [2763](https://github.com/PrefectHQ/prefect/pull/2763)
- Add `validate_configuration` utility to Fargate Agent for verifying it can manage tasks properly - [2768](https://github.com/PrefectHQ/prefect/pull/2768)
- Add option to specify task targets as callables - [2769](https://github.com/PrefectHQ/prefect/pull/2769)
- Improve `State.__repr__` when there is no message - [2773](https://github.com/PrefectHQ/prefect/pull/2773)
- Add support for db argument at run time in the SQLiteQuery and SQLiteScript - [2782](https://github.com/PrefectHQ/prefect/pull/2782)
- Add support for mapped argument in control flows - [2784](https://github.com/PrefectHQ/prefect/pull/2784)
- Use pagination in kubernetes resource manager to reduce memory usage - [2794](https://github.com/PrefectHQ/prefect/pull/2794)

Task Library

- Adds a task to expose Great Expectations checkpoints as a node in a Prefect pipeline - [2489](https://github.com/PrefectHQ/prefect/issues/2489)

Fixes

- Fix flow.visualize cleanup of source files when using `filename` - [2726](https://github.com/PrefectHQ/prefect/issues/2726)
- Fix `S3Result` handling of AWS credentials provided through kwargs - [2747](https://github.com/PrefectHQ/prefect/issues/2747)
- Fix `DaskKubernetesEnvironment` requiring that an `env` block is set when using custom specs - [2657](https://github.com/PrefectHQ/prefect/pull/2657)
- Fix `PostgresExecute` task auto commit when commit is set to `False` - [2658](https://github.com/PrefectHQ/prefect/issue/2658)
- Remove need for `{filename}` in mapped templates - [2640](https://github.com/PrefectHQ/prefect/issues/2640)
- Fix issue with Results erroring out on multi-level mapped pipelines - [2716](https://github.com/PrefectHQ/prefect/issues/2716)
- Fix issue with dask resource tags not being respected - [2735](https://github.com/PrefectHQ/prefect/pull/2735)
- Ensure state deserialization works even when another StateSchema exists - [2738](https://github.com/PrefectHQ/prefect/pull/2738)
- Remove implicit payload size restriction from Apollo - [2764](https://github.com/PrefectHQ/prefect/pull/2764)
- Fix issue with declared storage secrets in K8s job environment and Dask K8s environment - [2780](https://github.com/PrefectHQ/prefect/pull/2780)
- Fix context handling for Cloud when working with in-process retries - [2783](https://github.com/PrefectHQ/prefect/pull/2783)

Deprecations

- Accessing `prefect.core.task.Parameter` is deprecated in favor of `prefect.core.parameter.Parameter` - [2758](https://github.com/PrefectHQ/prefect/pull/2758)

Breaking Changes

- Environment `setup` and `execute` function signatures now accept Flow objects - [2796](https://github.com/PrefectHQ/prefect/pull/2796)
- `create_flow_run_job` logic has been moved into `execute` for `DaskKubernetesEnvironment` and `KubernetesJobEnvironment` - [2796](https://github.com/PrefectHQ/prefect/pull/2796)

Contributors

- [Alex Cano](https://github.com/alexisprince1994)
- [David Haines](https://github.com/davidfhaines)
- [Paweł Cieśliński](https://github.com/pcieslinski)

Released on June 2, 2020.

Enhancements

- Allow for manual approval of locally Paused tasks - [2693](https://github.com/PrefectHQ/prefect/issues/2693)
- Task instances define a `__signature__` attribute, for improved introspection and tab-completion - [2602](https://github.com/PrefectHQ/prefect/pull/2602)
- Tasks created with `task` forward the wrapped function's docstring - [2602](https://github.com/PrefectHQ/prefect/pull/2602)
- Support creating temporary dask clusters from within a `DaskExecutor` - [2667](https://github.com/PrefectHQ/prefect/pull/2667)
- Add option for setting any build kwargs on Docker storage - [2668](https://github.com/PrefectHQ/prefect/pull/2668)
- Add flow run ID option to `get logs` CLI command - [2671](https://github.com/PrefectHQ/prefect/pull/2671)
- Add ID to output of `get` command for `flows` and `flow-runs` - [2671](https://github.com/PrefectHQ/prefect/pull/2671)

Fixes

- Fix issue with Google imports being tied together - [2661](https://github.com/PrefectHQ/prefect/issues/2661)
- Don't warn about unused tasks defined inline and copied - [2677](https://github.com/PrefectHQ/prefect/issues/2677)
- Remove unnecessary volume mount from dev infrastructure Docker compose - [2676](https://github.com/PrefectHQ/prefect/issues/2676)
- Fix issue with instantiating LocalResult on Windows with dir from other drive - [2683](https://github.com/PrefectHQ/prefect/issues/2683)
- Fix invalid IP address error when running `server start` on Ubuntu using rootless Docker - [2691](https://github.com/PrefectHQ/prefect/pull/2691)

Deprecations

- Deprecate `local_processes` and `**kwargs` arguments for `DaskExecutor` - [2667](https://github.com/PrefectHQ/prefect/pull/2667)
- Deprecate `address='local'` for `DaskExecutor` - [2667](https://github.com/PrefectHQ/prefect/pull/2667)

Contributors

- [Alex Cano](https://github.com/alexisprince1994)

Released on May 27, 2020.

Fixes

- Revert GraphQL endpoint change - [2660](https://github.com/PrefectHQ/prefect/pull/2660)