Prefect

Changelog

Released on Nov 26, 2019

Features

- Add graceful cancellation hooks to Flow and Task runners - [1758](https://github.com/PrefectHQ/prefect/pull/1758)

Enhancements

- Add option to specify a run name for `cloud run` CLI command - [1756](https://github.com/PrefectHQ/prefect/pull/1756)
- Add `work_stealing` option to `DaskKubernetesEnvironment` - [1760](https://github.com/PrefectHQ/prefect/pull/1760)
- Improve heartbeat thread management - [1770](https://github.com/PrefectHQ/prefect/pull/1770)
- Add unique scheduler Job name to `DaskKubernetesEnvironment` - [1772](https://github.com/PrefectHQ/prefect/pull/1772)
- Add informative error when trying to map with the `LocalDaskExecutor` using processes - [1777](https://github.com/PrefectHQ/prefect/pull/1777)

Task Library

- None

Fixes

- Fix issue with heartbeat thread deadlocking dask execution when using a `worker_client` - [1750](https://github.com/PrefectHQ/prefect/pull/1750)
- Fix issue with Environments not calling `run_flow` on Environment stored on Flow object - [1752](https://github.com/PrefectHQ/prefect/pull/1752)
- Fix issue with Docker build context when providing custom docker files - [1762](https://github.com/PrefectHQ/prefect/pull/1762)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on Nov 15, 2019.

Features

- Allow users to provide a custom version group ID for controlling Cloud versioning - [1665](https://github.com/PrefectHQ/prefect/issues/1665)
- Stop autogenerating constant tasks - [1730](https://github.com/PrefectHQ/prefect/pull/1730)

Enhancements

- Raise an informative error when context objects are pickled - [1710](https://github.com/PrefectHQ/prefect/issues/1710)
- Add an option to pass in `run_name` to a flow run to override the auto-generated names when calling `create_flow_run` [1661](https://github.com/PrefectHQ/cloud/pull/1661)
- Add informative logs in the event that a heartbeat thread dies - [1721](https://github.com/PrefectHQ/prefect/pull/1721)
- Loosen Job spec requirements for `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Loosen `containerDefinitions` requirements for `FargateTaskEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Local Docker agent proactively fails flow runs if image cannot be pulled - [1395](https://github.com/PrefectHQ/prefect/issues/1395)
- Add graceful keyboard interrupt shutdown for all agents - [1731](https://github.com/PrefectHQ/prefect/pull/1731)
- `agent start` CLI command now allows for Agent kwargs - [1737](https://github.com/PrefectHQ/prefect/pull/1737)
- Add users to specify a custom Dockerfile for Docker storage - [1738](https://github.com/PrefectHQ/prefect/pull/1738)
- Expose `labels` kwarg in `flow.deploy` for convenient labeling of Flows - [1742](https://github.com/PrefectHQ/prefect/pull/1742)

Task Library

- None

Fixes

- `FargateTaskEnvironment` now uses provided `family` for task definition naming - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix executor initialization missing `self` in `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix `identifier_label` not being generated on each run for Kubernetes based environments - [1718](https://github.com/PrefectHQ/prefect/pull/1718)
- Fix issue where users could not override their user config path when deploying Docker to Cloud - [1719](https://github.com/PrefectHQ/prefect/pull/1719)
- Respect order of inputs in merge - [1736](https://github.com/~/1736)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Brett Naul](https://github.com/bnaul)

Released on Nov 5, 2019

Features

- None

Enhancements

- Add a `save`/`load` interface to Flows - [1685](https://github.com/PrefectHQ/prefect/pull/1685), [#1695](https://github.com/PrefectHQ/prefect/pull/1695)
- Add option to specify `aws_session_token` for the `FargateTaskEnvironment` - [1688](https://github.com/PrefectHQ/prefect/pull/1688)
- Add `EnvVarSecrets` for loading sensitive information from environment variables - [1683](https://github.com/PrefectHQ/prefect/pull/1683)
- Add an informative version header to all Cloud client requests - [1690](https://github.com/PrefectHQ/prefect/pull/1690)
- Auto-label Flow environments when using Local storage - [1696](https://github.com/PrefectHQ/prefect/pull/1696)
- Batch upload logs to Cloud in a background thread for improved performance - [1691](https://github.com/PrefectHQ/prefect/pull/1691)
- Include agent labels within each flow's configuration environment - [1671](https://github.com/PrefectHQ/prefect/issues/1671)

Task Library

- None

Fixes

- Fix Fargate Agent access defaults and environment variable support - [1687](https://github.com/PrefectHQ/prefect/pull/1687)
- Removed default python version for docker builds - [1705](https://github.com/PrefectHQ/prefect/pull/1705)
- Attempt to install prefect in any docker image (if it is not already installed) - [1704](https://github.com/PrefectHQ/prefect/pull/1704)
- Kubernetes Agent deployment yaml now respects new `prefecthq/prefect` image tagging convention - [1707](https://github.com/PrefectHQ/prefect/pull/1707)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on October 29th, 2019.

Features

- Flow Affinity: Environments and Agents now support labeling for execution specification - [1651](https://github.com/PrefectHQ/prefect/pull/1651)
- Add new Secret Tasks for a pluggable and reusable Secrets API - [1346](https://github.com/PrefectHQ/prefect/issues/1346), [#1587](https://github.com/PrefectHQ/prefect/issues/1587)

Enhancements

- Add the ability to delete task tag limits using the client - [1622](https://github.com/PrefectHQ/prefect/pull/1622)
- Adds an "Ask for help" button with a link to the prefect.io support page - [1637](https://github.com/PrefectHQ/prefect/pull/1637)
- Reduces the size of the `prefecthq/prefect` Docker image by ~400MB, which is now the base Docker image used in Flows - [1648](https://github.com/PrefectHQ/prefect/pull/1648)
- Add a new healthcheck for environment dependencies - [1653](https://github.com/PrefectHQ/prefect/pull/1653)
- Add default 30 second timeout to Client requests - [1672](https://github.com/PrefectHQ/prefect/pull/1672)

Task Library

- Add new Secret Tasks for a pluggable and reusable Secrets API - [1346](https://github.com/PrefectHQ/prefect/issues/1346), [#1587](https://github.com/PrefectHQ/prefect/issues/1587)
- Add support for directly passing credentials to task library tasks, instead of passing secret names - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Fixes

- Fix defaults for unspecified ARNs in the Fargate Agent - [1634](https://github.com/PrefectHQ/prefect/pull/1634)
- Fix ShellTask return value on empty stdout - [1632](https://github.com/PrefectHQ/prefect/pull/1632)
- Fix issue with some Cloud Secrets not being converted from strings - [1655](https://github.com/PrefectHQ/prefect/pull/1655)
- Fix issue with Agent logging config setting not working - [1657](https://github.com/PrefectHQ/prefect/pull/1657)
- Fix issue with SnowflakeQuery tasks not working - [1663](https://github.com/PrefectHQ/prefect/pull/1663)

Deprecations

- Tasks that accepted the name of a secret (often `credentials_secret`) will raise a deprecation warning - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Breaking Changes

- Fargate Agent now takes in all boto3 camel case arguments instead of specific snake case options - [1649](https://github.com/PrefectHQ/prefect/pull/1649)
- `kubernetes` is no longer installed by default in deployed flow images - [1653](https://github.com/PrefectHQ/prefect/pull/1653)
- Tasks that accepted the name of a secret (often `credentials_secret`) no longer have a default value for that argument, as it has been deprecated - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Contributors

- [Tobias Schmidt](https://github.com/royalts)

Released on Oct 16, 2019

Features

- Environments now allow for optional `on_start` and `on_exit` callbacks - [1610](https://github.com/PrefectHQ/prefect/pull/1610)

Enhancements

- Raise more informative error when calling `flow.visualize()` if Graphviz executable not installed - [1602](https://github.com/PrefectHQ/prefect/pull/1602)
- Allow authentication to Azure Blob Storage with SAS token - [1600](https://github.com/PrefectHQ/prefect/pull/1600)
- Additional debug logs to `Docker Container` and `Docker Image` tasks - [920](https://github.com/PrefectHQ/prefect/issues/920)
- Changes to Fargate agent to support temporary credentials and IAM role based credentials within AWS compute such as a container or ec2 instance. [1607](https://github.com/PrefectHQ/prefect/pull/1607)
- Local Secrets set through environment variable now retain their casing - [1601](https://github.com/PrefectHQ/prefect/issues/1601)
- Agents can accept an optional `name` for logging and debugging - [1612](https://github.com/PrefectHQ/prefect/pull/1612)
- Added AWS configuration options for Fargate Agent (task_role_arn, execution_role_arn) - [1614](https://github.com/PrefectHQ/prefect/pull/1614)
- Change EmailTask to accept SMTP server settings as well as an email_from kwarg - [1619](https://github.com/PrefectHQ/prefect/pull/1619)

Task Library

- Add `return_all` kwarg to `ShellTask` for optionally returning all lines of stdout - [1598](https://github.com/PrefectHQ/prefect/pull/1598)
- Add `CosmosDBCreateItem`, `CosmosDBReadItems`, `CosmosDBQueryItems` and for interacting with data stored on Azure Cosmos DB - [1617](https://github.com/PrefectHQ/prefect/pull/1617)

Fixes

- Fix issue with running local Flow without a schedule containing cached tasks - [1599](https://github.com/PrefectHQ/prefect/pull/1599)
- Remove blank string for `task_run_id` in k8s resource manager - [1604](https://github.com/PrefectHQ/prefect/pull/1604)
- Fix issue with merge task not working for pandas dataframes and numpy arrays - [1609](https://github.com/PrefectHQ/prefect/pull/1609)

Deprecations

- None

Breaking Changes

- Local Secrets set through environment variable now retain their casing - [1601](https://github.com/PrefectHQ/prefect/issues/1601)

Contributors

- [Mark McDonald](https://github.com/mhmcdonal)
- [Sherman K](https://github.com/shrmnk)

Released on October 3, 2019

Features

- Added `KubernetesJobEnvironment` - [1548](https://github.com/PrefectHQ/prefect/pull/1548)
- Add ability to enforce Task concurrency limits by tag in Prefect Cloud - [1570](https://github.com/PrefectHQ/prefect/pull/1570)
- Added `FargateTaskEnvironment` - [1592](https://github.com/PrefectHQ/prefect/pull/1592)

Enhancements

- Allow the `Client` to more gracefully handle failed login attempts on initialization - [1535](https://github.com/PrefectHQ/prefect/pull/1535)
- Replace `DotDict` with `box.Box` - [1518](https://github.com/PrefectHQ/prefect/pull/1518)
- Store `cached_inputs` on Failed states and call their result handlers if they were provided - [1557](https://github.com/PrefectHQ/prefect/pull/1557)
- `raise_on_exception` no longer raises for Prefect Signals, as these are typically intentional / for control flow - [1562](https://github.com/PrefectHQ/prefect/pull/1562)
- `run cloud` CLI command takes in optional `--parameters` as a file path pointing to a JSON file - [1582](https://github.com/PrefectHQ/prefect/pull/1582)
- Always consider `Constant` tasks successful and unpack them immediately instead of submitting them for execution - [1527](https://github.com/PrefectHQ/prefect/issues/1527)

Task Library

- Add `BlobStorageDownload` and `BlobStorageUpload` for interacting with data stored on Azure Blob Storage - [1538](https://github.com/PrefectHQ/prefect/pull/1538)
- Loosen Kubernetes Tasks' requirement of an API secret key - [1559](https://github.com/PrefectHQ/prefect/pull/1559)
- Add tasks for working in Azure Machine Learning Serviec with Datastores and Datasets - [1590](https://github.com/PrefectHQ/prefect/pull/1590)

Fixes

- Fix issue with certain Pause / Resume / Retry pipelines retrying indefinitely - [1177](https://github.com/PrefectHQ/prefect/issues/1177)
- Kubernetes Agent deployment YAML generation defaults to local Prefect version - [1573](https://github.com/PrefectHQ/prefect/pull/1573)
- Fix issue with custom result handlers not working when called in `cached_inputs` - [1585](https://github.com/PrefectHQ/prefect/pull/1585)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Fredrik Sannholm](https://github.com/frsann)