Prefect

Released on Jan 15, 2020.

Features

- Added the ability to leverage native ECS task definition revisions for flow versions in Fargate agent - [1870](https://github.com/PrefectHQ/prefect/pull/1870)
- Added the ability to pull in kwargs per flow version from S3 on flow submission in Fargate agent - [1870](https://github.com/PrefectHQ/prefect/pull/1870)
- Add sensible default result handlers to non-Docker storage options - [1888](https://github.com/PrefectHQ/prefect/issues/1888)

Enhancements

- Allow for task looping to beyond Python's maximum recursion depth - [1862](https://github.com/PrefectHQ/prefect/pull/1862)
- Prevent duplication of stdout logs from multiple instantiated agents - [1866](https://github.com/PrefectHQ/prefect/pull/1866)
- Allow intervals less than 60 seconds in `IntervalClock`s - [1880](https://github.com/PrefectHQ/prefect/pull/1880)
- Introduce new `Secret.exists` method for checking whether a Secret is available - [1882](https://github.com/PrefectHQ/prefect/pull/1882)
- Introduce new `-e` CLI options on agent start commands to allow passing environment variables to flow runs - [1878](https://github.com/PrefectHQ/prefect/issues/1878)
- Stop persisting `None` when calling result handlers - [1894](https://github.com/PrefectHQ/prefect/pull/1894)
- Change Cancelled state to indicate Finished instead of Failed - [1903](https://github.com/PrefectHQ/prefect/pull/1903)
- All States now store `cached_inputs` for easier recovery from failure - [1898](https://github.com/PrefectHQ/prefect/issues/1898)
- Always checkpoint tasks which have result handlers - [1898](https://github.com/PrefectHQ/prefect/issues/1898)


Task Library

- Remove implicit requirement that Google Tasks use Prefect Cloud Secrets - [1882](https://github.com/PrefectHQ/prefect/pull/1882)

Fixes

- Enforce provision of `max_retries` if specifying `retry_delay` for a `Task` - [1875](https://github.com/PrefectHQ/prefect/pull/1875)
- Fix issue with reduce tasks in `flow.visualize()` - [1793](https://github.com/PrefectHQ/prefect/issues/1793)

Deprecations

- The checkpointing kwarg will be removed from Tasks as it is now a default behavior - [1898](https://github.com/PrefectHQ/prefect/issues/1898)

Breaking Changes

- Remove default value for `aws_credentials_secret` on all S3 hooks - [1886](https://github.com/PrefectHQ/prefect/issues/1886)
- Remove `config.engine.result_handler` section of Prefect config - [1888](https://github.com/PrefectHQ/prefect/issues/1888)
- Remove default value for `credentials_secret` on `GCSResultHandler` - [1888](https://github.com/PrefectHQ/prefect/issues/1888)
- Remove default value for `azure_credentials_secret` on `AzureResultHandler` - [1888](https://github.com/PrefectHQ/prefect/issues/1888)

Contributors

- [Daryll Strauss](daryll.straussgmail.com)
- [Braun Reyes](https://github.com/braunreyes)

Released on Dec 17, 2019

Features

- None

Enhancements

- Enhanced treatment of nested and ordered constant values - [1829](https://github.com/PrefectHQ/prefect/pull/1829)
- Add `on_datetime`, `on_date`, and `at_time` filters - [1837](https://github.com/PrefectHQ/prefect/pull/1837)
- Add `--latest` flag for Kubernetes Agent install CLI command - [1842](https://github.com/PrefectHQ/prefect/pull/1842)
- Add `--no-cloud-logs` flag for all agents to optionally opt-out of logs being sent to Prefect Cloud - [1843](https://github.com/PrefectHQ/prefect/pull/1843)
- Agents mark Flow Runs as `Failed` if a deployment error occurs - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- `Submitted` states from Agents include deployment identifier information - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- Update heartbeats to respect Cloud flow settings - [1851](https://github.com/PrefectHQ/prefect/pull/1851)
- Add flow run name to `prefect.context` - [1855](https://github.com/PrefectHQ/prefect/pull/1855)
- Add `--namespace` option for Kubernetes Agent start CLI command - [1859](https://github.com/PrefectHQ/prefect/pull/1859)
- Add Prefect job resource configuration for Kubernetes Agent - [1859](https://github.com/PrefectHQ/prefect/pull/1859)

Task Library

- None

Fixes

- Fix Agent deployment errors interrupting full list of found Flow Runs - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- Fix issue with a single bad log preventing all logs from being sent to Cloud - [1845](https://github.com/PrefectHQ/prefect/pull/1845)
- Fix Kubernetes Agent passing empty default namespace - [1839](https://github.com/PrefectHQ/prefect/pull/1839)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on Dec 11, 2019

Features

- Universal Deploy: Added new Local Agent to run Flows from Local Storage, Azure Storage, S3 Storage, and GCS Storage - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Added Azure Blob Storage for Flows - [1831](https://github.com/PrefectHQ/prefect/pull/1831)
- Added GCS Storage for Flows - [1809](https://github.com/PrefectHQ/prefect/pull/1809)
- Added S3 Storage for Flows - [1753](https://github.com/PrefectHQ/prefect/pull/1753)

Enhancements

- Add convenience `parents()` and `children()` classmethods to all State objects for navigating the hierarchy - [1784](https://github.com/PrefectHQ/prefect/pull/1784)
- Add `--rbac` flag to `prefect agent install` for Kubernetes Agent - [1822](https://github.com/PrefectHQ/prefect/pull/1822)
- Add `flow_run_name` to the context - [1815](https://github.com/PrefectHQ/prefect/pull/1815)
- Add `prefect agent install` option to output `supervisord.conf` file for Local Agent - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Add convenience `parents()` and `children()` classmethods to all State objects for navigating the hierarchy - [1784](https://github.com/PrefectHQ/prefect/pull/1784)
- Add new `not_all_skipped` trigger and set it as the default for merge tasks - [1768](https://github.com/PrefectHQ/prefect/issues/1768)


Task Library

- Azure Blob tasks now use newer `BlockBlobService` with connection string authentication - [1831](https://github.com/PrefectHQ/prefect/pull/1831)

Fixes

- Fix issue with `flow.visualize()` for mapped tasks which are skipped - [1765](https://github.com/PrefectHQ/prefect/issues/1765)
- Fix issue with timeouts only being softly enforced - [1145](https://github.com/PrefectHQ/prefect/issues/1145), [#1686](https://github.com/PrefectHQ/prefect/issues/1686)
- Log agent errors using `write_run_logs` instead of the deprecated `write_run_log` - [1791](https://github.com/PrefectHQ/prefect/pull/1791)
- Fix issue with `flow.update()` not transferring constants - [1785](https://github.com/PrefectHQ/prefect/pull/1785)

Deprecations

- `flow.deploy` is deprecated in favor of `flow.register` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)

Breaking Changes

- Default Flow storage is now `Local` instead of `Docker` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Docker based `LocalAgent` is renamed `DockerAgent` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- `prefect agent start` now defaults to new `LocalAgent` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)

Changelog

Released on Nov 26, 2019

Features

- Add graceful cancellation hooks to Flow and Task runners - [1758](https://github.com/PrefectHQ/prefect/pull/1758)

Enhancements

- Add option to specify a run name for `cloud run` CLI command - [1756](https://github.com/PrefectHQ/prefect/pull/1756)
- Add `work_stealing` option to `DaskKubernetesEnvironment` - [1760](https://github.com/PrefectHQ/prefect/pull/1760)
- Improve heartbeat thread management - [1770](https://github.com/PrefectHQ/prefect/pull/1770)
- Add unique scheduler Job name to `DaskKubernetesEnvironment` - [1772](https://github.com/PrefectHQ/prefect/pull/1772)
- Add informative error when trying to map with the `LocalDaskExecutor` using processes - [1777](https://github.com/PrefectHQ/prefect/pull/1777)

Task Library

- None

Fixes

- Fix issue with heartbeat thread deadlocking dask execution when using a `worker_client` - [1750](https://github.com/PrefectHQ/prefect/pull/1750)
- Fix issue with Environments not calling `run_flow` on Environment stored on Flow object - [1752](https://github.com/PrefectHQ/prefect/pull/1752)
- Fix issue with Docker build context when providing custom docker files - [1762](https://github.com/PrefectHQ/prefect/pull/1762)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on Nov 15, 2019.

Features

- Allow users to provide a custom version group ID for controlling Cloud versioning - [1665](https://github.com/PrefectHQ/prefect/issues/1665)
- Stop autogenerating constant tasks - [1730](https://github.com/PrefectHQ/prefect/pull/1730)

Enhancements

- Raise an informative error when context objects are pickled - [1710](https://github.com/PrefectHQ/prefect/issues/1710)
- Add an option to pass in `run_name` to a flow run to override the auto-generated names when calling `create_flow_run` [1661](https://github.com/PrefectHQ/cloud/pull/1661)
- Add informative logs in the event that a heartbeat thread dies - [1721](https://github.com/PrefectHQ/prefect/pull/1721)
- Loosen Job spec requirements for `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Loosen `containerDefinitions` requirements for `FargateTaskEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Local Docker agent proactively fails flow runs if image cannot be pulled - [1395](https://github.com/PrefectHQ/prefect/issues/1395)
- Add graceful keyboard interrupt shutdown for all agents - [1731](https://github.com/PrefectHQ/prefect/pull/1731)
- `agent start` CLI command now allows for Agent kwargs - [1737](https://github.com/PrefectHQ/prefect/pull/1737)
- Add users to specify a custom Dockerfile for Docker storage - [1738](https://github.com/PrefectHQ/prefect/pull/1738)
- Expose `labels` kwarg in `flow.deploy` for convenient labeling of Flows - [1742](https://github.com/PrefectHQ/prefect/pull/1742)

Task Library

- None

Fixes

- `FargateTaskEnvironment` now uses provided `family` for task definition naming - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix executor initialization missing `self` in `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix `identifier_label` not being generated on each run for Kubernetes based environments - [1718](https://github.com/PrefectHQ/prefect/pull/1718)
- Fix issue where users could not override their user config path when deploying Docker to Cloud - [1719](https://github.com/PrefectHQ/prefect/pull/1719)
- Respect order of inputs in merge - [1736](https://github.com/~/1736)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Brett Naul](https://github.com/bnaul)

Released on Nov 5, 2019

Features

- None

Enhancements

- Add a `save`/`load` interface to Flows - [1685](https://github.com/PrefectHQ/prefect/pull/1685), [#1695](https://github.com/PrefectHQ/prefect/pull/1695)
- Add option to specify `aws_session_token` for the `FargateTaskEnvironment` - [1688](https://github.com/PrefectHQ/prefect/pull/1688)
- Add `EnvVarSecrets` for loading sensitive information from environment variables - [1683](https://github.com/PrefectHQ/prefect/pull/1683)
- Add an informative version header to all Cloud client requests - [1690](https://github.com/PrefectHQ/prefect/pull/1690)
- Auto-label Flow environments when using Local storage - [1696](https://github.com/PrefectHQ/prefect/pull/1696)
- Batch upload logs to Cloud in a background thread for improved performance - [1691](https://github.com/PrefectHQ/prefect/pull/1691)
- Include agent labels within each flow's configuration environment - [1671](https://github.com/PrefectHQ/prefect/issues/1671)

Task Library

- None

Fixes

- Fix Fargate Agent access defaults and environment variable support - [1687](https://github.com/PrefectHQ/prefect/pull/1687)
- Removed default python version for docker builds - [1705](https://github.com/PrefectHQ/prefect/pull/1705)
- Attempt to install prefect in any docker image (if it is not already installed) - [1704](https://github.com/PrefectHQ/prefect/pull/1704)
- Kubernetes Agent deployment yaml now respects new `prefecthq/prefect` image tagging convention - [1707](https://github.com/PrefectHQ/prefect/pull/1707)

Deprecations

- None

Breaking Changes

- None

Contributors

- None