Prefect

Changelog

Released on September 10, 2019

Features

- Improve Windows compatibility for local development and deploying to Prefect Cloud - [1441](https://github.com/PrefectHQ/prefect/pull/1441), [#1456](https://github.com/PrefectHQ/prefect/pull/1456), [#1465](https://github.com/PrefectHQ/prefect/pull/1465), [#1466](https://github.com/PrefectHQ/prefect/pull/1466)

Enhancements

- Add OS platform check to Local Agent for running on Windows machines - [1441](https://github.com/PrefectHQ/prefect/pull/1441)
- Add `--base-url` argument for Docker daemons to `agent start` CLI command - [1441](https://github.com/PrefectHQ/prefect/pull/1441)
- Add environment labels for organizing / tagging different Flow execution environments - [1438](https://github.com/PrefectHQ/prefect/issues/1438)
- Use `-U` option when installing `prefect` in Docker containers to override base image version - [1461](https://github.com/PrefectHQ/prefect/pull/1461)
- Remove restriction that prevented `DotDict` classes from having keys that shadowed dict methods - [1462](https://github.com/PrefectHQ/prefect/pull/1462)
- Added livenessProbe to Kubernetes Agent - [1474](https://github.com/PrefectHQ/prefect/pull/1474)
- Ensure external Dask Clusters do not require Prefect Cloud environment variables to run Cloud flows - [1481](https://github.com/PrefectHQ/prefect/pull/1481)

Task Library

- None

Fixes

- Fix incorrect import in `DaskKubernetesEnvironment` job template - [1458](https://github.com/PrefectHQ/prefect/pull/1458)
- Raise error on Agents started without an appropriate API token - [1459](https://github.com/PrefectHQ/prefect/pull/1459)
- Fix bug when calling `as_nested_dict` on `DotDicts` with an `items` key - [1462](https://github.com/PrefectHQ/prefect/pull/1462)
- Fix `--resource-manager` flag on agent install invalidating `imagePullSecrets` - [1469](https://github.com/PrefectHQ/prefect/pull/1469)
- Fix issue with user-written result handlers in Prefect Cloud preventing some states from being set - [1480](https://github.com/PrefectHQ/prefect/pull/1480)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Joe Schmid](https://github.com/joeschmid)
- [Brett Naul](https://github.com/bnaul)

Released August 30, 2019

Fixes

- Fix issue with reduced mapped tasks not respecting retries - [1436](https://github.com/PrefectHQ/prefect/issues/1436)

Released August 30, 2019

Features

- Added Local, Kubernetes, and Nomad agents - [1341](https://github.com/PrefectHQ/prefect/pull/1341)
- Add the ability for Tasks to sequentially loop - [1356](https://github.com/PrefectHQ/prefect/pull/1356)

Enhancements

- Adds a copy to clipboard button for codeblocks - [213](https://github.com/prefecthq/prefect/issues/213)
- Updates Vuepress to v1.0.3 - [770](https://github.com/prefecthq/prefect/issues/770)
- Introduce configurable default for storage class on Flows - [1044](https://github.com/PrefectHQ/prefect/issues/1044)
- Allow for min and max workers to be specified in `DaskKubernetesEnvironment` - [1338](https://github.com/PrefectHQ/prefect/pulls/1338)
- Use task and flow names for corresponding logger names for better organization - [1355](https://github.com/PrefectHQ/prefect/pull/1355)
- `Paused` states subclass `Scheduled` and can have predefined expirations - [1375](https://github.com/PrefectHQ/prefect/pull/1375)
- Introduce new Flow health checks prior to Cloud deployment - [1372](https://github.com/PrefectHQ/prefect/issues/1372)
- Improve logging functionality to include tracebacks - [1374](https://github.com/PrefectHQ/prefect/issues/1374)
- Improve CLI user experience while working with Cloud - [1384](https://github.com/PrefectHQ/prefect/pull/1384/)
- Users can now create projects from the CLI - [1388](https://github.com/PrefectHQ/prefect/pull/1388)
- Add a health check to confirm that serialized flows are valid prior to Cloud deploy - [1397](https://github.com/PrefectHQ/prefect/pull/1397)
- Add `task_slug`, `flow_id`, and `flow_run_id` to context - [1405](https://github.com/PrefectHQ/prefect/pull/1405)
- Support persistent `scheduled_start_time` for scheduled flow runs when run locally with `flow.run()` - [1418](https://github.com/PrefectHQ/prefect/pull/1418), [#1429](https://github.com/PrefectHQ/prefect/pull/1429)
- Add `task_args` to `Task.map` - [1390](https://github.com/PrefectHQ/prefect/issues/1390)
- Add auth flows for `USER`-scoped Cloud API tokens - [1423](https://github.com/PrefectHQ/prefect/pull/1423)
- Add `AzureResultHandler` for handling results to / from Azure Blob storage containers - [1421](https://github.com/PrefectHQ/prefect/pull/1421)
- Add new configurable `LocalDaskExecutor` - [1336](https://github.com/PrefectHQ/prefect/issues/1336)
- Add CLI commands for working with Prefect Cloud auth - [1431](https://github.com/PrefectHQ/prefect/pull/1431)

Task Library

- Add new `SnowflakeQuery` task for using snowflake data warehouse - [1113](https://github.com/PrefectHQ/prefect/issues/1113)

Fixes

- Fix issue with Docker storage not respecting user-provided image names - [1335](https://github.com/PrefectHQ/prefect/pull/1335)
- Fix issue with local retries in Cloud not always running in-process - [1348](https://github.com/PrefectHQ/prefect/pull/1348)

Deprecations

- Rename `SynchronousExecutor` as `LocalDaskExecutor` - [1434](https://github.com/PrefectHQ/prefect/pull/1434)

Breaking Changes

- Rename `CloudEnvironment` to `DaskKubernetesEnvironment` - [1250](https://github.com/PrefectHQ/prefect/issues/1250)
- Remove unused `queue` method from all executors - [1434](https://github.com/PrefectHQ/prefect/pull/1434)

Contributors

- [Alex Kravetz](http://github.com/akravetz)

Released August 8, 2019

Features

- Introduce new `flows.checkpointing` configuration setting for checkpointing Tasks in local execution - [1283](https://github.com/PrefectHQ/prefect/pull/1283)
- Introduce new, flexible `Schedule` objects - [1320](https://github.com/PrefectHQ/prefect/pull/1320)

Enhancements

- Allow passing of custom headers in `Client` calls - [1255](https://github.com/PrefectHQ/prefect/pull/1255)
- Autogenerate informative names and tags for Docker images built for Flow storage - [1237](https://github.com/PrefectHQ/prefect/issues/1237)
- Allow mixed-case configuration keys (environment variables are interpolated as lowercase) - [1288](https://github.com/PrefectHQ/prefect/issues/1288)
- Ensure state handler errors are logged informatively - [1326](https://github.com/PrefectHQ/prefect/issues/1326)

Task Library

- Add `BigQueryLoadGoogleCloudStorage` task for loading data into BigQuery from Google Cloud Storage [1317](https://github.com/PrefectHQ/prefect/pull/1317)

Fixes

- Fix issue with logs not always arriving in long-standing Dask clusters - [1244](https://github.com/PrefectHQ/prefect/pull/1244)
- Fix issue with `BuildImage` docker task not actually running to completion - [1243](https://github.com/PrefectHQ/prefect/issues/1243)
- Fix `run --logs` CLI command not exiting on flow run finished state - [1319](https://github.com/PrefectHQ/prefect/pull/1319)

Deprecations

- `OneTimeSchedule` and `UnionSchedule` are deprecated, but remain callable as convenience functions - [1320](https://github.com/PrefectHQ/prefect/pull/1320)
- Old-style schedules can be deserialized as new-style schedules, but support will eventually be dropped - [1320](https://github.com/PrefectHQ/prefect/pull/1320)

Breaking Changes

- `prefect.Client.graphql()` and `prefect.Client.post()` now use an explicit keyword, not `**kwargs`, for variables or parameters - [1259](https://github.com/PrefectHQ/prefect/pull/1259)
- `auth add` CLI command replaced with `auth login` - [1319](https://github.com/PrefectHQ/prefect/pull/1319)

Released July 16, 2019

Features

- Add the Prefect CLI for working with core objects both locally and in cloud - [1059](https://github.com/PrefectHQ/prefect/pull/1059)
- Add RemoteEnvironment for simple executor based executions - [1215](https://github.com/PrefectHQ/prefect/pull/1215)
- Add the ability to share caches across Tasks and Flows - [1222](https://github.com/PrefectHQ/prefect/pull/1222)
- Add the ability to submit tasks to specific dask workers for task / worker affinity - [1229](https://github.com/PrefectHQ/prefect/pull/1229)

Enhancements

- Refactor mapped caching to be independent of order - [1082](https://github.com/PrefectHQ/prefect/issues/1082)
- Refactor caching to allow for caching across multiple runs - [1082](https://github.com/PrefectHQ/prefect/issues/1082)
- Allow for custom secret names in Result Handlers - [1098](https://github.com/PrefectHQ/prefect/issues/1098)
- Have `execute cloud-flow` CLI immediately set the flow run state to `Failed` if environment fails - [1122](https://github.com/PrefectHQ/prefect/pull/1122)
- Validate configuration objects on initial load - [1136](https://github.com/PrefectHQ/prefect/pull/1136)
- Add `auto_generated` property to Tasks for convenient filtering - [1135](https://github.com/PrefectHQ/prefect/pull/1135)
- Disable dask work-stealing in kubernetes via scheduler config - [1166](https://github.com/PrefectHQ/prefect/pull/1166)
- Implement backoff retry settings on Client calls - [1187](https://github.com/PrefectHQ/prefect/pull/1187)
- Explicitly set Dask keys for a better Dask visualization experience - [1218](https://github.com/PrefectHQ/prefect/issues/1218)
- Implement a local cache which persists for the duration of a Python session - [1221](https://github.com/PrefectHQ/prefect/issues/1221)
- Implement in-process retries for Cloud Tasks which request retry in less than one minute - [1228](https://github.com/PrefectHQ/prefect/pull/1228)
- Support `Client.login()` with API tokens - [1240](https://github.com/PrefectHQ/prefect/pull/1240)
- Add live log streaming for `prefect run cloud` command - [1241](https://github.com/PrefectHQ/prefect/pull/1241)

Task Library

- Add task to trigger AWS Step function workflow [1012](https://github.com/PrefectHQ/prefect/issues/1012)
- Add task to copy files within Google Cloud Storage - [1206](https://github.com/PrefectHQ/prefect/pull/1206)
- Add task for downloading files from Dropbox - [1205](https://github.com/PrefectHQ/prefect/pull/1205)

Fixes

- Fix issue with mapped caching in Prefect Cloud - [1096](https://github.com/PrefectHQ/prefect/pull/1096)
- Fix issue with Result Handlers deserializing incorrectly in Cloud - [1112](https://github.com/PrefectHQ/prefect/issues/1112)
- Fix issue caused by breaking change in `marshmallow==3.0.0rc7` - [1151](https://github.com/PrefectHQ/prefect/pull/1151)
- Fix issue with passing results to Prefect signals - [1163](https://github.com/PrefectHQ/prefect/issues/1163)
- Fix issue with `flow.update` not preserving mapped edges - [1164](https://github.com/PrefectHQ/prefect/issues/1164)
- Fix issue with Parameters and Context not being raw dictionaries - [1186](https://github.com/PrefectHQ/prefect/issues/1186)
- Fix issue with asynchronous, long-running mapped retries in Prefect Cloud - [1208](https://github.com/PrefectHQ/prefect/pull/1208)
- Fix issue with automatically applied collections to task call arguments when using the imperative API - [1211](https://github.com/PrefectHQ/prefect/issues/1211)

Breaking Changes

- The CLI command `prefect execute-flow` and `prefect execute-cloud-flow` no longer exist - [1059](https://github.com/PrefectHQ/prefect/pull/1059)
- The `slack_notifier` state handler now uses a `webhook_secret` kwarg to pull the URL from a Secret - [1075](https://github.com/PrefectHQ/prefect/issues/1075)
- Use GraphQL for Cloud logging - [1193](https://github.com/PrefectHQ/prefect/pull/1193)
- Remove the `CloudResultHandler` default result handler - [1198](https://github.com/PrefectHQ/prefect/pull/1198)
- Rename `LocalStorage` to `Local` - [1236](https://github.com/PrefectHQ/prefect/pull/1236)

Contributors

- [Kwangyoun Jung](https://github.com/initialkommit)
- [Anes Benmerzoug](https://github.com/AnesBenmerzoug)

This release fixes a versioning issue caused by a new release of `marshmallow-oneofschema` that is incompatible with the `marshmallow` version pin in 0.5.4.