Prefect

Features

- Add support for rich iCal style scheduling via RRules - [4901](https://github.com/PrefectHQ/prefect/pull/4901)
- Add Google Cloud Vertex agent and run configuration - [4989](https://github.com/PrefectHQ/prefect/pull/4989)

Enhancements

- Allow `Azure` flow storage to overwrite existing blobs - [5103](https://github.com/PrefectHQ/prefect/pull/5103)
- Provide option to specify a dockerignore when using Docker storage - [4980](https://github.com/PrefectHQ/prefect/pull/4980)
- Add keep-alive connections for kubernetes client API connections - [5066](https://github.com/PrefectHQ/prefect/pull/5066)
- Add `idempotency_key` to `create_flow_run` task - [5125](https://github.com/PrefectHQ/prefect/pull/5125)
- Add `raise_final_state` to `wait_for_flow_run` task to reflect child flow run state - [5129](https://github.com/PrefectHQ/prefect/pull/5129)

Task Library

- Bump maximum `google-cloud-bigquery` version to support 2.x - [5084](https://github.com/PrefectHQ/prefect/pull/5084)
- Add `Glob` task for collecting files in directories - [5077](https://github.com/PrefectHQ/prefect/pull/5077)
- Add `DbtCloudRunJob` task for triggering dbt cloud run jobs - [5085](https://github.com/PrefectHQ/prefect/pull/5085)
- Added Kafka Tasks entry to website docs - [5094](https://github.com/PrefectHQ/prefect/pull/5094)

Fixes

- Update the `FlowView` to be more robust to serialized flow changes in the backend - [5116](https://github.com/PrefectHQ/prefect/pull/5116)

Deprecations

- Move artifacts functions to `prefect.backend.artifacts` - [5117](https://github.com/PrefectHQ/prefect/pull/5117)

Server

This release includes a Prefect Server update that updates an upstream dependency to fix a security vulnerability. See the [release changelog](https://github.com/PrefectHQ/server/blob/master/Changelog.md#november-09-2021-) for more details.

Contributors

- [Alessandro Lollo](https://github.com/AlessandroLollo)
- [Bradley Axen](https://github.com/baxen)
- [Damien Ramunno-Johnson](https://github.com/damienrj)
- [Jonas Miederer](https://github.com/jonasmiederer)
- [Josh Wang](https;//github.com/wangjoshuah)
- [Nitay Joffe](https://github.com/nitay)
- [Timo S.](https://github.com/sti0)
- [Brett Naul](https://github.com/bnaul)

Enhancements

- Add flatten support to `apply_map` - [4996](https://github.com/PrefectHQ/prefect/pull/4996)
- Add dask performance report to `DaskExecutor` - [5032](https://github.com/PrefectHQ/prefect/pull/5032)
- Update git storage `repo` parameter to be optional if specifying `git_clone_url_secret_name` - [5033](https://github.com/PrefectHQ/prefect/pull/5033)
- Add `task_run_name` to `prefect.context` - [5055](https://github.com/PrefectHQ/prefect/pull/5055)

Fixes

- Reduce rate limit related failures with the ECS agent - [5059](https://github.com/PrefectHQ/prefect/pull/5059)

Task Library

- Add `data` parameter to `SQLiteQuery` task - [4981](https://github.com/PrefectHQ/prefect/pull/4981)
- Allow `EmailTask` to use insecure internal SMTP servers with `smtp_type="INSECURE"` - [5012](https://github.com/PrefectHQ/prefect/pull/5012)
- Fix Databricks `run_id` mutation during task runs - [4958](https://github.com/PrefectHQ/prefect/issues/4958)
- Add `manual` setting to `FivetranSyncTask` allowing retention of Fivetan scheduling -[5065](https://github.com/PrefectHQ/prefect/pull/5065)

Contributors

- [Daniel Saxton](https://github.com/dsaxton)
- [Emre Akgün](https://github.com/Fraznist)
- [Jessica Smith](https://github.com/NodeJSmith)

Enhancements

- Improve setting the Azure storage connection string - [4955](https://github.com/PrefectHQ/prefect/pull/4955)
- Allow disabling retries for a task with `max_retries=0` when retries are globally configured - [4971](https://github.com/PrefectHQ/prefect/pull/4971)

Task Library

- Allow Exasol Tasks to handle Prefect Secrets directly - [4436](https://github.com/PrefectHQ/prefect/pull/4436)
- Adding [Census](https://www.getcensus.com/) Syncs to the task library - [#4935](https://github.com/PrefectHQ/prefect/pull/4935)

Fixes

- Fix bug where `LocalDaskExecutor` did not respond to a `PrefectSignal` - [4924](https://github.com/PrefectHQ/prefect/pull/4924)
- Fix `PostgresFetch` with headers for one row - [4968](https://github.com/PrefectHQ/prefect/pull/4968)
- Fix bug where `apply_map` could create acyclic flows - [4970](https://github.com/PrefectHQ/prefect/pull/4970)

Contributors

- [Donny Flynn](https://github.com/dflynn20)
- [Noah Holm](https://github.com/noppaz)
- [Timo S.](https://github.com/sti0)

Released on September 2, 2021.

Features

- Python 3.9 docker images are now published - [4896](https://github.com/PrefectHQ/prefect/pull/4896)

Enhancements

- Add `--expose` flag to `prefect server` cli to make the Core server and UI listen to all interfaces - [4821](https://github.com/PrefectHQ/prefect/pull/4821)
- Pass existing/local environment variables to agentless flow runs - [4917](https://github.com/PrefectHQ/prefect/pull/4917)
- Add `--idempotency-key` to `prefect run` - [4928](https://github.com/PrefectHQ/prefect/pull/4928)
- Add support for larger flow registration calls - [4930](https://github.com/PrefectHQ/prefect/pull/4930)
- Ignore schedules by default for CLI flow runs and add flag to run based on schedule for local only runs [4817](https://github.com/PrefectHQ/prefect/pull/4817)

Task Library

- Feature: Added `SnowflakeQueryFromFile` task [3744](https://github.com/PrefectHQ/prefect/pull/4363)
- Enhancement: Log boto exceptions encountered in the in AWS `BatchSubmit` task - [4771](https://github.com/PrefectHQ/prefect/pull/4771)
- Breaking: Legacy Dremio authentication has been updated to the new pattern in `DremioFetch` - [4872](https://github.com/PrefectHQ/prefect/pull/4872)
- Fix: Use runtime arguments over init arguments instead of ignoring them for MySQL Tasks - [4907](https://github.com/PrefectHQ/prefect/pull/4907)
Fixes

- Adjust log limits to match backend logic for better UX - [4900](https://github.com/PrefectHQ/prefect/pull/4900)
- Fix use of `marshmallow.fields.Dict` to use `keys` as a kwarg rather than `key`. - [4903](https://github.com/PrefectHQ/prefect/pull/4903)
- API server settings are passed correctly to task workers when using Prefect Server - [4914](https://github.com/PrefectHQ/prefect/pull/4914)
- Do not attempt attempt to set `host_gateway` if using an unsupported Docker Engine version - [4809](https://github.com/PrefectHQ/prefect/pull/4809)
- Ignore jobs without a `flow_run_id` label in `KubernetesAgent.manage_jobs` - [4934](https://github.com/PrefectHQ/prefect/pull/4934)
Breaking Changes

- Services run by `prefect server` cli are now local by default (listen to localhost instead of 0.0.0.0); use `--expose` if you want to connect from a remote location - [4821](https://github.com/PrefectHQ/prefect/pull/4821)
- The changes in flow registration require Prefect Server 2021.09.02. Prefect Server will need to be upgraded before flows can be registered from this version - [4930](https://github.com/PrefectHQ/prefect/pull/4930)

Contributors

- [Deepyaman Datta](https://github.com/deepyaman)
- [Henri Hannetel](https://github.com/HenriTEL)
- [Johnny Tirado](https://github.com/jclocks)
- [Kathryn Klarich](https://github.com/klarich)
- [Tenzin Choedak](https://github.com/tchoedak)
- [Vincent Xue](https://github.com/xuevin)

Released on August 17, 2021.

Docs

- Add a getting started section with a quick start guide for both core and orchestration sections - [4734](https://github.com/PrefectHQ/prefect/pull/4734)
Enhancements

- Expose Snowflake cursor type to SnowflakeQuery task arguments [4786](https://github.com/PrefectHQ/prefect/issues/4786)
- Add ability to use threaded flow heartbeats - [4844](https://github.com/PrefectHQ/prefect/pull/4844)
- Improve behavior when API rate limits are encountered - [4852](https://github.com/PrefectHQ/prefect/pull/4852)
- Allow custom git clone url for `Git` storage - [4870](https://github.com/PrefectHQ/cloud/pull/4870)
- Add `on_worker_status_changed` callback to the `DaskExecutor` - [4874](https://github.com/PrefectHQ/prefect/pull/4874)
- Add `--agent-config-id` to `prefect agent <kubernetes|local> install` - [4876](https://github.com/PrefectHQ/prefect/pull/4876)

Task Library

- Add new prometheus task to push to gateway - [4623](https://github.com/PrefectHQ/prefect/pull/4623)

Fixes

- Fix binding of named volumes to flow containers with Docker agent - [4800](https://github.com/PrefectHQ/prefect/pull/4800)
- Fix `ImportError` typo in dropbox module - [4855](https://github.com/PrefectHQ/prefect/pull/4855)
- Fix default safe char for gitlab storage repo path - [4828](https://github.com/PrefectHQ/prefect/pull/4828)

Contributors

- [Austen Bouza](https://github.com/austen-bouza)
- [David Zucker](https://github.com/davzucky)
- [Jacob Dawang](https://github.com/jdawang)
- [Tom Forbes](https://github.com/orf)
- [Mat](https://github.com/matmiad)

Enhancements

- Add new `evaluation_parameters` parameter to `RunGreatExpectationsValidation` task - [4798](https://github.com/PrefectHQ/prefect/pull/4798)

Fixes

- Fix `create_flow_run` compatibility with auth tokens - [4801](https://github.com/PrefectHQ/prefect/pull/4801)
- Fix auto-quoting for strings that begin with numeric characters - [4802](https://github.com/PrefectHQ/prefect/pull/4802)

Contributors

- [Michal Zawadzki](https://github.com/trymzet)