Metaflow

What's Changed
* Fix escape hatch to make __getattr__ behavior more standard by romain-intel in https://github.com/Netflix/metaflow/pull/1883
* Fix/remember namespace by romain-intel in https://github.com/Netflix/metaflow/pull/1873
* Remove global nature of parameters and flow decorators by romain-intel in https://github.com/Netflix/metaflow/pull/1886
* Install PyPI packages in interpreter's site packages by savingoyal in https://github.com/Netflix/metaflow/pull/1890
* bump version to 2.12.4 by saikonen in https://github.com/Netflix/metaflow/pull/1891


**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.12.3...2.12.4

What's Changed
* Fix default azure client provider setting by oavdeev in https://github.com/Netflix/metaflow/pull/1877

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.12.2...2.12.3

What's Changed
* Fix issue with setting metadata when using the runner by romain-intel in https://github.com/Netflix/metaflow/pull/1875
* Bump version to 2.12.2 by romain-intel in https://github.com/Netflix/metaflow/pull/1876


**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.12.1...2.12.2

Features

Configurable default decorators
This release adds the ability to configure default decorators that will be applied to all steps. This is achieved by setting the decospecs as a value (space separated) for `METAFLOW_DECOSPECS` either as an environment variable or in a config.json

The following example would add retry and kubernetes decorators with a custom memory value to all steps:

export METAFLOW_DECOSPECS="kubernetes:memory=4096 retry"


Defining a decorator with the `--with` keyword will override the defaults configured. Same applies for explicitly adding a decorator in the flow file.

Improvements

Correctly clean up Argo Workflow sensors when using `project`
This release fixes an issue where `argo-workflows delete` did not correctly remove possible sensors associated with the workflow if the workflow used the `project` decorator.

What's Changed
* Add the possibility of defining default decorators for steps by romain-intel in https://github.com/Netflix/metaflow/pull/1837
* bugfix: properly deletes Argo Events trigger sensors when `project` is used by gabriel-rp in https://github.com/Netflix/metaflow/pull/1871
* S3PubObject was not used properly after 1807 by romain-intel in https://github.com/Netflix/metaflow/pull/1872
* bump version to 2.12.1 by saikonen in https://github.com/Netflix/metaflow/pull/1874

New Contributors
* gabriel-rp made their first contribution in https://github.com/Netflix/metaflow/pull/1871

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.12.0...2.12.1

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.11.16...2.12.0

Features
Support GCP Secret Manager
This release adds support for using GCP Secret manager to supply secret values for steps environment.

In order to enable the secret manager, you should specify the type by setting `METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE` to `gcp-secret-manager` or specifying it in the decorator

python
secrets(sources=[{"type": "gcp-secret-manager", "id": "some-secret-key"}])


`METAFLOW_GCP_SECRET_MANAGER_PREFIX` can be set in order to not have to write full secret locations.

Support Azure Key Vault
This release also adds support for Azure Key Vault as a secrets backend. Specify `az-key-vault` as the secret backend type to use this.

Same as with the other secret managers, we provide a prefix config to avoid having to repeat common parts in the secret keys. Configure this by setting `METAFLOW_AZURE_KEY_VAULT_PREFIX`

**Note:** Currently only `Secret` object types are supported when using Azure Key Vault.

`parallel` for Kubernetes
This release adds support for `parallel` when flows are run `--with kubernetes`

Example:

python
step
def start(self):
self.next(self.parallel_step, num_parallel=3)

kubernetes(cpu=1, memory=512)
parallel
step
def parallel_step(self):
...


Configurable runtime limits
It is now possible to configure the default timeout for the `timeout` decorator. This can be done by setting `METAFLOW_DEFAULT_RUNTIME_LIMIT` in the environment, or in a config.json

Improvements

Resumed flows should record task competions correctly
Fixes an issue where tasks that were cloned from a previous run by `resume` would not show up as completed on the Metaflow UI due to missing metadata

Fix accessing task index of a foreach task
There was an issue accessing the index of a foreach task via the client. With this release it is possible to do the following
python
from metaflow import Task
task = Task("ForeachFlow/123/foreach_step/task-00000000")
task.index


What's Changed
* [runtime limits] make runtime limits configurable by valayDave in https://github.com/Netflix/metaflow/pull/1834
* [Ready for Review] fix bug where client can not access foreach stack by darinyu in https://github.com/Netflix/metaflow/pull/1766
* [Ready for Review] add attempt_ok flag so that UI will not show up unknown node by darinyu in https://github.com/Netflix/metaflow/pull/1830
* pluggable azure credentials provider by oavdeev in https://github.com/Netflix/metaflow/pull/1756
* Secret Backend Support for Azure Key Vault by iamsgarg-ob in https://github.com/Netflix/metaflow/pull/1839
* reducing the dep version to 4.7.0 (47) by iamsgarg-ob in https://github.com/Netflix/metaflow/pull/1840
* [`parallel` on Kubernetes] support for Jobsets by valayDave in https://github.com/Netflix/metaflow/pull/1804
* Pluggable GCP auth by oavdeev in https://github.com/Netflix/metaflow/pull/1841
* GCP secret manager support by oavdeev in https://github.com/Netflix/metaflow/pull/1842
* [jobsets] py3.5 compatibility fixes. by valayDave in https://github.com/Netflix/metaflow/pull/1844
* Support Python 3.5 for tests by savingoyal in https://github.com/Netflix/metaflow/pull/1843
* py3.5 compatibility fixes [azure/gcp/jobsets] by valayDave in https://github.com/Netflix/metaflow/pull/1848
* [version bump] for release by valayDave in https://github.com/Netflix/metaflow/pull/1847
* [OB-625] adding metaflow/cron annotation to argo workflows by iamsgarg-ob in https://github.com/Netflix/metaflow/pull/1852

New Contributors
* iamsgarg-ob made their first contribution in https://github.com/Netflix/metaflow/pull/1839

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.11.15...2.11.16