Metaflow

Features

EFS Volume support for `batch`
This release adds support for mounting existing EFS volumes with `batch`

Usage
You can specify one or multiple volumes to be mounted with the batch decorators `efs_volumes=` attribute. The volumes will be mounted under `/mnt/volume-name` by default, but you can also specify a custom mount point separated by a semicolon with the volume.

Examples
python
batch(efs_volumes="fs-001")
step
...
batch(efs_volumes=["fs-002", "fs-003"])
step
...
batch(efs_volumes="fs-003:/mnt/custom-mountpoint"])
step


Support custom mount point for host volumes with `batch`
This release also adds support for custom mount points for host volumes with `batch`.
Example
python
batch(host_volumes="/home:/mnt/host-homedir")
step


Improvements
`conda` and `pypi` changes
`pypi` can now handle dependencies with special characters in versions

Fixes a `conda` bug where environments were not appending to `PATH` during runtime if the decorator was applied implicitly to the step. This bug affected dependencies that install a binary, causing them to not be found during runtime due to the environment missing from the `PATH`.

Decorator lifecycle improvement
Previously the lifecycle methods `task_pre_step` and `task_decorate` were being called consecutively on a per-decorator basis. This release changes the lifecycle so that `task_pre_step` is called on all decorators before any `task_decorate` is called.

Better checking of datastore dependencies
This release improves the way that datastore dependencies are checked, in order to avoid an issue where previously run metadata could be published to the metadata service, but the execution fails to start due to missing dependencies for pushing data to the datastore.


What's Changed
* [card] Realtime cards (1/N) by valayDave in https://github.com/Netflix/metaflow/pull/1550
* fix rendering of zero-length DataFrames in default card by amerberg in https://github.com/Netflix/metaflow/pull/1652
* fix: pypi decorator handling packages with special characters by saikonen in https://github.com/Netflix/metaflow/pull/1643
* fix: conda environment counts as disabled with implicit decorator by saikonen in https://github.com/Netflix/metaflow/pull/1668
* fix: decorator init order by saikonen in https://github.com/Netflix/metaflow/pull/1669
* don't create run ids if dependencies for data store are missing i.e. fail fast by madhur-ob in https://github.com/Netflix/metaflow/pull/1666
* Bump follow-redirects from 1.15.2 to 1.15.4 in /metaflow/plugins/cards/ui by dependabot in https://github.com/Netflix/metaflow/pull/1675
* [card] Realtime cards (2/N) by valayDave in https://github.com/Netflix/metaflow/pull/1551
* AWS Batch decorator: EFS volumes support (and custom mount point) by enricomarchesin in https://github.com/Netflix/metaflow/pull/1650
* add linting for EFS volume mount by madhur-ob in https://github.com/Netflix/metaflow/pull/1679
* Bump version to 2.10.9 by saikonen in https://github.com/Netflix/metaflow/pull/1680

New Contributors
* madhur-ob made their first contribution in https://github.com/Netflix/metaflow/pull/1666
* enricomarchesin made their first contribution in https://github.com/Netflix/metaflow/pull/1650

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.8...2.10.9

Improvements
- Warnings about regex patterns not being raw strings were fixed.
- An issue with how the environment escape dealt with aliased modules was fixed.
What's Changed
* fix: change regex patterns to raw strings by saikonen in https://github.com/Netflix/metaflow/pull/1645
* raw strings to fix deprecation warnings by russellbrooks in https://github.com/Netflix/metaflow/pull/1646
* Fix an issue with aliases in environment escape server mappings. by romain-intel in https://github.com/Netflix/metaflow/pull/1649

New Contributors
* russellbrooks made their first contribution in https://github.com/Netflix/metaflow/pull/1646

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.7...2.10.8

Improvements

pypi decorator enhancements
This release fixes support for pip environment variables that specify a custom location for the config file (`PIP_CONFIG_FILE` or `PIP_CONFIG`).

The release also adds support for defining a custom index-url through the pip supported environment variable `PIP_INDEX_URL`

What's Changed
* Add deployed-at timestamp in annotations for argo-workflows by rohanrebello in https://github.com/Netflix/metaflow/pull/1633
* fix: pip config issues with pypi decorator by saikonen in https://github.com/Netflix/metaflow/pull/1626
* Bump version to 2.10.7 by saikonen in https://github.com/Netflix/metaflow/pull/1636


**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.6...2.10.7

Improvements

Fix environment activation issue with `pypi` decorator
The `pypi` decorator had a bug that caused it to be treated as `disabled` unless specifically passing `disabled=False` as an attribute to it.
This release fixes the default case so that `pypi` environments activate correctly.

Add debug flag to tracing
This release adds a `METAFLOW_DEBUG_TRACING` environment variable to toggle more verbose output for tracing related issues.

By default any errors related to missing tracing dependencies are now silenced completely, in order to not affect platforms that might want tracing environment variables present for all deployments, whether they have the required dependencies or not.

What's Changed
* fix: output ImportError for tracing module to stderr by saikonen in https://github.com/Netflix/metaflow/pull/1619
* feature: add debug_tracing flag by saikonen in https://github.com/Netflix/metaflow/pull/1621
* fix: pypi environment issues by saikonen in https://github.com/Netflix/metaflow/pull/1623
* Bump version to 2.10.6 by saikonen in https://github.com/Netflix/metaflow/pull/1625


**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.5...2.10.6

What's Changed
* fix nulls in pandas columns on default cards by amerberg in https://github.com/Netflix/metaflow/pull/1606
* CatchDecorator: Fix function signature by tfurmston in https://github.com/Netflix/metaflow/pull/1615
* fix: remove leftover use of LooseVersion by saikonen in https://github.com/Netflix/metaflow/pull/1617
* bump version to 2.10.5 by saikonen in https://github.com/Netflix/metaflow/pull/1618


**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.4...2.10.5

Features
Support for tracing
With this release it is possible to gather telemetry data using an opentelemetry endpoint.

Specifying an endpoint in one of the environment variables
- `METAFLOW_OTEL_ENDPOINT`
- `METAFLOW_ZIPKIN_ENDPOINT`

will enable the corresponding tracing provider.

Some additional dependencies are required for the tracing functionality in the execution environment. These can be installed in the base Docker image, or supplied through a conda environment. The relevant packages are

opentelemetry-sdk, opentelemetry-api, opentelemetry-instrumentation, opentelemetry-instrumentation-requests

and depending on your endpoint, either `opentelemetry-exporter-otlp` or `opentelemetry-exporter-zipkin`

Custom index support for the `pypi` decorator
The `pypi` decorator now supports using a custom index in the users Pip configuration under `global.index-url`.
This enables using private indices, even ones that require authentication.

For example the following would set up one authenticated and two extra non-authenticated indices for package resolution
sh
pip config set global.index-url "https://user:tokenexample.com"
pip config set global.extra-index-url "https://extra.example.com https://extra2.example.com"


Specify Kubernetes job ephemeral storage size through `resources` decorator
It is now possible to specify the ephemeral storage size for Kubernetes jobs when using the `resources` decorator with the `disk=` attribute.

Introduce `argo-workflows status` command
Adds a command for easily checking the current status of a workflow on Argo workflows.
sh
python flow.py argo-workflows status [run-id]


Improvements

Add more randomness to Kubernetes pod names to avoid collisions
There was an issue where relying solely on the Kubernetes apiserver for generating random pod names was resulting in significant collisions with sufficiently large number of executions.

This release adds more randomness to the pod names besides what is generated by Kubernetes.

Fix issues with `resources` decorator in combination with step functions
This release fixes an issue where deploying flows on AWS Step Functions was failing in the following cases
- `resources(shared_memory=)` with any value
- combining `resources` and `batch(use_tmpfs=True)`


What's Changed
* Bump postcss from 8.4.24 to 8.4.31 in /metaflow/plugins/cards/ui by dependabot in https://github.com/Netflix/metaflow/pull/1599
* introduce argo-workflows status by savingoyal in https://github.com/Netflix/metaflow/pull/1600
* remove dependency on wget for micromamba by savingoyal in https://github.com/Netflix/metaflow/pull/1601
* fix: resource decorator step functions issue by saikonen in https://github.com/Netflix/metaflow/pull/1610
* feature: add status grouping to argo-workflows output by saikonen in https://github.com/Netflix/metaflow/pull/1604
* fix: apply only supported attributes from resources decorator in batch by saikonen in https://github.com/Netflix/metaflow/pull/1586
* fix: use correct deco for limiting keys. by saikonen in https://github.com/Netflix/metaflow/pull/1611
* Otel oss integration by wangchy27 in https://github.com/Netflix/metaflow/pull/1462
* Remove use of distutils from code (not setup.py yet) by romain-intel in https://github.com/Netflix/metaflow/pull/1585
* Add a uid for the kubernetes job creation by tylerpotts in https://github.com/Netflix/metaflow/pull/1588
* feature: support custom index config for pypi decorator by saikonen in https://github.com/Netflix/metaflow/pull/1613
* Adding 'disk' parameter to resources decorator by jaypond in https://github.com/Netflix/metaflow/pull/1500
* Bump version to 2.10.4 by saikonen in https://github.com/Netflix/metaflow/pull/1614

New Contributors
* jaypond made their first contribution in https://github.com/Netflix/metaflow/pull/1500

**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.10.3...2.10.4