Features
Displaying task attempt logs
When running a task with the `retry` decorator, previously we were able to only view the logs of the latest attempt of the task.
With this release it is now possible to target a specific attempt with the `--attempt` option for the logs command
python example.py logs 123/retry_step/456 --attempt 2
Scrubbing log contents
This release introduces a new command for scrubbing log contents of tasks in case they contain sensitive information that needs to be redacted.
Simplest use case is scrubbing the latest task logs. By default both `stdout` and `stderr` are scrubbed
python example.py logs scrub 123/example/456
There are also options to target only a specific log stream
python example.py logs scrub 123/example/456 --stderr
python example.py logs scrub 123/example/456 --stdout
when using the`retry` decorator, tasks can have multiple attempts with separate logs that require scrubbing. By default only the latest attempt is scrubbed. There are options to make scrubbing multiple attempts easier
scrub specific attempt
python example.py logs scrub 123/retry_step/456 --attempt 1
scrub all attempts
python example.py logs scrub 123/retry_step/456 --all
scrub specified attempt and all prior to it (this would scrub attempts 0,1,2,3)
python example.py logs scrub 123/retry_step/456 --all --attempt 3
The command also accepts only specifying a step for scrubbing. This is useful for steps with multiple tasks, like a foreach split.
python example.py logs scrub 123/foreach_step
all the above options also apply when targeting a step for scrubbing.
*Note*: Log scrubbing for running tasks is not recommended, and is actively protected against. There can be occasions where a task has failed in such a way that it still counts as not completed. For such a case you can supply the `--include-not-done` option to try and scrub it as well.
What's Changed
* feature: scrub logs by saikonen in https://github.com/Netflix/metaflow/pull/1802
* Fix issue 1805 by narayanacharya6 in https://github.com/Netflix/metaflow/pull/1807
* bug fix: unused arg for batch step --help by mae5357 in https://github.com/Netflix/metaflow/pull/1817
* bump version to 2.11.15 by saikonen in https://github.com/Netflix/metaflow/pull/1829
New Contributors
* narayanacharya6 made their first contribution in https://github.com/Netflix/metaflow/pull/1807
* mae5357 made their first contribution in https://github.com/Netflix/metaflow/pull/1817
**Full Changelog**: https://github.com/Netflix/metaflow/compare/2.11.14...2.11.15