Pypyr

Latest version: v5.9.1

Safety actively analyzes 638699 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 3 of 14

5.0.0

Implement [adr2 relative pipelines + api changes](docs/adr/0002-relative-pipelines-api-changes.md).

In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

Breaking Changes
This is a major version increment because it comes with BREAKING CHANGES:
1. API: `pipelinerunner.run()` replaces both `pipelinerunner.main()` and `pipelinerunner.main_with_context()`
2. API: `def get_pipeline_definition(pipeline_name, working_directory)` signature for custom pype loaders changes to `def get_pipeline_definition(pipeline_name, parent)`
3. CLI: the `—dir` flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
4. Final removal of deprecated `get_formatted_iterable`, `get_formatted_string` 195 & `pypyr.steps.contextset` 184. Where previously these would just give deprecation warnings, they are now completely removed.
5. `pypyr.pypeloaders.fileloader` renamed `pypyr.loaders.file`

Non-Breaking Changes
- You can now access the current pipeline’s metadata & loader information from within a pipeline with `context.current_pipeline`
- Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
- Typing support added for the pypyr API entrypoint.

Detailed Technical Breakdown:
- Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
- `PipelineInfo` - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
- `PipelineDefinition` - this wraps the pipeline payload and its metadata (`PipelineInfo`) to allow pypyr to cache it all with one reference
- Add new `Pipeline` class for the run-time properties of a single run.
- The `Pipeline` references the shared cached `PipelineDefinition`.
- Move run + load_and_run logic from pypyr.pipelinerunner to the new `Pipeline` class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared `Pipeline` state rather than sling a bunch of args between different functions as before.
- Add a call-stack of running `Pipeline` instances on `Context`. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via `pype`
- Add `current_pipeline` attribute to `Context`, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
- This means that steps can access current pipeline’s properties.
- This allows `pypyr.steps.pype` to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
- When a child pipeline completes, the calling pipeline (i.e the previous `Pipeline` in the call-stack) becomes the current pipeline
- Amend `pypyr.steps.pype` to instantiate `Pipeline` object to `load_and_run()` child.
- `pype` now deals the `context.current_pipeline.pipeline_definition.info` metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
- Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
- Given the context manager controlling current pipeline scope in `Pipeline.load_and_run_pipeline` remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
- Remove global PipelineCache. Replace with distinct pipeline cache per loader.
- This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
- The per-loader pipeline cache stores `PipelineDefinition` objects.
- Introduce `Loader` class, which wraps loader & its pipeline cache. `Loader` is what the `LoaderCache` caches.
- Thus `LoaderCache` -> `Loader` -> `_pipelineCache` -> `PipelineDefinition`
- File loader has a private file cache keyed on absolute path of file
- This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
- e.g `(name=‘dir/mypipe’, parent=None)` and `(name=‘mypipe’, parent=‘dir’)` both resolve to `dir/mypipe.yaml`
- Caching a reference to the `PipelineDefinition` object, so not duplicating memory
- Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
- File loader `get_pipeline_definition` now returns a `PipelineDefinition` with `PipelineFileInfo` to store file-system specific metadata for the loader pipeline
- Remove `working_dir` global. The `py_dir` input on `run()` now refers ONLY to module paths, NOT pipeline locations.
- The CLI `—dir` flag, or `py_dir` input on `run()` basically adds the specified directory to `sys.path`.
- Add current pipeline’s parent directory to `sys.path` on load. This allows child pipelines to resolve custom modules relative to itself.
- `pypyr.dsl.Step` does not need `StepsRunner` anymore, because it can get it from the `context.current_pipeline` instead.
- Recode (some) integration tests to take advantage of list `pypyr.steps.append` step and checking that for output on return context rather than intercepting logger.NOTIFY.
- Rename master branch to main in CI/CD GitHub actions
- Add typing annotations to the public `run()` function and the `Pipeline` class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include `py.typed` in `pypyr` package.
- Remedy packaging snafu where `tests.common` was deploying alongside pypyr because exclude condition in `find_packages` didn't include wildcard for sub packages.

What's Changed
* Relative pipelines & API run() replaces main/main_with_context by yaythomas in https://github.com/pypyr/pypyr/pull/243


**Full Changelog**: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0

4.6.0

What's Changed
* Fix failing ops/build (failing on linting issues and locale dependent failure) by vlcinsky in https://github.com/pypyr/pypyr/pull/226
* Evaluate step skip only if the run check returned true by Reskov in https://github.com/pypyr/pypyr/pull/234
* pypyr.steps.append & pypyr.steps.add by yaythomas in https://github.com/pypyr/pypyr/pull/235
* shorter alias `set` for `contextsetf`. py 3.10. by yaythomas in https://github.com/pypyr/pypyr/pull/238

New Contributors
* vlcinsky made their first contribution in https://github.com/pypyr/pypyr/pull/226

**Full Changelog**: https://github.com/pypyr/pypyr/compare/v4.5.0...v4.6.0

4.5.0

- add retry backoff strategies. 216.
- prevent duplicates in `sys.path` (218) on main* api entry points. thread-safe existence check on `sys.path` list.
- prevent redundant multiple addition of notify log-level on main* api entry. Move initialization code to package `__init__` (219).

4.4.1

- Context parsers that create an entry in context now initialise to empty rather than `None`. This means you can directly use something like `{argList}`, `{argDict}` and `argString` (initialising respectively to `[]`, `{}`, `''`) directly for things like `foreach` loops without having to worry about `None` checks. Your existing truthy checks for these values will work as before. closes 214.

4.4.0

- New step `pypyr.steps.pyimport` to import references to the py-string namespace.
- This includes an underlying api signature change by removal of `pypyr.utils.expressions.eval_string()`, but this is sufficiently far down the call-chain that it shouldn’t affect any normal pipeline operator or api consumer.
- See 110 for details.
- `pypyr.steps.contextclearall` wipes `pyimport` imported references in addition to the key/values inside context.
- Simplify `pypyr.steps.py` syntax by allowing a new `py` (rather than `pycode`) input. This allows pipeline authors to use context key names directly as variables, rather than have to specify them as keys in context (`my_var` vs `context[‘my_var’]`).
- see 204 for details on simplifying the `pypyr.steps.py` syntax.
- the old `pycode` will keep on working in the same way, so no need to worry about backwards compatibility for your existing pipelines.
- Allow substitutions on Retry `max`. Resolves 207. Excellent bug find & fix by Reskov, much thanks 🙌 🙌 🙌 as ever for a superb contribution! 🔥 🔥 🔥
- `foreach` can now use any iterable, including generators. Closes 209

4.3.0

- Streamline main entrypoint API. close 201.
- `main()` allows consumer to set pype loader, rather than having to drop further down into api to `load_and_run_pipeline()`
- new `main_with_context()` to input dict to initialise context, and bypass context_parser entirely. Also returns the `Context` object after pipeline run completes.
- make all non-essential args optional to allow minimal calls to main entrypoint without having to add `optional=None` style inputs.
- This is fully backwards compatible.
- `pypyr.steps.pype`
- defaults `useParentContext` to `False` is `pipeArgs` specified.
- `pipeArgs` shlex-es input string
- set `pipeline_name` on child pipeline rather than use parent pipeline name
- `working_dir` uses `Path` object rather than string
- `pypyr.steps.echo` remove redundant string check.

Page 3 of 14

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.