Implement [adr2 relative pipelines + api changes](docs/adr/0002-relative-pipelines-api-changes.md).
In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.
Breaking Changes
This is a major version increment because it comes with BREAKING CHANGES:
1. API: `pipelinerunner.run()` replaces both `pipelinerunner.main()` and `pipelinerunner.main_with_context()`
2. API: `def get_pipeline_definition(pipeline_name, working_directory)` signature for custom pype loaders changes to `def get_pipeline_definition(pipeline_name, parent)`
3. CLI: the `—dir` flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
4. Final removal of deprecated `get_formatted_iterable`, `get_formatted_string` 195 & `pypyr.steps.contextset` 184. Where previously these would just give deprecation warnings, they are now completely removed.
5. `pypyr.pypeloaders.fileloader` renamed `pypyr.loaders.file`
Non-Breaking Changes
- You can now access the current pipeline’s metadata & loader information from within a pipeline with `context.current_pipeline`
- Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
- Typing support added for the pypyr API entrypoint.
Detailed Technical Breakdown:
- Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
- `PipelineInfo` - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
- `PipelineDefinition` - this wraps the pipeline payload and its metadata (`PipelineInfo`) to allow pypyr to cache it all with one reference
- Add new `Pipeline` class for the run-time properties of a single run.
- The `Pipeline` references the shared cached `PipelineDefinition`.
- Move run + load_and_run logic from pypyr.pipelinerunner to the new `Pipeline` class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared `Pipeline` state rather than sling a bunch of args between different functions as before.
- Add a call-stack of running `Pipeline` instances on `Context`. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via `pype`
- Add `current_pipeline` attribute to `Context`, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
- This means that steps can access current pipeline’s properties.
- This allows `pypyr.steps.pype` to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
- When a child pipeline completes, the calling pipeline (i.e the previous `Pipeline` in the call-stack) becomes the current pipeline
- Amend `pypyr.steps.pype` to instantiate `Pipeline` object to `load_and_run()` child.
- `pype` now deals the `context.current_pipeline.pipeline_definition.info` metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
- Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
- Given the context manager controlling current pipeline scope in `Pipeline.load_and_run_pipeline` remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
- Remove global PipelineCache. Replace with distinct pipeline cache per loader.
- This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
- The per-loader pipeline cache stores `PipelineDefinition` objects.
- Introduce `Loader` class, which wraps loader & its pipeline cache. `Loader` is what the `LoaderCache` caches.
- Thus `LoaderCache` -> `Loader` -> `_pipelineCache` -> `PipelineDefinition`
- File loader has a private file cache keyed on absolute path of file
- This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
- e.g `(name=‘dir/mypipe’, parent=None)` and `(name=‘mypipe’, parent=‘dir’)` both resolve to `dir/mypipe.yaml`
- Caching a reference to the `PipelineDefinition` object, so not duplicating memory
- Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
- File loader `get_pipeline_definition` now returns a `PipelineDefinition` with `PipelineFileInfo` to store file-system specific metadata for the loader pipeline
- Remove `working_dir` global. The `py_dir` input on `run()` now refers ONLY to module paths, NOT pipeline locations.
- The CLI `—dir` flag, or `py_dir` input on `run()` basically adds the specified directory to `sys.path`.
- Add current pipeline’s parent directory to `sys.path` on load. This allows child pipelines to resolve custom modules relative to itself.
- `pypyr.dsl.Step` does not need `StepsRunner` anymore, because it can get it from the `context.current_pipeline` instead.
- Recode (some) integration tests to take advantage of list `pypyr.steps.append` step and checking that for output on return context rather than intercepting logger.NOTIFY.
- Rename master branch to main in CI/CD GitHub actions
- Add typing annotations to the public `run()` function and the `Pipeline` class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include `py.typed` in `pypyr` package.
- Remedy packaging snafu where `tests.common` was deploying alongside pypyr because exclude condition in `find_packages` didn't include wildcard for sub packages.
What's Changed
* Relative pipelines & API run() replaces main/main_with_context by yaythomas in https://github.com/pypyr/pypyr/pull/243
**Full Changelog**: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0