Zenml

ZenML release 0.40.0 introduces two big updates: a fresh and more flexible pipeline interface and a new way to connect and authenticate with external services in ZenML Connectors. See below for full details on these two major new sets of functionality.

The release also contains many bug fixes and quality-of-life improvements. Specifically, we reworked our documentation from the ground up with particular attention to the structure to help you find what you need. Our [Label Studio integration example](https://github.com/zenml-io/zenml/tree/main/examples/label_studio_annotation) is now working again and allows you to use more recent versions of the `label-studio` package that backs it.

A Fresh Pipeline Interface

This release introduces a completely reworked interface for developing your ZenML steps and pipelines:

No More BaseParameters Class

In the previous version, you had to define a separate class for step parameters using `BaseParameters`. This is no longer necessary, although it is still supported for backward compatibility. You can now pass parameters directly in the step function:

python
step
def trainer(data: pd.Dataframe, lr: float = 0.1, gamma: Optional[float] = 0.02) -> ...:
print(lr)
print(gamma)


Simplified Pipeline Execution

With the new changes, you no longer need to create a pipeline instance and then run it separately. You can now pass parameters directly at pipeline instance creation and execute the pipeline in a single step:

python
my_pipeline(lr=0.000001)


Multiple step invocations

You can now call steps multiple times inside a pipeline, allowing you to create more complex workflows and reuse steps with different parameters:

python
pipeline
def my_pipeline(step_count: int) -> None:
data = load_data_step()
after = []
for i in range(step_count):
train_step(data, learning_rate=i * 0.0001, id=f"train_step_{i}")
after.append(f"train_step_{i}")
model = select_model_step(..., after=after)


Pipeline inputs and outputs

Pipelines can now define inputs and outputs, providing a clearer interface for working with data and dependencies between pipelines:

python
pipeline(enable_cache=False)
def subpipeline(pipeline_param: int):
out = step_1(k=None)
step_2(a=3, b=pipeline_param)
return 17


Call pipelines from within pipelines (alpha)

You can now call pipelines within other pipelines. This currently does not execute the inner pipeline but instead adds its steps to the parent pipeline, allowing you to create modular and reusable workflows:

python
pipeline(enable_cache=False)
def my_pipeline(a: int = 1):
p1_output = subpipeline(pipeline_param=22)
step_2(a=a, b=p1_output)


Flexible types on steps

Increased flexibility when defining steps: Steps can now have `Optional`, `Union`, and `Any` type annotations for their inputs and outputs. Additionally, default values are allowed for step inputs.

python
step
def trainer(data: pd.Dataframe, start_model: Union[svm.SVC, svm.SVR], coef0: Optional[int] = None) -> Any:
pass


Easy to debug steps

You can now easily run a step outside of a pipeline, making it easier to test and debug your code:

python
trainer(data=pd.Dataframe(...), start_model=svc.SVC(...))


External Artifacts

External artifacts can be used to pass values to steps that are not produced by an upstream step. This provides more flexibility when working with external data or models:

python
from zenml.steps import ExternalArtifact

pipeline
def my_pipeline(lr: float):
data = process_data()
trainer(data=data, start_model=ExternalArtifact(svc.SVC(...)))


Get Started with the new interface and features!

To get started, simply import the new `step` and `pipeline` decorator and check out our new [starter guide](https://docs.zenml.io/user-guide/starter-guide) for more information.

python
from zenml import step, pipeline

step
def my_step(...):
...

pipeline
def my_pipeline(...):
...


The old pipeline and step interface is still working using the imports from previous ZenML releases but is deprecated and will be removed in the future.

'Connectors' for authentication

In this update, we're pleased to present a new feature to ZenML: Service Connectors. The intention behind these connectors is to offer a reliable and more user-friendly method for integrating ZenML with external resources and services. We aim to simplify processes such as validating, storing, and generating security-sensitive data, along with the authentication and authorization of access to external services. We believe ZenML Service Connectors will be a useful tool to alleviate some of the common challenges in managing pipeline across various Stack Components.

Regardless of your background in infrastructure management - whether you're a beginner looking for quick cloud stack integration, or an experienced engineer focused on maintaining robust infrastructure security practices - our Service Connectors are designed to assist your work while maintaining high security standards.

Here are just a few ways you could use ZenML Service Connectors:

- Easy utilization of cloud resources: With ZenML's Service Connectors, you can use resources from AWS, GCP, and Azure without the need for extensive knowledge of cloud infrastructure or environment configuration. All you'll need is a ZenML Service Connector and a few Python libraries.
- Assisted setup with security in mind: Our Service Connectors come with features for configuration validation and verification, the generation of temporary, low-privilege credentials, and pre-authenticated and pre-configured clients for Python libraries.
- Easy local configuration transfer: ZenML's Service Connectors aim to resolve the reproducibility issue in ML pipelines. They do this by automatically transferring authentication configurations and credentials from your local machine, storing them securely, and allowing for effortless sharing across different environments.

[Visit our documentation pages](https://docs.zenml.io/platform-guide/set-up-your-mlops-platform/connect-to-your-cloud-provider) to learn more about ZenML Connectors and how you can use them in a way that supports your ML workflows.

What's Changed

* Cleanup remaining references of `zenml.artifacts` by fa9r in https://github.com/zenml-io/zenml/pull/1534
* Upgrading the `black` version by bcdurak in https://github.com/zenml-io/zenml/pull/1535
* Remove dev breakpoints by strickvl in https://github.com/zenml-io/zenml/pull/1540
* Removing old option command from contribution doc by bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1544
* Improve CLI help text for `zenml integration install -i ...` by strickvl in https://github.com/zenml-io/zenml/pull/1545
* Fix RestZenStore error handling for list responses by fa9r in https://github.com/zenml-io/zenml/pull/1539
* Simplify Dashboard UX via `zenml.show()` by fa9r in https://github.com/zenml-io/zenml/pull/1511
* Removed hardcoded variable by bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1543
* Revert Quickstart Changes by fa9r in https://github.com/zenml-io/zenml/pull/1546
* Deprecate some long overdue functions by AlexejPenner in https://github.com/zenml-io/zenml/pull/1541
* ZenML Connectors by stefannica in https://github.com/zenml-io/zenml/pull/1514
* Fix automatic dashboard opening after `zenml up` by fa9r in https://github.com/zenml-io/zenml/pull/1551
* Update Neptune README by strickvl in https://github.com/zenml-io/zenml/pull/1554
* Update example READMEs following deployment PR by strickvl in https://github.com/zenml-io/zenml/pull/1555
* Fix and update Label Studio example by strickvl in https://github.com/zenml-io/zenml/pull/1542
* Fix linter errors by stefannica in https://github.com/zenml-io/zenml/pull/1557
* Add Vertex as orchestrator and step operator to deploy CLI by wjayesh in https://github.com/zenml-io/zenml/pull/1559
* Fix dashboard secret references by stefannica in https://github.com/zenml-io/zenml/pull/1561
* New pipeline and step interface by schustmi in https://github.com/zenml-io/zenml/pull/1466
* Major Documentation Rehaul by AlexejPenner in https://github.com/zenml-io/zenml/pull/1562
* Easy CSV Visualization by fa9r in https://github.com/zenml-io/zenml/pull/1556

New Contributors
* bhatt-priyadutt made their first contribution in https://github.com/zenml-io/zenml/pull/1544

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.39.1...0.40

Minor hotfix release for running ZenML in Google Colab environments.

What's Changed
* Fix Source Resolving in Colab by fa9r in https://github.com/zenml-io/zenml/pull/1530

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.39.0...0.39.1

<img width="1499" alt="artifact_visualization" src="https://github.com/zenml-io/zenml/assets/23118541/a18c74ce-c18c-44d3-a105-f866c8e50e76">

ZenML release 0.39.0 introduces several big new features:
- The `zenml stack recipe` CLI commands now support fine-grained handling of individual stack components.
- Artifacts are now automatically visualized in the dashboard.
- Materializers received an overhaul: a new `cloudpickle` default materializer was added that works for arbitrary objects, and a `pycaret` materializer that can handle various modeling frameworks in a unified format.

The release also contains many bug fixes and quality-of-life improvements, such as new settings options for the SageMaker and Kubernetes orchestrators.

Individual Stack Component Deployment

In this release, we've enhanced the ZenML stack recipe CLI to support conditional deployment, destruction, and configuration of individual stack components. Users can now quickly deploy and destroy components with options for each flavor, and pass a config file for custom variables. The new `output` CLI command allows users to retrieve outputs from their recipes. Overall, this update streamlines deploying and managing stack components by providing a more efficient and user-friendly experience.

Artifact Visualization

Artifact visualizations are now automatically extracted by ZenML and embedded in the ZenML dashboard. Visualizations can now be defined by overriding the `save_visualizations` method of the materializer that handles an artifact. These visualizations are then automatically shown in the dashboard and can also be displayed in Jupyter notebooks using the new `visualize` post-execution method.

Default Cloudpickle Materializer

ZenML now uses `cloudpickle` under the hood to save/load artifacts that other materializers cannot handle. This makes it even easier to get started with ZenML since you no longer need to define custom materializers if you just want to experiment with some new data types.

What's Changed
* Docs/zenml hub documentation by bcdurak in https://github.com/zenml-io/zenml/pull/1490
* Sort integration list before display by strickvl in https://github.com/zenml-io/zenml/pull/1494
* Update docs to surface CLI filtering syntax by strickvl in https://github.com/zenml-io/zenml/pull/1496
* ZenML Hub Tests & CLI Improvements by fa9r in https://github.com/zenml-io/zenml/pull/1495
* Delete Legacy Docs by fa9r in https://github.com/zenml-io/zenml/pull/1497
* Improve the REST API error handling by stefannica in https://github.com/zenml-io/zenml/pull/1451
* Fix circular import of PipelineRunConfiguration by schustmi in https://github.com/zenml-io/zenml/pull/1501
* Delete Deprecated Artifacts and Materializer Code by fa9r in https://github.com/zenml-io/zenml/pull/1498
* Allow filtering runs by code repo id by schustmi in https://github.com/zenml-io/zenml/pull/1499
* Add example to docs for passing stack component specific settings by christianversloot in https://github.com/zenml-io/zenml/pull/1506
* Increase step run field lengths by schustmi in https://github.com/zenml-io/zenml/pull/1503
* Fix Sagemaker orchestrator pipeline name bug by strickvl in https://github.com/zenml-io/zenml/pull/1508
* Generate unique SageMaker training job name based on pipeline and ste… by christianversloot in https://github.com/zenml-io/zenml/pull/1505
* [CI Fix] Pin Llama Index Version by fa9r in https://github.com/zenml-io/zenml/pull/1516
* Basic PyCaret integration and materializer by christianversloot in https://github.com/zenml-io/zenml/pull/1512
* Specify line endings for different operating systems by strickvl in https://github.com/zenml-io/zenml/pull/1513
* Extend SageMakerOrchestratorSettings with processor_args enabling step level configuration by christianversloot in https://github.com/zenml-io/zenml/pull/1509
* Fix post execution `get_pipeline()` and `pipeline.get_runs()` by fa9r in https://github.com/zenml-io/zenml/pull/1510
* Default `cloudpickle` Materializer & Materializer Inheritance by fa9r in https://github.com/zenml-io/zenml/pull/1507
* Artifact Visualization by fa9r in https://github.com/zenml-io/zenml/pull/1472
* Add Kubernetes Orchestrator Settings by fa9r in https://github.com/zenml-io/zenml/pull/1518
* Bump `ruff` to 0.0.265 by strickvl in https://github.com/zenml-io/zenml/pull/1520
* feat: Set cloud function service account to the one defined in Vertex… by francoisserra in https://github.com/zenml-io/zenml/pull/1519
* Fix Kubernetes Orchestrator Config Loading by fa9r in https://github.com/zenml-io/zenml/pull/1523
* Resolve path during module resolving by schustmi in https://github.com/zenml-io/zenml/pull/1521
* Fix `SO_REUSEPORT` issue by fa9r in https://github.com/zenml-io/zenml/pull/1524
* Add individual stack component deployment through recipes by wjayesh in https://github.com/zenml-io/zenml/pull/1328
* Raise 501 for Unauthenticated Artifact Stores by fa9r in https://github.com/zenml-io/zenml/pull/1522
* Fix Duplicate Step Error by fa9r in https://github.com/zenml-io/zenml/pull/1527
* Fix pulling of stack recipes on `zenml init` by wjayesh in https://github.com/zenml-io/zenml/pull/1528
* Store dockerfile and requirements for builds by schustmi in https://github.com/zenml-io/zenml/pull/1525

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.38.0...0.39.0

Get ready to supercharge your ML pipelines with [the revolutionary ZenML Hub](https://docs.zenml.io/starter-guide/collaborate/zenml-hub), a central
platform that enables our users to search, share and discover community-contributed code,
now available with the ZenML 0.38.0 release! Our cutting-edge plugin system lets you
seamlessly contribute and consume a variety of code pieces, including stack component
flavors, pipelines, steps, materializers, and more. With the ZenML Hub, you can easily
integrate these components into your workflow and take your ML game to the next level.

Say goodbye to tedious integration processes and hello to exciting, effortless experimentation!

<img src="https://user-images.githubusercontent.com/3963946/231549808-b0702647-a6d8-4cbe-a9f3-687fa606b5bc.png" width="600"/>

The ZenML Hub makes its debut already equipped with a set of plugins that include
ZenML-verified steps (with a special focus on data loader steps), setting the standard
for seamless integration. And that's just the beginning! Our team is already hard at work
on expanding the Hub's capabilities, with plans to introduce exciting new entities like
materializers and flavors. Stay tuned for more groundbreaking updates as we continue to
revolutionize the ML landscape!

But why wait ? With the ZenML Hub, anyone can contribute their own plugins and take
part in accelerating innovation in the ML field. Be a part of the cutting-edge community
that is shaping the future of machine learning and driving impactful solutions forward.
Becoming a contributor starts with logging in to the ZenML Hub through the ZenML
`zenml hub login` CLI command or through the ZenML dashboard. More information on
this process is available [in our documentation](https://docs.zenml.io/starter-guide/collaborate/zenml-hub#how-do-i-create-my-own-plugin).

<img src="https://user-images.githubusercontent.com/3963946/231553210-a280cb9a-8add-49e8-a93f-ec98553746b5.png" width="600"/>

If you're interested in learning more about our motivation for implementing the ZenML
Hub and our plans for its future, we invite you to read [our new blog post](https://blog.zenml.io/zenml-hub-launch). In addition
to our technical documentation, the blog post provides a comprehensive overview of the
ZenML Hub's goals and objectives, as well as the features that we plan to introduce
in the future.

Aside from this major new feature, the release also includes a number of small
improvements and bug fixes, the most notable of which is the ability to manage secrets
from the ZenML dashboard, as well as to reference them in the configuration of stack
components.

<img src="https://user-images.githubusercontent.com/3963946/231554518-1da1c8bf-4be1-4073-94b2-1e9dca4e9d65.png" width="600"/>
<img src="https://user-images.githubusercontent.com/3963946/231554500-2bb94b27-75d5-461e-ac0d-e9af3ccccc24.png" width="600"/>

What's Changed
* Fix broken ENV variable by strickvl in https://github.com/zenml-io/zenml/pull/1458
* fix screenshot size in code repo by safoinme in https://github.com/zenml-io/zenml/pull/1467
* Fix CI (Deepchecks integration tests) by fa9r in https://github.com/zenml-io/zenml/pull/1470
* chore: update teams.yml by Cahllagerfeld in https://github.com/zenml-io/zenml/pull/1459
* Fix `BuiltInContainerMaterializer` for subtypes and non-built-in types by fa9r in https://github.com/zenml-io/zenml/pull/1464
* Kubernetes Orchestrator Improvements by fa9r in https://github.com/zenml-io/zenml/pull/1460
* Fix flaky CLI tests by schustmi in https://github.com/zenml-io/zenml/pull/1465
* Fix circular import during type checking by schustmi in https://github.com/zenml-io/zenml/pull/1463
* Allow secret values replacement in REST API PUT by stefannica in https://github.com/zenml-io/zenml/pull/1471
* Fix two steps race condition by safoinme in https://github.com/zenml-io/zenml/pull/1473
* Downgrading ZenML Version in global config by safoinme in https://github.com/zenml-io/zenml/pull/1474
* Revert "Downgrading ZenML Version in global config" by safoinme in https://github.com/zenml-io/zenml/pull/1476
* Add metadata to stack components by wjayesh in https://github.com/zenml-io/zenml/pull/1416
* remove modules from the list output for stack recipes by wjayesh in https://github.com/zenml-io/zenml/pull/1480
* Pin `openai` integration to `>0.27.0` by strickvl in https://github.com/zenml-io/zenml/pull/1461
* Apply formatting fixes to `/scripts` by strickvl in https://github.com/zenml-io/zenml/pull/1462
* Move import outside of type checking by schustmi in https://github.com/zenml-io/zenml/pull/1482
* Delete extra word from `bentoml` docs by strickvl in https://github.com/zenml-io/zenml/pull/1484
* Remove top-level config from recommended repo structure by schustmi in https://github.com/zenml-io/zenml/pull/1485
* Bump `mypy` and `ruff` by strickvl in https://github.com/zenml-io/zenml/pull/1481
* ZenML Version Downgrade - Silence Warnning by safoinme in https://github.com/zenml-io/zenml/pull/1477
* Update ZenServer recipes to include secret stores by wjayesh in https://github.com/zenml-io/zenml/pull/1483
* Fix alembic order by schustmi in https://github.com/zenml-io/zenml/pull/1487
* Fix source resolving for classes in notebooks by schustmi in https://github.com/zenml-io/zenml/pull/1486
* fix: use pool_pre_ping to discard invalid SQL connections when borrow… by francoisserra in https://github.com/zenml-io/zenml/pull/1489

New Contributors
* Cahllagerfeld made their first contribution in https://github.com/zenml-io/zenml/pull/1459
* francoisserra made their first contribution in https://github.com/zenml-io/zenml/pull/1489

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.37.0...0.38.0

In this ZenML release, we are pleased to introduce a compelling new feature:
[ZenML Code Repositories](https://docs.zenml.io/starter-guide/production-fundamentals/code-repositories).
This innovative addition formalizes the principles of code versioning and
tracking while consolidating their pivotal role in executing pipelines and
caching pipeline steps. With Code Repositories, ZenML is equipped to maintain an
accurate record of the code version employed in your pipeline runs. Furthermore,
executing a pipeline that is monitored by a registered code repository can
significantly accelerate the Docker image building process for containerized
stack components.

As is the case with everything ZenML, we designed the ZenML Code Repository
concept as a highly extensible abstraction. The update defines the basic Code
Repository interface an includes two implementations integrating ZenML with
two popular code repository flavors: GitHub and GitLab.

Other Enhancements

We've updated the `pytorch-lightning` integration to support the `2.0` version.
We also updated the `mlflow` integration to support the `2.2.2` version.

**IMPORTANT**: it is not recommended to continue using MLflow older than `2.2.1`
as a model registry with ZenML, as [it is vulnerable to a security issue](https://github.com/advisories/GHSA-xg73-94fp-g449).

Last but not least, two stellar additions from our community members:

* `zenml stack delete` now supports a `--recursive` flag to delete all
components in a stack. Many thanks to KenmogneThimotee for the contribution!
* the ZenML Sagemaker step operator has been expanded to support S3 input data
and additional input arguments. Many thanks to christianversloot for the
contribution!

Breaking Changes

The ZenML GitHub Orchestrator and GitHub Secrets Manager have been removed in
this release. Given that their concerns overlapped with the new ZenML GitHub
Code Repository and they didn't provide sufficient value on their own, we
decided to discontinue them. If you were using these components, you can
continue to use GitHub Actions to run your pipelines, in combination with the
ZenML GitHub Code Repository.

What's Changed
* Test integration for seldon example by safoinme in https://github.com/zenml-io/zenml/pull/1285
* Update `pytorch-lightning` to support `2.0` by safoinme in https://github.com/zenml-io/zenml/pull/1425
* Code repository by schustmi in https://github.com/zenml-io/zenml/pull/1344
* Bump `ruff` to 0.259 by strickvl in https://github.com/zenml-io/zenml/pull/1439
* Change `pipeline_run_id` to `run_name` by safoinme in https://github.com/zenml-io/zenml/pull/1390
* Update `mypy>=1.1.1` and fix new errors by safoinme in https://github.com/zenml-io/zenml/pull/1432
* Add `--upgrade` option to ZenML integration install by safoinme in https://github.com/zenml-io/zenml/pull/1435
* Bump `MLflow` to 2.2.2 by safoinme in https://github.com/zenml-io/zenml/pull/1441
* HuggingFace Spaces server deployment option by strickvl in https://github.com/zenml-io/zenml/pull/1427
* Bugfix for server import by bcdurak in https://github.com/zenml-io/zenml/pull/1442
* Fix HF Spaces URL by strickvl in https://github.com/zenml-io/zenml/pull/1444
* Remove all `zenml.cli` imports outside of `zenml.cli` by fa9r in https://github.com/zenml-io/zenml/pull/1447
* Add recursive deletion of components for `zenml stack delete` by KenmogneThimotee in https://github.com/zenml-io/zenml/pull/1437
* Temporarily disable primary key requirement for newer mysql versions by schustmi in https://github.com/zenml-io/zenml/pull/1450
* Add step name suffix for sagemaker job name by schustmi in https://github.com/zenml-io/zenml/pull/1452
* Code repo docs by schustmi in https://github.com/zenml-io/zenml/pull/1448
* Allow resource settings for airflow kubernetes pod operators by schustmi in https://github.com/zenml-io/zenml/pull/1378
* SageMaker step operator: expand input arguments and add support for S3 input data by christianversloot in https://github.com/zenml-io/zenml/pull/1381
* Add Screenshots to Code Repo Token by safoinme in https://github.com/zenml-io/zenml/pull/1454

New Contributors
* KenmogneThimotee made their first contribution in https://github.com/zenml-io/zenml/pull/1437
* christianversloot made their first contribution in https://github.com/zenml-io/zenml/pull/1381

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.36.1...0.37.0

This minor release contains some small fixes and improvements.

- We fixed a bug with the way hooks were being parsed, which was causing
pipelines to fail.
- We brought various parts of the documentation up to date with features that
had previously been added, notably the new image building functionality.
- We added a failure hook that connects to OpenAI's ChatGPT API to allow you to
receive a message when a pipeline fails that includes suggestions on how to
fix the failing step.
- We added a new integration with `langchain` and `llama_hub` to allow you to
build on top of those libraries as part of a more robust MLOps workflow.
- We made the first some bigger changes to our analytics system to make it more
robust and secure. This release begins that migration. Users should expect no
changes in behavior and all telemetry-related preferences will be preserved.

What's Changed
* Fix hook parser by strickvl in https://github.com/zenml-io/zenml/pull/1428
* Fix some pipeline bugs by schustmi in https://github.com/zenml-io/zenml/pull/1426
* Add image builders to Examples by safoinme in https://github.com/zenml-io/zenml/pull/1434
* ZenML Failure Hook for OpenAI ChatGPT fixes by strickvl in https://github.com/zenml-io/zenml/pull/1430
* Integrations with `langchain` and `llama_hub` by fa9r in https://github.com/zenml-io/zenml/pull/1404
* Add basic tests for the server and recipes CLI by wjayesh in https://github.com/zenml-io/zenml/pull/1306
* Add to our alembic migration guide by strickvl in https://github.com/zenml-io/zenml/pull/1423
* Analytics 2.0 by bcdurak in https://github.com/zenml-io/zenml/pull/1411
* Improve Slack Alerter by adding message blocks by soubenz in https://github.com/zenml-io/zenml/pull/1402
* Add HF deployment type by strickvl in https://github.com/zenml-io/zenml/pull/1438

New Contributors
* soubenz made their first contribution in https://github.com/zenml-io/zenml/pull/1402

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.36.0...0.36.1