Zenml

UPDATE: a breaking update to two of our dependencies means that you should run the following update after installing this version of `zenml`:

shell
pip install 'bcrypt==4.0.1' 'sqlmodel==0.0.8'


This release, introduces SkyPilot, a new VM orchestrator for ZenML that lets users run pipelines on their choice of cloud provider VMs, offering a GPU option without Kubernetes or serverless orchestrators. This release also brings bug fixes and improvements, including a streamlined 'connect' command, interactive configuration for 'zenml stack deploy,' and enhanced documentation covering SageMaker, GCP, and service connectors with MFA.

New Orchestrator: SkyPilot (1765)

This release introduces a new orchestrator called SkyPilot. SkyPilot is a VM orchestrator
that can be used to run ZenML pipelines on a VM of choice in one of the three supported
cloud providers. It is an excellent choice for users who want to run ZenML pipelines on a GPU
instance, but don't want to use Kubernetes or serverless orchestrators like SageMaker.

Fixes and Improvements

This release fixes several bugs and improves the user experience of the CLI and the
documentation. The most notable changes are:
* The new `connect` command that allows connecting all stack components within a stack to a
service connector with a single command.
* Adding an interactive flow to the `zenml stack deploy` command that allows users to
configure their stack in a guided manner.
* Add documentation on how to debug the SageMaker orchestrator, how to get started with
a quick cloud stack on GCP, and documentation on using service connectors with
enabled MFA.

What's Changed
* Add support for empty API token in Kubernetes service connector. by stefannica in https://github.com/zenml-io/zenml/pull/1808
* Use the container registry credentials to build images with the local image builder by stefannica in https://github.com/zenml-io/zenml/pull/1804
* Fix CI by fa9r in https://github.com/zenml-io/zenml/pull/1809
* Add documentation on how to debug the SageMaker orchestrator by fa9r in https://github.com/zenml-io/zenml/pull/1810
* Bump `rich` and `uvicorn` by jlopezpena in https://github.com/zenml-io/zenml/pull/1750
* SageMaker: Enable configuring authentication credentials explicitly by fa9r in https://github.com/zenml-io/zenml/pull/1805
* Fix: ZenML DB migrations don't run if zenml is installed in path with spaces by stefannica in https://github.com/zenml-io/zenml/pull/1815
* Fix mlflow 'run_name' variable overwriting by iraadit in https://github.com/zenml-io/zenml/pull/1821
* Add `SECURITY.md` file for vulnerability disclosures. by strickvl in https://github.com/zenml-io/zenml/pull/1824
* Add MFA limitation to service-connectors docs by safoinme in https://github.com/zenml-io/zenml/pull/1827
* Improve `zenml stack describe` to show `mlstacks` outputs by strickvl in https://github.com/zenml-io/zenml/pull/1826
* Documentation to get started with a quick cloud stack on GCP by AlexejPenner in https://github.com/zenml-io/zenml/pull/1807
* Fix missing text in git repo docs by strickvl in https://github.com/zenml-io/zenml/pull/1831
* Handle irregular plural of `code_repository` for error message by strickvl in https://github.com/zenml-io/zenml/pull/1832
* Connect stack to a service account by safoinme in https://github.com/zenml-io/zenml/pull/1828
* SkyPilot Integration with VM Orchestrators by htahir1 in https://github.com/zenml-io/zenml/pull/1765
* Add interactive CLI flow for `zenml stack deploy` by strickvl in https://github.com/zenml-io/zenml/pull/1829
* Add `README` file for helm chart by strickvl in https://github.com/zenml-io/zenml/pull/1830
* Fix slack environment variable in in `generative_chat` example README by bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1836

New Contributors
* iraadit made their first contribution in https://github.com/zenml-io/zenml/pull/1821

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.44.2...tes

This release contains updates for some of the most popular integrations, as well as several bug fixes and documentation improvements.

Minor Default Behavior Changes
* The default page size for `zenml list` commands was reduced to 20 (from 50) to speed up the runtime of such commands.
* Simultaneous connection to local and remote ZenML servers is no longer possible since this caused several unexpected behaviors in the past.

Integration Updates
- The `mlflow` integration now supports the newest MLflow version `2.6.0`.
- The `evidently` integration now supports the latest Evidently version `0.4.4`.
- The SageMaker orchestrator of the `aws` integration now supports authentication via service connectors.

What's Changed

* Add `bandit` to CI for security linting by strickvl in https://github.com/zenml-io/zenml/pull/1775
* Add `mlstacks` compatibility check to CI by strickvl in https://github.com/zenml-io/zenml/pull/1767
* extend `StepContext` visibility to materializers by avishniakov in https://github.com/zenml-io/zenml/pull/1769
* Revert GH changes to fix colima bug in macos gh by safoinme in https://github.com/zenml-io/zenml/pull/1779
* Reduce CI runner count by strickvl in https://github.com/zenml-io/zenml/pull/1777
* Add E2E template as example by avishniakov in https://github.com/zenml-io/zenml/pull/1766
* Fix CI step names by avishniakov in https://github.com/zenml-io/zenml/pull/1784
* Add vulnerability scanner by strickvl in https://github.com/zenml-io/zenml/pull/1776
* Stop CI from running on push to `develop` by strickvl in https://github.com/zenml-io/zenml/pull/1788
* Skip update templates outside PR by avishniakov in https://github.com/zenml-io/zenml/pull/1786
* Fix azure service connector docs by stefannica in https://github.com/zenml-io/zenml/pull/1778
* fix: use k8s V1CronJob instead of V1beta1CronJob (1781) by francoisserra in https://github.com/zenml-io/zenml/pull/1787
* Page limit adjustment by bcdurak in https://github.com/zenml-io/zenml/pull/1791
* Prevent simultaneous connection to local and remote servers by fa9r in https://github.com/zenml-io/zenml/pull/1792
* Update `MLflow` version to allow support for 2.6.0 by safoinme in https://github.com/zenml-io/zenml/pull/1782
* Improve `ConnectionError` error message by fa9r in https://github.com/zenml-io/zenml/pull/1783
* Stop old MLflow services when deploying new ones by fa9r in https://github.com/zenml-io/zenml/pull/1793
* Prevent adding private components into shared stacks by fa9r in https://github.com/zenml-io/zenml/pull/1794
* Publish server helm chart as part of CI by wjayesh in https://github.com/zenml-io/zenml/pull/1740
* Docs on the use of ZenML-specific environment variables by strickvl in https://github.com/zenml-io/zenml/pull/1796
* Add support for newer Evidently versions by fa9r in https://github.com/zenml-io/zenml/pull/1780
* Link E2E example to docs by avishniakov in https://github.com/zenml-io/zenml/pull/1790
* Copy step instance before applying configuration by schustmi in https://github.com/zenml-io/zenml/pull/1798
* Fix AWS container registry image pushing with service connectors by fa9r in https://github.com/zenml-io/zenml/pull/1797
* Make Sagemaker orchestrator work with connectors by fa9r in https://github.com/zenml-io/zenml/pull/1799
* Add rebase Pre-requisite to PRs template by safoinme in https://github.com/zenml-io/zenml/pull/1801

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.44.1...0.44.2

This release brings various improvements over the previous version, mainly focusing on using the newly refactored `mlstacks` package, ZenML's `logging` module and the changes in our analytics.

IMPORTANT NOTICE

**Note:** *0.44.0 was removed from Pypi due to an issue with the alembic versions which could affect the database state. A branch occurred in the versions: 0.42.1 -> [0.43.0, e1d66d91a099] -> 0.44.0. This release fixes the issue.<br>
The primary issue arises when deploying version 0.44.0 using a MySQL backend. Although the alembic migration executes all tasks up to 0.44.0, the alembic version represented in the database remains at 0.43.0. This issue persists irrespective of the measures taken, including trying various versions after 0.43.0.<br>
This imbalance leads to failure when running a second replica migration because the database's state is at 0.44.0 while the alembic version remains at 0.43.0. Similarly, attempts to run a second replica or restart the pod fail as the alembic tries to migrate from 0.43.0 to 0.44.0, which is not possible because these changes already exist in the database.<br>
Please note: If you encounter this problem, we recommend you roll back to previous versions and upgrade to 0.43.0. If you still experience difficulties, please join our Slack community at https://zenml.io/slack. We're ready to help you work through this issue.*

What's Changed

* Remove e2e example and point to templates by avishniakov in https://github.com/zenml-io/zenml/pull/1752
* Add cloud architecture docs by htahir1 in https://github.com/zenml-io/zenml/pull/1751
* Update docs/docstrings following `mlstacks` repo name change by strickvl in https://github.com/zenml-io/zenml/pull/1754
* Update Cloud deployment scenarios by stefannica in https://github.com/zenml-io/zenml/pull/1757
* Fixing the logging message regarding caching by bcdurak in https://github.com/zenml-io/zenml/pull/1748
* Improvements to the step logs storage functionality by bcdurak in https://github.com/zenml-io/zenml/pull/1733
* Fix `qemu`/`colima` Github Actions bug by safoinme in https://github.com/zenml-io/zenml/pull/1760
* Bump `ruff` and `mypy` by strickvl in https://github.com/zenml-io/zenml/pull/1762
* Add Template Testing in Core by avishniakov in https://github.com/zenml-io/zenml/pull/1745
* Removing analytics v1 and optimizing v2 by bcdurak in https://github.com/zenml-io/zenml/pull/1753
* Update publish script to take a token by strickvl in https://github.com/zenml-io/zenml/pull/1758
* Update variable name for release publication token by strickvl in https://github.com/zenml-io/zenml/pull/1764
* Lock the `MYSQL` Database during DB migrations by safoinme in https://github.com/zenml-io/zenml/pull/1763
* `mlstacks` integration (and deprecation of old deployment logic) by strickvl in https://github.com/zenml-io/zenml/pull/1721
* Upgrade typing extensions within api docs build workflow by AlexejPenner in https://github.com/zenml-io/zenml/pull/1741
* Fix branching alembic history by AlexejPenner in https://github.com/zenml-io/zenml/pull/1772
* Remove pinned `zenml` version specified in TOC for SDK docs by strickvl in https://github.com/zenml-io/zenml/pull/1770
* Modified the track metadata for the opt-in event by bcdurak in https://github.com/zenml-io/zenml/pull/1774
* Check alembic branch divergence in CI by strickvl in https://github.com/zenml-io/zenml/pull/1773
* Remove the DB lock by safoinme in https://github.com/zenml-io/zenml/pull/1771

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.43.0...0.44.1

IMPORTANT NOTICE

This release has been removed from pypi due to an issue with the alembic versions which could affect the database state. A branch occurred in the versions: 0.42.1 -> [0.43.0, e1d66d91a099] -> 0.44.0.

The primary issue arises when deploying version 0.44.0 using a MySQL backend. Although the alembic migration executes all tasks up to 0.44.0, the alembic version represented in the database remains at 0.43.0. This issue persists irrespective of the measures taken, including trying various versions after 0.43.0.

This imbalance leads to failure when running a second replica migration because the database's state is at 0.44.0 while the alembic version remains at 0.43.0. Similarly, attempts to run a second replica or restart the pod fail as the alembic tries to migrate from 0.43.0 to 0.44.0, which is not possible because these changes already exist in the database.

Please note: If you encounter this problem, we recommend that you rollback to previous versions and then upgrade to 0.43.0. If you still experience difficulties, please join our Slack community at https://zenml.io/slack. We're ready to help you work through this issue.
____

Older Notes

This release brings various improvements over the previous version, mainly
focusing on the usage of newly refactored `mlstacks` package, ZenML's `logging`
module and the changes in our analytics.

What's Changed

* Remove e2e example and point to templates by avishniakov in https://github.com/zenml-io/zenml/pull/1752
* Add cloud architecture docs by htahir1 in https://github.com/zenml-io/zenml/pull/1751
* Update docs/docstrings following `mlstacks` repo name change by strickvl in https://github.com/zenml-io/zenml/pull/1754
* Update Cloud deployment scenarios by stefannica in https://github.com/zenml-io/zenml/pull/1757
* Fixing the logging message regarding caching by bcdurak in https://github.com/zenml-io/zenml/pull/1748
* Improvements to the step logs storage functionality by bcdurak in https://github.com/zenml-io/zenml/pull/1733
* Fix `qemu`/`colima` Github Actions bug by safoinme in https://github.com/zenml-io/zenml/pull/1760
* Bump `ruff` and `mypy` by strickvl in https://github.com/zenml-io/zenml/pull/1762
* Add Template Testing in Core by avishniakov in https://github.com/zenml-io/zenml/pull/1745
* Removing analytics v1 and optimizing v2 by bcdurak in https://github.com/zenml-io/zenml/pull/1753
* Update publish script to take a token by strickvl in https://github.com/zenml-io/zenml/pull/1758
* Update variable name for release publication token by strickvl in https://github.com/zenml-io/zenml/pull/1764
* Lock `MYSQL` Database during DB migrations by safoinme in https://github.com/zenml-io/zenml/pull/1763
* `mlstacks` integration (and deprecation of old deployment logic) by strickvl in https://github.com/zenml-io/zenml/pull/1721

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.43.0...0.44.0

Backports some important fixes that have been introduced in more recent versions
of ZenML to the 0.43.x release line.

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.43.0...0.43.1

This release brings limited support for Python 3.11, improves quickstart experience with the fully reworked flow, enhances the user experience while dealing with ZenML docs, offers new extended templates for projects and fixes GCP connector creation issue.

Limited support for Python 3.11
This release adds limited support for Python 3.11.
The following integrations are currently not tested and may behave unexpectedly with Python 3.11:
- gcp
- kubeflow
- tekton

This is because:
- GCP packages that support Python 3.11 are not compatible with KFP 1
- Upgrade to KFP 2 is blocked by the fact that Tekton doesn't have any release compatible with KFP 2 yet (https://github.com/zenml-io/zenml/pull/1697)

Breaking Changes

A minor breaking change in CLI for `zenml init`:
- previously supported flag `--starter`
- new flag `--template-with-defaults`
- behavior remains the same - flag is responsible for usage of default settings in the template

What's Changed
* Disable implicit auth methods for service connectors by default by stefannica in https://github.com/zenml-io/zenml/pull/1704
* New quickstart by strickvl in https://github.com/zenml-io/zenml/pull/1692
* Set `MLflow` configuration as environment variables before deployment subprocess by safoinme in https://github.com/zenml-io/zenml/pull/1705
* Fix Migration Guide Links by fa9r in https://github.com/zenml-io/zenml/pull/1706
* Improve Input Validation Error Message by fa9r in https://github.com/zenml-io/zenml/pull/1712
* Update link in cloudpickle_materializer.py by duarteocarmo in https://github.com/zenml-io/zenml/pull/1713
* catch exceptions in `list_model_versions` by avishniakov in https://github.com/zenml-io/zenml/pull/1703
* Rename `transition_model_stage` to `transition_model_version_stage` by avishniakov in https://github.com/zenml-io/zenml/pull/1707
* pandas input to `predict` by avishniakov in https://github.com/zenml-io/zenml/pull/1715
* Small fixes to global config docs page by schustmi in https://github.com/zenml-io/zenml/pull/1714
* Allow specifying extra hosts for LocalDockerOrchestrator by schustmi in https://github.com/zenml-io/zenml/pull/1709
* Flexible use of `ignore_cols` in `evidently_report_step` by avishniakov in https://github.com/zenml-io/zenml/pull/1711
* Add external artifacts and direct links to run DAG by fa9r in https://github.com/zenml-io/zenml/pull/1718
* E2E flow example for templates by avishniakov in https://github.com/zenml-io/zenml/pull/1710
* Fix bug in service connector, Closes 1720 by soubenz in https://github.com/zenml-io/zenml/pull/1726
* Document the namespace and service account k8s orchestrator settings by stefannica in https://github.com/zenml-io/zenml/pull/1722
* Refactoring done and reduced some functions complexity and work-time by thanseefpp in https://github.com/zenml-io/zenml/pull/1719
* Update custom orchestrator guide by schustmi in https://github.com/zenml-io/zenml/pull/1728
* Improve error message when passing non-json serializable parameter by schustmi in https://github.com/zenml-io/zenml/pull/1729
* Bump `ruff` to 0.0.282 by strickvl in https://github.com/zenml-io/zenml/pull/1730
* Docs and README update for ZenML Cloud by bcdurak in https://github.com/zenml-io/zenml/pull/1723
* bump `MLflow` to 2.5.0 by safoinme in https://github.com/zenml-io/zenml/pull/1708
* Move Examples to Tests by fa9r in https://github.com/zenml-io/zenml/pull/1673
* Add Error Handling for Empty Pipelines by fa9r in https://github.com/zenml-io/zenml/pull/1734
* Revert "Add Error Handling for Empty Pipelines" by fa9r in https://github.com/zenml-io/zenml/pull/1735
* Changing the links to the public roadmap by bcdurak in https://github.com/zenml-io/zenml/pull/1737
* Add Error Handling for Empty Pipelines by fa9r in https://github.com/zenml-io/zenml/pull/1736
* Revisit `init --template` CLI for new templates by avishniakov in https://github.com/zenml-io/zenml/pull/1731
* Add Python 3.11 Support by fa9r in https://github.com/zenml-io/zenml/pull/1702
* fix error on scheduled pipelines with KubernetesOrchestrator by francoisserra in https://github.com/zenml-io/zenml/pull/1738
* Bugfix for identify calls with empty email strings by bcdurak in https://github.com/zenml-io/zenml/pull/1739

New Contributors
* duarteocarmo made their first contribution in https://github.com/zenml-io/zenml/pull/1713
* thanseefpp made their first contribution in https://github.com/zenml-io/zenml/pull/1719

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.42.0...0.43.0