Zenml

This release resolves several minor bugs and inconveniences introduced during the filtering and pagination overhaul in the last release. Additionally, the release includes new integration tests to improve future stability.

What's Changed
* Update and improve docker and helm deployment docs by stefannica in https://github.com/zenml-io/zenml/pull/1246
* Fixed broken link returned form pipeline runs by AlexejPenner in https://github.com/zenml-io/zenml/pull/1257
* Fix project scoping on `list_project_...` endpoints by fa9r in https://github.com/zenml-io/zenml/pull/1256
* Orchestrator tests by schustmi in https://github.com/zenml-io/zenml/pull/1258
* Add integration tests for lineage graph creation by fa9r in https://github.com/zenml-io/zenml/pull/1253
* Always instantiate a zen_store before startup. by AlexejPenner in https://github.com/zenml-io/zenml/pull/1261
* Fix post execution run fetching by schustmi in https://github.com/zenml-io/zenml/pull/1263
* Implemented the option to choose between ascending and descending on list calls by AlexejPenner in https://github.com/zenml-io/zenml/pull/1260
* Fix logger warning message by strickvl in https://github.com/zenml-io/zenml/pull/1267

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.32.0...0.32.1

* A new stack component, the "image builder", with a corresponding new Kaniko
integration.
* Logic for filtering and pagination of list requests.

Image Builder Abstraction and Kaniko Integration
ZenML stacks can now contain an image builder as additional optional stack
component. The image builder defines how the Docker images are built that are
required by many of the other stack components such as Airflow or Kubeflow.
Previously, all image building was handled implicitly by ZenML using local
Docker, which has now been refactored into the "local" image builder flavor.
As an alternative, you can now install the new "kaniko" integration to build
your images in Kubernetes using Kaniko.

Filtering and Pagination
All list commands in ZenML are now capable of advanced filtering such as
`zenml stack list --created="gt:22-12-04 17:00:00" --name contains:def`.

Additionally, list commands now return pages of results, which significantly
improves performance for power ZenML users that have already created many runs
or other entities.

What's Changed
* UserResponseModel contains roles, block recursion properly on more Models, reduce amount of Runs on a PipelineResponseModel by AlexejPenner in https://github.com/zenml-io/zenml/pull/1180
* Bump ruff version by strickvl in https://github.com/zenml-io/zenml/pull/1232
* Zenfile becomes project by strickvl in https://github.com/zenml-io/zenml/pull/1235
* Fix class resolution in notebooks under Python>=3.10 by fa9r in https://github.com/zenml-io/zenml/pull/1234
* Fix Sagemaker README images & pipeline addition by strickvl in https://github.com/zenml-io/zenml/pull/1239
* Step/Pipeline configuration tests by schustmi in https://github.com/zenml-io/zenml/pull/1233
* Removed gRPC from diagrams by AlexejPenner in https://github.com/zenml-io/zenml/pull/1242
* Fix MLflow tracking example bug for Macs by strickvl in https://github.com/zenml-io/zenml/pull/1237
* Fix copy function to copyfile in registered filesystem by safoinme in https://github.com/zenml-io/zenml/pull/1243
* Image builder abstraction by schustmi in https://github.com/zenml-io/zenml/pull/1198
* Add support for modular recipes to the recipe CLI by wjayesh in https://github.com/zenml-io/zenml/pull/1247
* Add docs on upgrading and troubleshooting zenml server by wjayesh in https://github.com/zenml-io/zenml/pull/1244
* Improve Seldon and Kserve Docs by wjayesh in https://github.com/zenml-io/zenml/pull/1236
* Add Pagination to all List commands by AlexejPenner in https://github.com/zenml-io/zenml/pull/1113

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.31.1...0.32.0

This release includes several bug fixes and new additions under the hood such as testing for various internal utility functions. This should help keep ZenML more stable over time. Additionally, we added the ability to customize default materializers for custom artifact stores, and the ability to track system info and the Python version of pipeline runs (both where pipelines are initially executed as well as wherever they eventually run). We added better support for pipeline scheduling (particularly from within the CLI) and tracking of the source code of steps. The release also includes the addition of information about whether the pipeline is running on a stack created by the active user, and the ability to specify Kubernetes container resource requests and limits. Finally, we addressed issues with caching such that caching is enabled for steps that have explicit enable_cache=True specified (even when pipelines have it turned off).

What's Changed
* Test for `enum_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1209
* Add missing space in Azure docs by strickvl in https://github.com/zenml-io/zenml/pull/1218
* Test for `dashboard_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1202
* Cloud version gets love by htahir1 in https://github.com/zenml-io/zenml/pull/1219
* ZenFiles to ZenML Projects by htahir1 in https://github.com/zenml-io/zenml/pull/1220
* Track System Info and Python Version of Pipeline Runs by fa9r in https://github.com/zenml-io/zenml/pull/1215
* Tests for `pydantic_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1207
* Customizing Default Materializers for Custom Artifact Stores by safoinme in https://github.com/zenml-io/zenml/pull/1224
* Test `typed_model` utilities by strickvl in https://github.com/zenml-io/zenml/pull/1208
* Enable Airflow<2.4 by schustmi in https://github.com/zenml-io/zenml/pull/1222
* Fix `alembic_start` migration if tables exist by fa9r in https://github.com/zenml-io/zenml/pull/1214
* Tests for `network_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1201
* Tests for `io_utils` and removal of duplicate code by strickvl in https://github.com/zenml-io/zenml/pull/1199
* Use `ruff` to replace our linting suite by strickvl in https://github.com/zenml-io/zenml/pull/1211
* Test `materializer` utilities by safoinme in https://github.com/zenml-io/zenml/pull/1221
* Add information whether pipeline is running on a stack created by the active user by schustmi in https://github.com/zenml-io/zenml/pull/1229
* Test `daemon` util functions by strickvl in https://github.com/zenml-io/zenml/pull/1210
* Test `filesync_model` utils by strickvl in https://github.com/zenml-io/zenml/pull/1230
* Track Source Code of Steps by fa9r in https://github.com/zenml-io/zenml/pull/1216
* Track Pipeline Run Schedules by fa9r in https://github.com/zenml-io/zenml/pull/1227
* Tests for analytics by bcdurak in https://github.com/zenml-io/zenml/pull/1228
* Allow specifying Kubernetes container resource requests and limits by schustmi in https://github.com/zenml-io/zenml/pull/1223
* Enable cache for all steps that have explicit `enable_cache=True` by fa9r in https://github.com/zenml-io/zenml/pull/1217
* Make shared stacks visible again by AlexejPenner in https://github.com/zenml-io/zenml/pull/1225

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.31.0...0.31.1

The highlights of this release are:

* our Materializers have been redesigned to be more flexible and easier to use
* we have added a new integration test framework
* the SageMaker orchestrator has been added to our list of supported orchestrators
* pipeline runs and artifacts can now be deleted from the ZenML database via the CLI
or the Client API
* some integrations have been updated to a more recent version: Kubeflow, Seldon
Core and Tekton

This release also includes a few bug fixes and other minor improvements to
existing features.

What's Changed
* Fix installation instructions in readme and docs by schustmi in https://github.com/zenml-io/zenml/pull/1167
* Fix broken TOC for scheduling docs by strickvl in https://github.com/zenml-io/zenml/pull/1169
* Ensure model string fields have a max length by strickvl in https://github.com/zenml-io/zenml/pull/1136
* Integration test framework by stefannica in https://github.com/zenml-io/zenml/pull/1099
* Check if all ZenML server dependencies are installed for local zenml deployment using `zenml up` by dnth in https://github.com/zenml-io/zenml/pull/1144
* Persist the server ID in the database by stefannica in https://github.com/zenml-io/zenml/pull/1173
* Tiny docs improvements by strickvl in https://github.com/zenml-io/zenml/pull/1179
* Changing some interactions with analytics fields by bcdurak in https://github.com/zenml-io/zenml/pull/1174
* Fix `PyTorchDataLoaderMaterializer` for older torch versions by fa9r in https://github.com/zenml-io/zenml/pull/1178
* Redesign Materializers by fa9r in https://github.com/zenml-io/zenml/pull/1154
* Fixing the error messages when fetching entities by bcdurak in https://github.com/zenml-io/zenml/pull/1171
* Moved the active_user property onto the client, implemented get_myself as zenstore method by AlexejPenner in https://github.com/zenml-io/zenml/pull/1161
* Bugfix/bump evidently version by AlexejPenner in https://github.com/zenml-io/zenml/pull/1183
* Alembic migration to update size of flavor config schema by fa9r in https://github.com/zenml-io/zenml/pull/1181
* Deleting pipeline runs and artifacts by fa9r in https://github.com/zenml-io/zenml/pull/1164
* Signer email checked before setting in google cloud scheduler by htahir1 in https://github.com/zenml-io/zenml/pull/1184
* Fix zenml helm chart to not leak analytics events by stefannica in https://github.com/zenml-io/zenml/pull/1190
* Tests for `dict_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1196
* Adding exception tracking to `zeml init` by bcdurak in https://github.com/zenml-io/zenml/pull/1192
* Prevent crashes during Airflow server forking on MacOS by schustmi in https://github.com/zenml-io/zenml/pull/1186
* add alpha as server deployment type by wjayesh in https://github.com/zenml-io/zenml/pull/1197
* Bugfix for custom flavor registration by bcdurak in https://github.com/zenml-io/zenml/pull/1195
* Tests for `uuid_utils` by strickvl in https://github.com/zenml-io/zenml/pull/1200
* Sagemaker orchestrator integration by strickvl in https://github.com/zenml-io/zenml/pull/1177
* Fix Pandas Materializer Index by safoinme in https://github.com/zenml-io/zenml/pull/1193
* Add support for deploying custom stack recipes using the ZenML CLI by wjayesh in https://github.com/zenml-io/zenml/pull/1188
* Add cloud CI environments by stefannica in https://github.com/zenml-io/zenml/pull/1176
* Fix project scoping for artifact list through ZenServer by fa9r in https://github.com/zenml-io/zenml/pull/1203

In this release, ZenML finally adds Mac M1 support, Python 3.10 support and much
greater flexibility and configurability under the hood by deprecating some
large dependencies like `ml-pipelines-sdk`.

Scheduling

Based on some community feedback around scheduling, this release comes with
improved docs concerning scheduling in general. Additionally, the Vertex AI
orchestrator now also supports scheduling.

Slimmer Dependencies

By removing dependencies on some of the packages that ZenML was built on, this
version of ZenML is slimmer, faster and more configurable than ever. This also
finally makes ZenML run natively on Macs with M1 processors without the need for
Rosetta. This also finally enables ZenML to run on Python 3.10.

Breaking Changes

* The removal of `ml-pipelines-sdk` and `tfx` leads to some larger changes in
the database that is tracking your pipeline runs and artifacts. **Note**: There
is an automatic migration to upgrade this automatically, However, please note
that downgrading back down to 0.23.0 is not supported.
* The CLI commands to export and import pipeline runs have been deprecated.
Namely: `zenml pipeline runs export` and `zenml pipeline runs import`
These commands were meant for migrating from `zenml<0.20.0` to
`0.20.0<=zenml<0.30.0`.
* The `azure-ml` integration dependency on `azureml-core` has been upgraded

This pre-release builds on top of 0.30.0rc2.

Major changes:
- Improved database migration from previous versions
- Fixes for step operators

Note: Installing and using this will migrate your ZenML metadata database
tables without any way of downgrading to previous releases. Use at your own risk.