Zenml

What's Changed
* Remove stack extra from installation, enable re-running the quickstart by schustmi in https://github.com/zenml-io/zenml/pull/1133
* Secrets manager support to experiment trackers docs by safoinme in https://github.com/zenml-io/zenml/pull/1137
* Updating the README files of our examples by bcdurak in https://github.com/zenml-io/zenml/pull/1128
* Prevent running with local ZenStore and remote code execution by schustmi in https://github.com/zenml-io/zenml/pull/1134
* Remove `ml-pipelines-sdk` dependency by schustmi in https://github.com/zenml-io/zenml/pull/1103
* Fix Huggingface dataset materializer by safoinme in https://github.com/zenml-io/zenml/pull/1142
* Disallow alembic downgrades for 0.30.0 release by fa9r in https://github.com/zenml-io/zenml/pull/1140
* Fix Client flavor-related methods by schustmi in https://github.com/zenml-io/zenml/pull/1153
* Replace User Password with Token in docker images by safoinme in https://github.com/zenml-io/zenml/pull/1147
* Remove zenml pipeline runs export / import CLI commands by fa9r in https://github.com/zenml-io/zenml/pull/1150
* Context manager to track events by bcdurak in https://github.com/zenml-io/zenml/pull/1149
* Made explicit `is not None` calls to allow for empty pwd again by AlexejPenner in https://github.com/zenml-io/zenml/pull/1159
* Add Neptune exp tracker into flavors table by dnth in https://github.com/zenml-io/zenml/pull/1156
* Fix step operators by schustmi in https://github.com/zenml-io/zenml/pull/1155
* Display correct name when updating a stack component by schustmi in https://github.com/zenml-io/zenml/pull/1160
* Update mysql database creation by schustmi in https://github.com/zenml-io/zenml/pull/1152
* Adding component conditions to experiment tracker examples and adding to the environmental variable docs by bcdurak in https://github.com/zenml-io/zenml/pull/1162
* Increase dependency range for protobuf by schustmi in https://github.com/zenml-io/zenml/pull/1163
* Scheduling documentation by strickvl in https://github.com/zenml-io/zenml/pull/1158
* Adding scheduling for Vertex Pipelines by htahir1 in https://github.com/zenml-io/zenml/pull/1148
* Fix alembic migration for sqlite<3.25 by fa9r in https://github.com/zenml-io/zenml/pull/1165
* Fix pandas Series materializer by jordandelbar in https://github.com/zenml-io/zenml/pull/1146

New Contributors
* jordandelbar made their first contribution in https://github.com/zenml-io/zenml/pull/1146

The **ZenML 0.70.0** release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading.

Key Changes

* **Artifact Versioning Improvements**: The handling of artifact versions has been improved, including the API improvements like the ability to batch artifact version requests to improve the execution times and more types for the step input/output artifacts, including multiple versions of the same artifact (e.g. model checkpoints), to improve the UX using ZenML UI or while working directly with the API.
* **Scalability Enhancements**: Various scalability improvements have been made, such as reducing unnecessary server requests and incrementing artifact versions server-side. These enhancements are expected to provide significant speed and scale improvements for ZenML users.
* **Metadata management**: Now, all the metadata-creating functions are gathered under one method called `log_metadata`. It is possible to call this method with different inputs to log run metadata for artifact versions, model versions, steps, and runs.
* **The oneof filtering**: This allows to filter entities using a new operator called `oneof`. You can use this with IDs (UUID type) or tags (or other string-typed attributes) like this `PipelineRunFilter(tag='oneof:["cats", "dogs"]')`.
* **Documentation Improvements**: The ZenML documentation has been restructured and expanded, including the addition of new sections on [finetuning](https://docs.zenml.io/user-guide/llmops-guide/finetuning-llms) and [LLM/ML engineering](https://docs.zenml.io/user-guide/llmops-guide/evaluation) resources.
* **Bug Fixes**: This release includes several bug fixes, including issues with in-process main module source loading, and more.

Caution: Make sure to back up your data before upgrading!
While this release brings many valuable improvements, the database schema changes and migrations pose a potential risk to users. It is strongly recommended that users:

* **Test the upgrade on a non-production environment**: Before upgrading a production system, test the upgrade process in a non-production environment to identify and address any issues.
* **Back up your data**: Ensure that you have a reliable backup of your ZenML data before attempting the upgrade.

What's Changed
* Optimizing the CI workflows by bcdurak in https://github.com/zenml-io/zenml/pull/3145
* Adding 0.68.0 to the migration tests by bcdurak in https://github.com/zenml-io/zenml/pull/3144
* Move step durations to body by schustmi in https://github.com/zenml-io/zenml/pull/3046
* Docs on ZenML setup by strickvl in https://github.com/zenml-io/zenml/pull/3100
* Remove wrongly set Model.was_created_in_this_run attribute by schustmi in https://github.com/zenml-io/zenml/pull/3129
* Allow specifying run tags in pipeline configuration by schustmi in https://github.com/zenml-io/zenml/pull/3130
* Fix materializer type compatibility check during loading by schustmi in https://github.com/zenml-io/zenml/pull/3105
* [docs] Add icons to headers in docs by wjayesh in https://github.com/zenml-io/zenml/pull/3149
* fix icons and remove redundant file by wjayesh in https://github.com/zenml-io/zenml/pull/3150
* Merge 0.68.1 release into develop by schustmi in https://github.com/zenml-io/zenml/pull/3153
* Allow filtering pipeline runs by stack component by schustmi in https://github.com/zenml-io/zenml/pull/3142
* Allow artifact response as step input by schustmi in https://github.com/zenml-io/zenml/pull/3134
* Filter component by user name by schustmi in https://github.com/zenml-io/zenml/pull/3126
* [docs] Restructure how-to section to make it more readable by wjayesh in https://github.com/zenml-io/zenml/pull/3147
* ZenML Pro web login implementation by stefannica in https://github.com/zenml-io/zenml/pull/3141
* Scalability improvements: Reduce misc/hydration server requests by schustmi in https://github.com/zenml-io/zenml/pull/3093
* Fix in-process main module source loading by schustmi in https://github.com/zenml-io/zenml/pull/3119
* Catch assertion in GH library by schustmi in https://github.com/zenml-io/zenml/pull/3160
* Enable cache precomputation for run templates by schustmi in https://github.com/zenml-io/zenml/pull/3156
* Add LLM and ML engineering books to README by htahir1 in https://github.com/zenml-io/zenml/pull/3159
* Add helper method to quickly create run template from pipeline by schustmi in https://github.com/zenml-io/zenml/pull/3155
* Add CLI command to export stack requirements by schustmi in https://github.com/zenml-io/zenml/pull/3158
* Scalability improvements: Increment artifact version server side by schustmi in https://github.com/zenml-io/zenml/pull/3095
* Update OpenAI integration by safoinme in https://github.com/zenml-io/zenml/pull/3163
* Remove deprecated torch version constraint by safoinme in https://github.com/zenml-io/zenml/pull/3166
* vLLM model deployer by dudeperf3ct in https://github.com/zenml-io/zenml/pull/3032
* Don't initialize client during flavor sync by schustmi in https://github.com/zenml-io/zenml/pull/3168
* Cleanup materializer temporary directories after step execution by schustmi in https://github.com/zenml-io/zenml/pull/3162
* Fix langchain in API docs by avishniakov in https://github.com/zenml-io/zenml/pull/3171
* Finetuning guide by strickvl in https://github.com/zenml-io/zenml/pull/3157
* Fix mypy issue vllm evidently by safoinme in https://github.com/zenml-io/zenml/pull/3169
* Add artifact version batch request by schustmi in https://github.com/zenml-io/zenml/pull/3164
* Add missing section links by strickvl in https://github.com/zenml-io/zenml/pull/3172
* Fix uvloop mypy by avishniakov in https://github.com/zenml-io/zenml/pull/3174
* Multiple output versions for a step outputs by avishniakov in https://github.com/zenml-io/zenml/pull/3072
* Simplify Metadata handling by AlexejPenner in https://github.com/zenml-io/zenml/pull/3096
* assign value to component_name in preset stack registration by hirekk in https://github.com/zenml-io/zenml/pull/3178
* Updating the template versions with `zenml login` by bcdurak in https://github.com/zenml-io/zenml/pull/3177
* Better input artifacts typing by avishniakov in https://github.com/zenml-io/zenml/pull/3099
* Refactor environment setup and caching by safoinme in https://github.com/zenml-io/zenml/pull/3077
* Fix spelling errors by safoinme in https://github.com/zenml-io/zenml/pull/3181
* Prevent some race conditions by schustmi in https://github.com/zenml-io/zenml/pull/3167
* Update stack deployments with latest features by stefannica in https://github.com/zenml-io/zenml/pull/3183
* Terraform best practices by htahir1 in https://github.com/zenml-io/zenml/pull/3131
* Fix sagemaker pipeline URLs by stefannica in https://github.com/zenml-io/zenml/pull/3176
* Fix lightning orchestrator for multi-step pipelines by wjayesh in https://github.com/zenml-io/zenml/pull/3170
* Port bugfixes from 2497 by avishniakov in https://github.com/zenml-io/zenml/pull/3179
* Removing the `enable_cache` from the config files by bcdurak in https://github.com/zenml-io/zenml/pull/3184
* Don't pass tags to step config by schustmi in https://github.com/zenml-io/zenml/pull/3186
* New `log_metadata` function, new `oneof` filtering, additional `run_metadata` filtering by bcdurak in https://github.com/zenml-io/zenml/pull/3182

New Contributors
* hirekk made their first contribution in https://github.com/zenml-io/zenml/pull/3178

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.68.1...0.70.0

Bug fixes

Fixes an issue with some partially cached pipelines running on remote orchestrators.

What's Changed
* Remove unavailable upstream steps during cache precomputation by schustmi in https://github.com/zenml-io/zenml/pull/3146

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.68.0...0.68.1

Highlights

- **Stack Components on the Dashboard:** We're bringing back stack components. With this release, you will get access to the list of your stack components on the ZenML dashboard. More functionality is going to follow in the next releases.
- **Client-Side Caching:** Implemented client-side computation for cached steps, significantly reducing time and costs associated with remote orchestrator spin-up.
- **Streamlined Onboarding Process:** Unified the starter and production setup into a single sequential flow, providing a more intuitive user experience.
- **BentoML Integration:** Updated to version 1.3.5 with enhanced containerization support.
- **Artifact Management:** Introduced `register_artifact` function enabling direct linking of existing data in the artifact store, particularly useful for tools like PyTorch-Lightning that manage their own checkpoints.
- **Enhanced Error Handling:** Added Error Boundary to visualization components for improved reliability and user experience.

Additional Features and Improvements

- Added multiple access points for deleting pipeline runs
- Improved pipeline detail view functionality
- Improved service account handling for Kaniko image builder

Breaking Changes and Deprecations

- Discontinued Python 3.8 support
- Removed legacy pipeline and step interface
- Removed legacy post execution workflow
- Removed legacy dashboard option
- Removed `zenml stack up/down` CLI commands
- Removed `zenml deploy` and `zenml <stack-component> deploy`
- Removed `StepEnvironment` class
- Removed the option to specify a specific model version for step output artifacts using the `ArtifactConfig` class
- Removed the option to use the `ExternalArtifact` class to load an artifact from a model version
- Removed `Client.list_runs`, replacing it with `Client.list_pipeline_runs`
- Removed `ArtifactVersionResponse.read`, replacing it with `ArtifactVersionResponse.load`

Documentation Updates

Added new guides for the following topics:

- Kubernetes per-pod configuration
- Factory generation of artifact names
- Common stacks best practices
- Azure 1-click dashboard deployment
- ZenML server upgrade best practices
- Custom Dataset classes and Materializers
- Comprehensive ZenML Pro documentation
- Image building optimization during pipeline runs
- Enhanced BentoML integration documentation

What's Changed

Highlights

- **Improved Sagemaker Orchestrator:** Now supports warm pools for AWS Sagemaker, enhancing performance and reducing startup times for TrainingJobs.
- **New DAG Visualizer:** Shipped major enhancements to the DAG Visualizer for Pipeline Runs:
- Preview of the actual DAG before pipeline completion
- Visual adjustments for improved clarity
- Real-time updates during pipeline execution
- **Environment Variable References in Configurations:** Introduced the ability to reference environment variables in both code and configuration files using the syntax ${ENV_VARIABLE_NAME}, increasing flexibility in setups.
- **Enhanced UX for Major Cloud Providers:** Displaying direct pipeline/log URL when working with major cloud platforms.
- **Skypilot with Kubernetes Support:** Added compatibility for running Skypilot orchestrator on Kubernetes clusters.
- **Updated Deepchecks Integration:** The Deepchecks integration has been refreshed with the latest features and improvements.


Features and Improvements

- **AWS Integration:**
- Added permissions to workflow to enable assuming AWS role.
- Fixed expired credentials error when using the docker service connector.
- **Error Handling:** Improved error messages for stack components of uninstalled integrations.
- **API Key Management:** Added an option to write API keys to a file instead of using the CLI.

Pipeline Execution:

- Implemented fixes for executing steps as single step pipelines.
- Added filter option for templatable runs.
- Added additional filtering options for pipeline runs.
- MLflow Integration: Linked registered models in MLflow with the corresponding MLflow run.
- Analytics: Added missing analytics event to improve user insights.

Documentation Updates

- Updated documentation for various integrations including:
- Lightning AI orchestrator
- Kubeflow
- Comet experiment tracker
- Neptune
- Hugging Face deployer
- Weights & Biases (wandb)
- Added documentation for run templates.
- Fixed incorrect method name in Pigeon docs.
- Various small documentation fixes and improvements.

Bug Fixes

- Fixed YAML formatting issues.
- Resolved RBAC issues for subpages in response models.
- Fixed step output annotation in Discord test.
- Addressed MLFlow integration requirements duplication.
- Fixed Lightning orchestrator functionality.

What's Changed
* Error message for stack components of uninstalled integrations by bcdurak in https://github.com/zenml-io/zenml/pull/2996
* Enable migration testing for version 0.66.0 by schustmi in https://github.com/zenml-io/zenml/pull/2998
* Add permissions to workflow to enable assuming AWS role by schustmi in https://github.com/zenml-io/zenml/pull/2999
* Add option to write api key to file instead of CLI by schustmi in https://github.com/zenml-io/zenml/pull/3001
* Fix yaml formatting by schustmi in https://github.com/zenml-io/zenml/pull/3004
* Update ZenML Pro links for consistency.- Update ZenML Pro links for c… by htahir1 in https://github.com/zenml-io/zenml/pull/3007
* Fix incorrect method name in Pigeon docs by strickvl in https://github.com/zenml-io/zenml/pull/3008
* Fixes for executing steps as single step pipelines by schustmi in https://github.com/zenml-io/zenml/pull/3006
* Add filter option for templatable runs by schustmi in https://github.com/zenml-io/zenml/pull/3000
* Add missing analytics event by schustmi in https://github.com/zenml-io/zenml/pull/3009
* Fix expired credentials error when using the docker service connector by schustmi in https://github.com/zenml-io/zenml/pull/3002
* Fix Lightning docs by strickvl in https://github.com/zenml-io/zenml/pull/3013
* Remove image builder warning by htahir1 in https://github.com/zenml-io/zenml/pull/3014
* Fixed kubeflow docs by AlexejPenner in https://github.com/zenml-io/zenml/pull/3018
* Update Comet experiment tracker docs by htahir1 in https://github.com/zenml-io/zenml/pull/3019
* Small docs fixes by strickvl in https://github.com/zenml-io/zenml/pull/3022
* Feature/cleanup unused file by AlexejPenner in https://github.com/zenml-io/zenml/pull/3023
* MLFlow integration requirements duplicate fix by bcdurak in https://github.com/zenml-io/zenml/pull/3011
* Fix Neptune docs by htahir1 in https://github.com/zenml-io/zenml/pull/3026
* Fix huggingface deployer docs by htahir1 in https://github.com/zenml-io/zenml/pull/3024
* Fix step output annotation in Discord test by wjayesh in https://github.com/zenml-io/zenml/pull/3029
* Fix RBAC for subpages in response models by schustmi in https://github.com/zenml-io/zenml/pull/3031
* Allow env variable placeholders in configurations by schustmi in https://github.com/zenml-io/zenml/pull/3003
* Leverage warm pools for AWS Sagemaker by avishniakov in https://github.com/zenml-io/zenml/pull/3027
* Updated wandb docs by htahir1 in https://github.com/zenml-io/zenml/pull/3030
* Add hyperlint by htahir1 in https://github.com/zenml-io/zenml/pull/3035
* Bump NLP template by avishniakov in https://github.com/zenml-io/zenml/pull/3036
* Add additional filtering options by schustmi in https://github.com/zenml-io/zenml/pull/2951
* Bump starter template version by schustmi in https://github.com/zenml-io/zenml/pull/3038
* Docs for run templates by bcdurak in https://github.com/zenml-io/zenml/pull/3028
* Update Lightning AI orchestrator documentation by strickvl in https://github.com/zenml-io/zenml/pull/3016
* Add default value for PipelineRun.is_templatable by schustmi in https://github.com/zenml-io/zenml/pull/3040
* Use a generic OAuth2 client credentials flow to login to the Cloud API by stefannica in https://github.com/zenml-io/zenml/pull/3041
* fix lightning orchestrator by safoinme in https://github.com/zenml-io/zenml/pull/3010
* Linking registered models in MLflow with the corresponding MLflow run by aiakide in https://github.com/zenml-io/zenml/pull/3020
* Bugfixing mlflow registry linting issue by bcdurak in https://github.com/zenml-io/zenml/pull/3043
* Enhancing the orchestrator UX for major cloud providers by bcdurak in https://github.com/zenml-io/zenml/pull/3005
* Skypilot with Kubernetes by safoinme in https://github.com/zenml-io/zenml/pull/3033
* Update deepchecks integration by wjayesh in https://github.com/zenml-io/zenml/pull/2987

New Contributors
* aiakide made their first contribution in https://github.com/zenml-io/zenml/pull/3020

**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.66.0...0.67.0