Zenml

Earlier release to get the PostgreSQL datasource out quicker.

To upgrade:

pip install --upgrade zenml


New Features
* [sci-kit learn](https://github.com/maiot-io/zenml/tree/main/examples/scikit) example.
* [PostgreSQL Datasource](https://github.com/maiot-io/zenml/blob/main/zenml/core/datasources/postgres_datasource.py) added.

Bug Fixes + Refactor
* Slight change to telemetry utils -> Now opt-out also sends a signal.

This release is a big design change and refactor. It involves a significant change in the Configuration file structure, meaning this is a **breaking upgrade**. For those upgrading from 0.2.0, we ask to please delete their old `pipelines` dir and `.zenml` folders and start afresh with a `zenml init`.

If only working locally, this is as simple as:


cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/


And then another init:


pip install --upgrade zenml
zenml init


New Features
* [BatchInferencePipeline](https://github.com/maiot-io/zenml/tree/main/examples/batch_inference) added for offline batch inference use-cases.
* [Google Cloud Platform Bootstrapping Terraform](https://github.com/maiot-io/zenml/tree/main/examples/bootstrapping/gcp) script added for one-command bootstrapping of ZenML on GCP.
* `DeployPipeline` added to deploy a pipeline directly without having to create a `TrainingPipeline`.

Bug Fixes + Refactor
* Now you can run pipelines from within any subdirectory in the repo.
* Relaxed restriction on custom steps having sub-directories with their module.
* Relationship between `Datasource` and `Data Step` refined.
* Numerous small bugs and refinements to facilitate flexible API design.

Note: Future releases are also expected to be breaking. Until announced, please expect that upgrading ZenML versions may cause older-ZenML generated pipelines to behave unexpectedly.

This new release is a major one. Its the first to introduce our new integrations system, which is meant to be used to extend ZenML with various other ML/MLOps libraries easily. The first big advantage one gets is :rocket: PyTorch Support :rocket:!


pip install --upgrade zenml


And to enable the PyTorch extension:


pip install zenml[pytorch]


New Features
* Introduced integrations for ZenML with the [extra_requires](https://setuptools.readthedocs.io/en/latest/setuptools.html) setuptools paradigm.
* Added PyTorchTrainer support with easily extendable `TorchBaseTrainer` [example](https://github.com/maiot-io/zenml/tree/main/examples/pytorch).
* Restructured trainer steps to be more intuitive to extend from Tensorflow and PyTorch. Now, we have a `TrainerStep`, followed by `TFBaseTrainerStep` and `TorchBaseTrainerStep`.
* The `input_fn` of the TorchTrainer have implemented in a way that it can ingest from a tfrecords file. This marks one of the few projects out there
that have native support for ingesting the TFRecords format into PyTorch directly.

Bug Fixes
* Fixed an issue with `Repository.get_zenml_dir()` that caused any pipeline creates below root level to fail on creation.

Documentation Annoucement
The [docs](https://docs.zenml.io) are almost complete! We are at 80% completion. Keep an eye out as we update with more details on how to use/extend ZenML and [let us know via slack](https://zenml.io/slack-invite) if there is something missing!

New Features
* Added [Kubernetes Orchestrator](https://github.com/maiot-io/zenml/tree/main/zenml/core/backends/orchestrator/kubernetes) to run pipelines on a kubernetes cluster.
* Added timeseries support with [StandardSequencerStep](https://github.com/maiot-io/zenml/blob/main/zenml/core/steps/sequencer/standard_sequencer/standard_sequencer.py).
* Added more [CLI groups] such as `step`, `datasource` and `pipelines`. E.g. `zenml pipeline list` gives list of pipelines in current repo.
* Completed a significant portion of the [Docs](https://docs.zenml.io).
* Refactored Step Interfaces for easier integrations into other libraries.
* Added a [GAN Example](https://github.com/maiot-io/zenml/tree/main/examples/gan) to showcase ImageDatasource.
* Set up base for more Trainer Interfaces like PyTorch, scikit etc.
* Added ability to see historical steps.

Bug Fixes
* All files except YAML files picked up while parsing `pipelines_dir`, in reference to concerns raised in 13.

Upcoming changes
* Next release will be a major one and will involve refactoring of design decisions that might cause backward incompatible changes to existing ZenML repos.

New Features
* Ability to add a custom image to Dataflow ProcessingBackend.

Bug Fixes
* Fixed requirements.txt and setup.py to enable local build.
* Pip package should install without any requirement conflicts now.
* Added custom docs made by Jupyter book in the `docs/book` folder.

New Features
* Launch GCP preemptible VM instances to orchestrate pipelines with OrchestratorGCPBackend. See full example [here](https://github.com/maiot-io/zenml/tree/main/examples/gcp_orchestrated/run.py).
* Train using Google Cloud AI Platform with SingleGPUTrainingGCAIPBackend. See full example [here](https://github.com/maiot-io/zenml/tree/main/examples/gcp_trained/run.py)
* Use Dataflow for distributed preprocessing. See full example [here](https://github.com/maiot-io/zenml/tree/main/examples/gcp_dataflow/run.py).
* Run pipelines locally with SQLite Metadata Store, local Artifact Store, and local Pipelines Directory.
* Native Git integration: All steps are pinned with the Git SHA of the code when the pipelines it was used in is run. See details [here](https://docs.zenml.io/repository/integration-with-git).
* All pipelines run are reproducible with a unique combination of the Metadata Store, Artifact Store and the Pipelines Directory.

Bug Fixes
* Metadata Store and Artifact Store specified in pipelines disassociated from default .zenml_config file.
* Fixed typo in default docker images constants.