Zenml

This release finally brings model agnostic automatic evaluation to ZenML! Now you can easily use [TFMA](https://github.com/tensorflow/model-analysis) with any model type to produce evaluation visualizations. This means you can now use TFMA with PyTorch or Scikit - a big win for automated sliced evaluation! It also introduces a new language for differentiation between features, raw features, labels and predictions, in addition to solving a few big bugs in the `examples` directory! Read more below.

As has been the case in the last few releases, this release is yet another **breaking upgrade**.

For those upgrading from an older version of ZenML, we ask to please delete their old `pipelines` dir and `.zenml` folders and start afresh with a `zenml init`.

If only working locally, this is as simple as:


cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/


And then another ZenML init:


pip install --upgrade zenml
cd zenml_enabled_repo
zenml init


New Features
* Added a new interface into the trainer step called [`test_fn`](https://github.com/maiot-io/zenml/blob/b333b0bba7602e40a49168cc21c6405294386262/zenml/steps/trainer/base_trainer.py#L121) which is utilized to produce model predictions and save them as test results

* Implemented a new evaluator step called [`AgnosticEvaluator`](https://github.com/maiot-io/zenml/blob/b333b0bba7602e40a49168cc21c6405294386262/zenml/steps/evaluator/agnostic_evaluator.py) which is designed to work regardless of the model type as long as you run the `test_fn` in your trainer step

* The first two changes allow torch trainer steps to be followed by an agnostic evaluator step, see the example [here](https://github.com/maiot-io/zenml/blob/main/examples/pytorch/run.py).

* Proposed a new naming scheme, which is now integrated into the built-in steps, in order to make it easier to handle feature/label names.

* Modified the [`TorchFeedForwardTrainer`](https://github.com/maiot-io/zenml/blob/b333b0bba7602e40a49168cc21c6405294386262/zenml/steps/trainer/pytorch_trainers/torch_ff_trainer.py) to showcase how to use TensorBoard in conjunction with PyTorch


Bug Fixes + Refactor
* Refactored how ZenML treats relative imports for custom steps. Now, rather than doing absolute imports like:
python
from examples.scikit.step.trainer import MyScikitTrainer

One can also do the following:
python
from step.trainer import MyScikitTrainer


ZenML automatically figures out the absolute path of the module based on the root of the directory.

* Updated the [Scikit Example](https://github.com/maiot-io/zenml/tree/main/examples/scikit), [PyTorch Lightning Example](https://github.com/maiot-io/zenml/tree/main/examples/pytorch_lightning), [GAN Example](https://github.com/maiot-io/zenml/tree/main/examples/gan) accordingly. Now they should work according to their README's.

Big shout out to SaraKingGH in issue 55 for raising the above issues!

This release is a big design change and refactor. It involves a significant change in the Configuration file structure, meaning this is a **breaking upgrade**.

For those upgrading from an older version of ZenML, we ask to please delete their old `pipelines` dir and `.zenml` folders and start afresh with a `zenml init`.

If only working locally, this is as simple as:


cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/


And then another ZenML init:


pip install --upgrade zenml
cd zenml_enabled_repo
zenml init


New Features
* Introduced another higher-level pipeline: The [NLPPipeline](https://github.com/maiot-io/zenml/blob/main/zenml/pipelines/nlp_pipeline.py). This is a generic
NLP pipeline for a text-datasource based training task. Full example of how to use the NLPPipeline can be found [here](https://github.com/maiot-io/zenml/tree/main/examples/nlp)
* Introduced a [BaseTokenizerStep](https://github.com/maiot-io/zenml/blob/main/zenml/steps/tokenizer/base_tokenizer.py) as a simple mechanism to define how to train and encode using any generic
tokenizer (again for NLP-based tasks).
* Introduced a new [HuggingFace](https://huggingface.co/) integration, with the first concrete implementation of the BaseTokenizerStep, i.e., the [HuggingFaceTokenizer](https://github.com/maiot-io/zenml/blob/main/zenml/steps/tokenizer/hf_tokenizer.py).
* Show-cased how to use [HuggingFace](https://huggingface.co/) with the ZenML TrainerStep in the [NLP Example](https://github.com/maiot-io/zenml/blob/main/examples/nlp/training/trainer.py).

Bug Fixes + Refactor
* Significant change to imports: Now imports are way simpler and user-friendly. E.g. Instead of:
python
from zenml.core.pipelines.training_pipeline import TrainingPipeline


A user can simple do:

python
from zenml.pipelines import TrainingPipeline


The caveat is of course that this might involve a re-write of older ZenML code imports.

Note: Future releases are also expected to be breaking. Until announced, please expect that upgrading ZenML versions may cause older-ZenML generated pipelines to behave unexpectedly.

Special shout-out to nicholasmaiot for major contributions to this release!

This release is a significant one as it includes the first version of the AWS integration. It allows you to use ZenML to launch an EC2 instance as an orchestrator and execute a ZenML pipeline possibly coupled with an S3 artifact store and RDS metadata store.

It is a new feature and it does not include any breaking changes.

In order to install ZenML with the AWS integration attached, you can follow:

pip install --upgrade zenml[aws]
zenml init


New Features
* [OrchestratorAWSBackend](https://github.com/maiot-io/zenml/blob/main/zenml/core/backends/orchestrator/aws/orchestrator_aws_backend.py) implemented to launch an EC2 instance as the orchestrator.
* While you are using the new orchestrator backend, you may use S3 and RDS.
* Implemented an [example](https://github.com/maiot-io/zenml/tree/main/examples/aws_orchestrated) which covers the basic process if you would like to start testing it right away.
Bug Fixes + Refactor
* For more advanced use-cases, more examples will follow in the future.
* Numerous small bugs and refinements.

Earlier release to get the PostgreSQL datasource out quicker.

To upgrade:

pip install --upgrade zenml


New Features
* [sci-kit learn](https://github.com/maiot-io/zenml/tree/main/examples/scikit) example.
* [PostgreSQL Datasource](https://github.com/maiot-io/zenml/blob/main/zenml/core/datasources/postgres_datasource.py) added.

Bug Fixes + Refactor
* Slight change to telemetry utils -> Now opt-out also sends a signal.

This release is a big design change and refactor. It involves a significant change in the Configuration file structure, meaning this is a **breaking upgrade**. For those upgrading from 0.2.0, we ask to please delete their old `pipelines` dir and `.zenml` folders and start afresh with a `zenml init`.

If only working locally, this is as simple as:


cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/


And then another init:


pip install --upgrade zenml
zenml init


New Features
* [BatchInferencePipeline](https://github.com/maiot-io/zenml/tree/main/examples/batch_inference) added for offline batch inference use-cases.
* [Google Cloud Platform Bootstrapping Terraform](https://github.com/maiot-io/zenml/tree/main/examples/bootstrapping/gcp) script added for one-command bootstrapping of ZenML on GCP.
* `DeployPipeline` added to deploy a pipeline directly without having to create a `TrainingPipeline`.

Bug Fixes + Refactor
* Now you can run pipelines from within any subdirectory in the repo.
* Relaxed restriction on custom steps having sub-directories with their module.
* Relationship between `Datasource` and `Data Step` refined.
* Numerous small bugs and refinements to facilitate flexible API design.

Note: Future releases are also expected to be breaking. Until announced, please expect that upgrading ZenML versions may cause older-ZenML generated pipelines to behave unexpectedly.

This new release is a major one. Its the first to introduce our new integrations system, which is meant to be used to extend ZenML with various other ML/MLOps libraries easily. The first big advantage one gets is :rocket: PyTorch Support :rocket:!


pip install --upgrade zenml


And to enable the PyTorch extension:


pip install zenml[pytorch]


New Features
* Introduced integrations for ZenML with the [extra_requires](https://setuptools.readthedocs.io/en/latest/setuptools.html) setuptools paradigm.
* Added PyTorchTrainer support with easily extendable `TorchBaseTrainer` [example](https://github.com/maiot-io/zenml/tree/main/examples/pytorch).
* Restructured trainer steps to be more intuitive to extend from Tensorflow and PyTorch. Now, we have a `TrainerStep`, followed by `TFBaseTrainerStep` and `TorchBaseTrainerStep`.
* The `input_fn` of the TorchTrainer have implemented in a way that it can ingest from a tfrecords file. This marks one of the few projects out there
that have native support for ingesting the TFRecords format into PyTorch directly.

Bug Fixes
* Fixed an issue with `Repository.get_zenml_dir()` that caused any pipeline creates below root level to fail on creation.

Documentation Annoucement
The [docs](https://docs.zenml.io) are almost complete! We are at 80% completion. Keep an eye out as we update with more details on how to use/extend ZenML and [let us know via slack](https://zenml.io/slack-invite) if there is something missing!