Torchserve

Highlights:
+ **Native windows support** - Added [support](https://github.com/pytorch/serve/blob/master/docs/torchserve_on_win_native.md) for TorchServe on Windows 10 pro and Windows Server 2019
+ **KFServing Integration** - Added support for v1 KFServing predict and explain [APIs](https://github.com/pytorch/serve/tree/master/kubernetes/kfserving) with auto-scaling and canary deployments for serving models in Kubeflow/KFServing
+ **MLFlow-TorchServe:** New [MLflow TorchServe deployment plugin](https://github.com/mlflow/mlflow-torchserve) for serving models for MLflow MLOps lifecycle
+ **Captum explanations** - Added explain API for [Captum](https://captum.ai/) model interpretability of different models
+ **AKS Support** - Added support for TorchServe deployment on [Azure Kubernetes Service](https://github.com/pytorch/serve/tree/master/kubernetes/AKS)
+ **GKE Support** - Added support for TorchServe deployment on [Google Kubernetes Service](https://github.com/pytorch/serve/tree/master/kubernetes/GKE)
+ **gRPC support** - Added support for gRPC based management and inference [APIs](https://github.com/pytorch/serve/blob/master/docs/grpc_api.md)
+ **Request Envelopes** - Added support for [request envelopes](https://github.com/pytorch/serve/blob/master/docs/request_envelopes.md) which parses request from multiple Model serving frameworks like Seldon, KFServing, without any modifications in the handler code

Highlights:
+ **Kubernetes Support** - Torchserve deployment in Kubernetes using [Helm Charts](https://helm.sh/) and a [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
+ **Prometheus metrics** - Added Prometheus as the default metrics framework
+ **Requirements.txt support​** - Added support to specify model specific dependencies as a requirements file within a mar archive; Cleanup of unused parameters and addition of relevant ones for torch-model-archiver
+ **Pytorch Scripted Models Support** - Scripted model versions added to model zoo; Added testing for scripted models
+ **Default Handler Refactor: (breaking changes)** The default handlers have been refactored for code reuse and enhanced post-processing support. More details in _Backwards Incompatible Changes_ section below
+ **Windows Support** - Added support for torchserve on windows subsystem for Linux
+ **AWS Cloud Formation Support** - Added support for multi-node [AutoScaling Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) deployment, behind an [Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/) using [Elastic File System](https://aws.amazon.com/efs/) as the backing store
+ **Benchmark and Testing Enhancements** - Added models in benchmark and sanity tests, support for throughput with batch processing in benchmarking, support docker for jmeter and apache benchmark tests
+ **Regression Suite Enhancements** - Added new POSTMAN based test cases for API and pytest based intrusive test cases
+ **Docker Improvements** - Consolidated dev and codebuild dockerfiles
+ **Install and Build Script Streamlining** - Unified install scripts, added code coverage and sanity script
+ **Python Linting** - More exhaustive python linting checks across Torchserve and Model Archiver

Backwards Incompatible Changes
+ **Default Handler Refactor**:
* The default handlers have been refactored for code reuse and enhanced post-processing support. The output format for some of the following examples/models has been enhanced to include additional details like score/class probability.
* [object detector](https://github.com/pytorch/serve/tree/issue_411/examples/object_detector/fast-rcnn)
* [image segmentor](https://github.com/pytorch/serve/tree/issue_411/examples/image_segmenter)
* The following default-handlers have been equipped with batch support. Due to batch support, [resnet_152_batch](https://github.com/pytorch/serve/tree/issue_411/examples/image_classifier/resnet_152_batch) example is not a custom handler example anymore.
* image_classifier
* object_detector
* image_segmenter
* The [index_to_name.json](https://github.com/pytorch/serve/blob/issue_411/docs/default_handlers.md#index_to_namejson) file use for the class to name mapping has been standardized across vision/text related default handlers
* Refactoring and code reuse have resulted into reduced boilerplate code in all the `serve/examples`.
* [Custom handler](https://github.com/pytorch/serve/blob/issue_411/docs/custom_service.md) documentation has been restructured and enhanced to facilitate the different possible ways to build simple or complex custom handlers

Highlights:
+ **HuggingFace BERT Example** - Support for HuggingFace Models demonstrated with examples under examples/ directory.
+ **Waveglow Example** - Support for Nvidia Waveglow model demonstrated with examples under examples/ directory.
+ **Model Zoo** - Model Zoo with model archives created from popular pre-trained models from PyTorch Model Zoo
+ **AWS Cloud Formation Support** - Support added for spinning up TorchServe Model Server on an EC2 instance via the convenience of AWS Cloud Formation Template.
+ **Snakeviz Profiler** - Support for Profiling TorchServe Python execution via snakevize profiler for detailed execution time reporting.
+ **Docker improvements** - Docker image size optimization, detailed docs for running docker.
+ **Regression Test Suite** - Detailed Regression Test Suite to allow comprehensive tests for all supported REST APIs. Automating this test helps faster regression detection.
+ **Detailed Unit Test Reporting** - Detailed breakdown of Unit Test Reports from gradle build system.
+ **Installation Process Streamlining** - Easier user onboarding with detailed documentation for installation
+ **Documentation Clean up** - Refactored documentation with clear instructions
+ **GPU Device Assignment** - Object Detection Model now correctly runs on multiple GPU devices
+ **Model Store Clean-up** - Clean up Model store of all artifacts for a deleted model

TorchServe (Experimental) v0.1.0 Release Notes

This is the first release of TorchServe (Experimental), a new open-source model serving framework under the PyTorch project ([RFC 27610](https://github.com/pytorch/pytorch/issues/27610)).


Highlights
+ **Clean APIs** - Support for an [Inference API](https://github.com/pytorch/serve/blob/master/docs/inference_api.md) for predictions and a [Management API](https://github.com/pytorch/serve/blob/master/docs/management_api.md) for managing the model server.

+ **Secure Deployment** - Includes HTTPS support for secure deployment.

+ **Robust model management capabilities** - Allows full configuration of models, versions, and individual worker threads via command line interface, config file, or run-time API.

+ **Model archival** - Provides tooling to perform a ‘model archive’, a process of packaging a model, parameters, and supporting files into a single, persistent artifact. Using a simple command-line interface, you can package and export in a single ‘.mar’ file that contains everything you need for serving a PyTorch model. This `.mar’ file can be shared and reused. Learn more [here](https://github.com/pytorch/serve/tree/master/model-archiver).

+ **Built-in model handlers** - Support for [model handlers](https://github.com/pytorch/serve/tree/master/model-archiver#handler) covering the most common use-cases (image classification, object detection, text classification, image segmentation). TorchServe also supports [custom handlers](https://github.com/pytorch/serve/blob/master/docs/custom_service.md)

+ **Logging and Metrics** - Support for robust [logging](https://github.com/pytorch/serve/blob/master/docs/logging.md) and real-time [metrics](https://github.com/pytorch/serve/blob/master/docs/metrics.md) to monitor inference service and endpoints, performance, resource utilization, and errors. You can also generate custom logs and define [custom metrics](https://github.com/pytorch/serve/blob/master/docs/metrics.md#custom-metrics-api).

+ **Model Management** - Support for [management of multiple models](https://github.com/pytorch/serve/blob/master/docs/server.md#serving-multiple-models-with-torchserve) or multiple versions of the same model at the same time. You can use model versions to roll back to earlier versions or route traffic to different versions for A/B testing.

+ **Prebuilt Images** - Ready to go Dockerfiles and Docker images for deploying TorchServe on CPU and NVIDIA GPU based environments. The latest Dockerfiles and images can be found [here](https://hub.docker.com/r/pytorch/torchserve/).

Platform Support
- Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+

Known Issues
+ The default object detection handler only works on cuda:0 device on GPU machines [104](https://github.com/pytorch/serve/issues/104)
+ For torchtext based models, the sentencepiece dependency fails for MacOS with python 3.8 [232](https://github.com/pytorch/serve/issues/232)

Getting Started with TorchServe
+ Additionally, you can get started at [pytorch.org/serve](https://pytorch.org/serve/) with installation instructions, tutorials and docs.
+ Lastly, if you have questions, please drop it into the [PyTorch discussion forums](https://discuss.pytorch.org/c/deployment/) using the ‘deployment’ tag or file an issue on [GitHub](https://github.com/pytorch/serve) with a way to reproduce.