Multi-model-server

In this release of MXNet Model Server, we have added the following features.

Features and Bug fixes
* Changes for batching support.
* CORS headers support added to responses.
* Handle content-type returned by the backend code and pass ContentType to the service code
* Workaround import mxnet module timeout issue. Now MXNet startup time doesn't cause significant delay upon MMS start on compute optimized hosts
* Make sure that python prints are not buffered
* Refactor metrics emission logic
* Always use utf-8 to decode bytes.
* Avoid archiving a model archive file recursively.
* Pythonpath issues for MMS
* Documentation updates

In this release of MXNet Model Server, we have added the following major features.

Features
1. Loading and Unloading models at run-time (hot loading models). This is now available via management REST API exposed by MMS. More on management API [here](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/management_api.md)
2. Independently scale number of model-worker instances serving inference requests. This is available through management REST API.
3. Improved model archive representation. More on model-archiver is [here](https://github.com/awslabs/mxnet-model-server/tree/v1.0.0/model-archiver)
4. Improved docker container images.
5. Improved performance compared to MMS v0.4 and decreased dependencies. One of the major changes is replacing monolithic architecture with separate frontend and backend. Netty is used as frontend webserver instead of Flask+GUnicorn combo. Python is for the backend.
6. Improved logging and metrics collection. Using log4j and corresponding config to control metrics, including custom user metrics. More on logging config is [here](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/logging.md)

New and updated documents:
1. [Migration document](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/migration.md) to migrate from MMS 0.4 to MMS 1.0.
1. [New Management API](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/management_api.md).
1. [Updated model zoo](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/model_zoo.md).
1. [Updated Inference API](https://github.com/awslabs/mxnet-model-server/blob/v1.0.0/docs/inference_api.md).

For further documentation, please refer [/docs](https://github.com/awslabs/mxnet-model-server/tree/v1.0.0/docs) folder

Bug fixes:
This release fixes all the bugs logged on GitHub.

New Features:
Gluon imperative model support
1. Added support for serving [Gluon based imperative models](https://github.com/awslabs/mxnet-model-server/tree/master/examples/gluon_alexnet).

Docs, Improvements, Bug fixes
Docs:
* Added documentation on Gluon model services.
* Added [alexnet in Gluon](https://github.com/awslabs/mxnet-model-server/tree/master/examples/gluon_alexnet) to model serving to examples.
* Added [Character-level Convolutional Neural Networks in Gluon](https://github.com/awslabs/mxnet-model-server/tree/master/examples/gluon_character_cnn) to model serving examples.

Improvements:
* Gluon base service class implementation. (vdantu )
* Improved Docker image setup time by , layering docker images. (vrakesh, 343 )
* Docker image now can auto-detect number of available CPUs. (vrakesh, 360 )
* Added pylint support. (vdantu)
* Use cu90mkl mxnet on cuda gpu machines by default. (vrakesh, 390 )

Bug Fixes:
* Fixed an issue, where empty folder was created when invalid model path is specified. (vrakesh, 320 )
* Docker images now do not allow multiple instances of MMS to run. (vrakesh, 337 )
* fixed pypi summary issue. (aaronmarkham, 378 )
* Fixed error propagation from custom service to MMS. (vrakesh, 387 )
* Fixed documentation bugs. (vrakesh, 401 , 402 )
* Fixed version reading issue in MMS. (vrakesh, 395 )
* Fixed post process latencies being high due to inference variables being lazy evaluated. (vrakesh, 414 )

New Features:
New CLI to interact with MMS running in a container
1. New options to start/stop/restart MMS in container.
2. Option to point to different configuration files for each MMS run.
3. Multiple bug fixes.

Optimized and pre-configured MMS container images
1. Published the container image to Docker Hub.
2. The default configuration in these containers and the example configuration in the repository are optimized for CPU and GPU AWS EC2 instances.

Bug fixes and Docs
Docs:
* README documents.
* Added docs to depict orchestrating MMS as an AWS FARGATE service.
* Added docs for optimizing the MMS configuration for different EC2 instances.

Bug Fixes:
* Corrected Readme and advanced-settings doc for MMS container (aaronmarkham )
* Documentation for optimised setup for GPU and CPU EC2 instances (ankkhedia )
* Optimized MMS GPU-container to utilize all GPUs in an GPU instance (ankkhedia )
* Documentation for launching MMS on AWS Fargate service
* Added integration tests framework (ankkhedia )
* Doc update on Production usage. Describes why Container images are better for prod. (336)
* Streamlining Container based MMS orchestration (vdantu)
* Optimized the model file downloads for container runs of MMS. (vdantu)
* Fixed bugs in mxnet-model-export (ankkhedia )

New features

ONNX model support

Model server now supports models stored in the [Open Neural Network Exchange (ONNX)](https://github.com/onnx/onnx) format. See [Export an ONNX Model](https://github.com/awslabs/mxnet-model-server/blob/master/docs/export_from_onnx.md) for details.

Cloudwatch metrics

Model server can publish host and model related metrics to [Amazon Cloudwatch](https://aws.amazon.com/cloudwatch/). See [Cloudwatch metrics](https://github.com/awslabs/mxnet-model-server/blob/master/docs/metrics.md#cloudwatch-metrics) for details.

Improvements and bug fixes

- Fixing LatencyOverall unit reporting (lupesko, 317)
- update onnx-mxnet (jesterhazy, 316)
- remove docs images (jesterhazy, 315)
- added cloudwatch metrics section (aaronmarkham, 314)
- update docker scripts (jesterhazy, 313)
- added toc, logos, and kitten image (aaronmarkham, 311)
- add unit test for hyphenated model files (jesterhazy, 308)
- Fix validate_prefix_match (knjcode, 307)
- align metrics names/units with standard cloudwatch metrics (jesterhazy, 303)
- Fix race condition when multiple gunicorn workers try to download same models (yuruofeifei, 302)
- Fix epoch number validation (knjcode, 298)
- remove License field (jesterhazy, 297)
- update error messages for model export (jesterhazy, 295)
- Fixing and updating docker setup (lupesko, 294)
- public domain image examples for SSD outputs (aaronmarkham, 291)
- refactored export info; added shortcuts and other improvements (aaronmarkham, 290)
- added four onnx-exported models to zoo; added onnx support to server intro (aaronmarkham, 288)
- Documentation updates for 0.2 (aaronmarkham, 285)
- Improve cloudwatch integration, fix several issues. (yuruofeifei, 283)
- fail fast when user tries to serve onnx model directly (jesterhazy, 280)
- fix importlib warning (254) (jesterhazy, 279)
- bump version (yuruofeifei, 250)
- Onnx and metrics docs (yuruofeifei, 244)
- Add onnx support (yuruofeifei, 240)
- Zoo updates (aaronmarkham, 234)
- Zoo updates with details for each model (aaronmarkham, 233)

Key capabilities of Model Server for Apache MXNet v0.1.5:
- Tooling to package and export all model artifacts into a single “model archive” file that encapsulates everything required for serving an MXNet model.
- Automated setup of a serving stack, including HTTP inference endpoints, MXNet-based engine, all automatically configured for the specific models being hosted.
- Pre-configured Docker images, setup with NGINX, MXNet and MMS, for scalable model serving.
- Ability to customize every step in the inference execution pipeline, from model initialization, through pre-processing and inference, and up to post-processing the model’s output.
- Real time operational metrics to monitor the inference service and endpoints, covering key metrics such as latencies, resource utilization and errors.
- OpenAPI-enabled service, that is easy to integrate with, and that can auto-generate client code for popular stacks such as Java, JavaScript, C and more.