Deepsparse

This is a patch release for 0.6.0 that contains the following changes:

Users no longer experience crashes
- when running the ReduceSum operation in the DeepSparse Engine.
- when running operations on tensors that are 8- or 16-bit integers, or booleans, on AVX2.

New Features:

* DeepSparse Engine optimized for Sparse FP32 BERT.
* Optimized BERT model collection now in the [SparseZoo](https://sparsezoo.neuralmagic.com/?domain=nlp&sub_domain=question_answering&page=1).
* Performance improvement example includes 5x increased throughput on [PruneBERT](https://arxiv.org/abs/2005.07683) (281 seq/sec) compared to dense BERT (53 seq/sec) at batch size 32 and sequence length 128 (AWS c5.12xlarge).
* Optimized Tanh operator support provided.
* Hugging Face transformers pipeline [APIs added for NLP models](https://github.com/neuralmagic/deepsparse/tree/main/examples/huggingface-transformers).
* Hugging Face transformers [examples added for benchmarking, deploying, and sample application](https://docs.neuralmagic.com/main/source/model-pages/nlp-bert.html#sparse-inference).
* Ultralytics YOLOv5 [example support added](https://docs.neuralmagic.com/main/source/model-pages/cv-detection-yolov5.html#sparse-inference).

Changes:

* Performance improvements made for:
- all networks when running on multi-socket machines, especially those with large outputs.
- batched Softmax and Reduce operators with many threads available.
- Reshape operators when multiple dimensions are combined into one or one dimension is split into multiple.
- stacked matrix multiplications by supporting more input layouts.
* YOLOv3 example integration was generalized to ultralytics-yolo in support of both V3 and V5.

Resolved Issues:

* Engine now runs on architectures with more than one NUMA node per socket.

Known Issues:

* None

This is a patch release for 0.5.0 that contains the following changes:

- resolution to address an issue that caused a performance regression on YOLOv5 and could have affected the correctness of some models.

New Features:

* None

Changes:

* Performance optimizations implemented for binary elementwise operations, where both inputs come from the same source buffer. One of the inputs may have intermediate unary operations.
* Performance optimizations implemented for binary elementwise operations where one of the inputs is a constant scalar.
* Small performance improvement for large batch sizes (> 64) on quantized ResNet.

Resolved Issues:

* Assertion `deepsparse num_sockets` removed when too many sockets were requested, causing users to experience a crash.
* Rare assertion failure fixed when a nonlinearity appeared between an elementwise addition and a convolution or gemm.
* Broken URLs for classification and detection examples updated in the contained READMEs.

Known Issues:

* None

New Features:

* New operator support implemented for Expand.
* Slice operator support for positive step sizes. Only slice operations that operate on a single axis are supported. Previously, slice was only supported for constant tensors and step size equal to one.

Changes:

* Memory usage of compiled models reduced.
* Memory layout for matrix multiplications in Transformers optimized.
* Precision for swish and sigmoid operations improved.
* Runtime performance improved for some networks whose outputs are immediately preceded by transpose operators.
* Runtime performance of softmax operations improved.
* Readme redesigned for better clarity on the repository's purpose.

Resolved Issues:

* Using the multi-stream scheduler, when more threads were selected than the number of cores on the system, it no longer causes a performance hit.
* Neural Magic dependencies upgrade to intended bug versions instead of minor versions.

Known Issues:

* None

This is a patch release for 0.3.0 that contains the following changes:

- Docs updated for new Discourse and Slack links
- Check added for supported Python version so DeepSparse does not improperly install on unsupported systems