Key Updates
* Performance optimizations for Transformer models
* GPT2 - Enable optimizations for Attention with Past State and Attention Mask
* BERT - Improve EmbedLayerNormalization fusion coverage
* Quantization updates
* Added new quantization operators: QLinearAdd, QAttention
* Improved quantization performance for transformer based models on CPU
* More graph fusion
* Further optimization in MLAS kernel
* Introduced pre-packing for constant Matrix B of DynamicQuantizeMatMul and Qattention
* New [Python IOBinding APIs](https://github.com/microsoft/onnxruntime/blob/master/docs/python/api_summary.rst#iobinding) (bind_cpu_input, bind_output, copy_outputs_to_cpu) allow easier benchmarking
* Users no longer need to allocate inputs and outputs on non-CPU devices using third-party allocators.
* Users no longer need to copy inputs to non-CPU devices; ORT handles the copy.
* Users can now use copy_outputs_to_cpu to copy outputs from non-CPU devices to CPU for verification.
* CUDA support for Einsum (opset12)
* ONNX Runtime Training updates
* Opset 12 support
* New [sample](https://github.com/microsoft/onnxruntime-training-examples) for training experiment using Huggingface GPT-2.
* Upgraded docker image built from the latest PyTorch release
* Telemetry is now enabled by default for Python packages and Github release zip files (C API); [see more details](https://github.com/microsoft/onnxruntime/blob/master/docs/Privacy.md#official-builds) on what/how telemetry is collected in ORT
* **[Coming soon]** Availability of Python package for ONNX Runtime 1.4 for Jetpack 4.4
Execution Providers
New Execution Providers available for preview:
* **[Preview]** [AMD MIGraphX](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/MIGraphX-ExecutionProvider.md)
* **[Preview]** [ARM NN](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/ArmNN-ExecutionProvider.md)
Contributions
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:
[snnn](https://github.com/snnn), [tianleiwu](https://github.com/tianleiwu), [edgchen1](https://github.com/edgchen1), [hariharans29](https://github.com/hariharans29), [skottmckay](https://github.com/skottmckay), [tracysh](https://github.com/tracysh), [yufenglee](https://github.com/yufenglee), [fs-eire](https://github.com/fs-eire), [codemzs](https://github.com/codemzs), [tiagoshibata](https://github.com/tiagoshibata), [yuslepukhin](https://github.com/yuslepukhin), [gwang-msft](https://github.com/gwang-msft), [wschin](https://github.com/wschin), [smk2007](https://github.com/smk2007), [prabhat00155](https://github.com/prabhat00155), [liuziyue](https://github.com/liuziyue), [liqunfu](https://github.com/liqunfu), [ytaous](https://github.com/ytaous), [iK1D](https://github.com/iK1D), [BowenBao](https://github.com/BowenBao), [askhade](https://github.com/askhade), [pranavsharma](https://github.com/pranavsharma), [faxu](https://github.com/faxu), [jywu-msft](https://github.com/jywu-msft), [ryanlai2](https://github.com/ryanlai2), [xzhu1900](https://github.com/xzhu1900), [KeDengMS](https://github.com/KeDengMS), [tlh20](https://github.com/tlh20), [smkarlap](https://github.com/smkarlap), [weixingzhang](https://github.com/weixingzhang), [jeffbloo](https://github.com/jeffbloo), [RyanUnderhill](https://github.com/RyanUnderhill), [mrry](https://github.com/mrry), [jgbradley1](https://github.com/jgbradley1), [stevenlix](https://github.com/stevenlix), [zhanghuanrong](https://github.com/zhanghuanrong), [suffiank](https://github.com/suffiank), [Andrews548](https://github.com/Andrews548), [pengwa](https://github.com/pengwa), [SherlockNoMad](https://github.com/SherlockNoMad), [orilevari](https://github.com/orilevari), [duli2012](https://github.com/duli2012), [yangchen-MS](https://github.com/yangchen-MS), [yan12125](https://github.com/yan12125), [jornt-xilinx](https://github.com/jornt-xilinx), [ashbhandare](https://github.com/ashbhandare), [neginraoof](https://github.com/neginraoof), [Tixxx](https://github.com/Tixxx), [thiagocrepaldi](https://github.com/thiagocrepaldi), [Craigacp](https://github.com/Craigacp), [mayeut](https://github.com/mayeut), [chilo-ms](https://github.com/chilo-ms), [prasanthpul](https://github.com/prasanthpul), [martinb35](https://github.com/martinb35), [manashgoswami](https://github.com/manashgoswami), [zhangxiang1993](https://github.com/zhangxiang1993), [suryasidd](https://github.com/suryasidd), [wangyems](https://github.com/wangyems), [kit1980](https://github.com/kit1980), [RandySheriffH](https://github.com/RandySheriffH), [fdwr](https://github.com/fdwr)