Onednn

This is a release candidate for oneDNN v3.5. Please provide feedback and submit defect reports via [Github issues](https://github.com/oneapi-src/oneDNN/issues/new/choose).

Performance Optimizations

* Intel Architecture Processors:
* Improved performance for 4th generation Intel Xeon Scalable processors (formerly Sapphire Rapids).
* Improved performance for the future Intel Xeon Scalable processors (code-named Sierra Forest and Granite Rapids).
* Improved performance of group normalization primitive.
* Improved performance of matmul primitive with sum post-op for batched cases on processors with Intel AMX instruction set support.
* Improved performance of the following subgraphs with Graph API:
* Multi-Query Attention (MQA).
* Scaled Dot Product Attention (SDPA), including the variant with `select` operation.
* `LayerNorm` + `Multiply` + `Quantize` produced by SmoothQuant algorithm.
* `Convolution` + `Sigmoid` + `Multiply` with mixed precisions.
* Intel Graphics Products:
* Improved performance for Processor Graphics based on Xe2 architecture.
* Improved performance for the Intel Data Center GPU Max Series (formerly Ponte Vecchio).
* Improved performance for Intel Arc graphics (formerly Alchemist and DG2) and the Intel Data Center GPU Flex Series (formerly Arctic Sound).
* Improved RNN primitive performance for LSTM cell case.
* Improved performance of f8_e4m3 data type emulation on Intel Data Center GPU Max Series (formerly Ponte Vecchio).

* AArch64-based Processors:
* Improved convolution forward propagation, matmul, and softmax performance for processors with SVE support.
* Improved bf16 matmul performance with Arm Compute Library (ACL).
* Improved eltwise primitive performance with `gelu_erf` algorithm with ACL.

Functionality
* Introduced sum and binary post-ops support for layer normalization primitive. This functionality is currently implemented on CPUs only.
* Introduced support for int4 data type and extended quantization model with support for grouped scales and zero points.
* Introduced fp64 matmul support. This functionality is currently implemented on Intel GPUs only.
* Extended [floating point math mode API](https://oneapi-src.github.io/oneDNN/dev_guide_attributes_fpmath_mode.html) to support weight decompression scenarios. See [matmul weights decompression example](https://github.com/intel-innersource/libraries.performance.math.onednn/blob/main/examples/tutorials/matmul/weights_decompression_matmul.cpp) to get started. New floating mode is supported in the following configurations:
* bfloat16 matmul with int8 weights on Intel CPUs.
* float16 and bfloat16 matmul with int8 or int4 weights on Intel GPUs.
* **[experimental]** Introduced [microkernel API](https://oneapi-src.github.io/oneDNN/ukernels.html) for Intel Architecture Processors. This API exposes internal mechanisms used in matmul and convolution implementation to expert users.

Usability
* Extended error messages for engine and memory objects creation errors.
* Extended verbose mode diagnostics with information on dispatching decisions for all primitives.
* Introduced support for clang++ host compiler in SYCL builds.
* Introduced API for tensor serialization and deserialization.
* Extended verbose mode diagnostics for Graph API with information on pattern matcher decisions.
* Introduced OpenCL runtime support for Graph API.
* Added support for building oneDNN with installed Arm Compute Library (ACL).

Validation
* Extended benchdnn with support for tensor tags in RNN primitive validation.

Thanks to these Contributors
This release contains contributions from the project core team as well as AngryLoki, Crefeda Rodrigues cfRod, Daniel Richard G. iskunk, deepeshfujitsu, Dylan Angus dylan-angus-codeplay, Emanuele Rocca ema, Hernan Martinez hmartinez82, John Osorio kala855, Jonathan Deakin jondea, kasturedeeksha, Kentaro Kawakami kawakami-k, Nikita Shulga malfet, Radu Salavat Radu2k, Renato Barros Arantes renato-arantes, Roman Zhukov rozhukov, Shreyas-fuj Shreyas-fuj, Sunita Nadampalli snadampal, Tadej Ciglarič t4c1, Vineel Abhinav vineelabhinav, vishwascm. We would also like to thank everyone who asked questions and reported issues.

This is a patch release containing the following changes to v3.4.3:
* Fixed an issue with host compiler version detection in SYCL configurations (fcaa1b44110280919674c801f4c063f5651b5760)

This is a patch release containing the following changes to v3.4.2:
* Fixed GPU detection issues on systems with several different Intel GPUs (0fb7e6ed4f32e5d89832b2bd742bbf834cd296ed)

This is a patch release containing the following changes to v3.4.1:
* Fixed performance regression in deconvolution on processors with Intel AVX-512 instruction set (307b35bf7fa03ef7f030481329e23dcc287bbd7f, f46fffbb35f1213e6e98c1bc1e10353232ac08ee)
* Improved performance of batched matmul with binary post-op on processors with Intel AVX-512 instruction set (d39e1b7447979340ae7a882fff376fa14c12ddaa)
* Fixed performance regression in softmax with destination memory format set to `any` on processors with Intel AVX-512 instruction set (756d3cf5a2e28c0dc052a18caf8358f3c9dc22e0)
* Fixed incorrect results in int8 deconvolution with source zero points on processors with Intel AMX instruction set (d5ddbc851aa75e36ed9f651e01185443bfa903ff)
* Fixed performance regression in convolution on processors with Intel AVX2 instruction set (2968c8948225d18c9df19c94534ff7dc4343700c)
* Improved f8_e4m3 matmul performance on Intel Data Center GPU Max Series (068f8504de78b93fea0ed71fe87bf5bc86c79724, 668abae7f109709ed7b0ac107cb46e8983a625a6, c3972ef0c49db9724d143bd486c6c7fbdc52f8a3, ad943825dcd5b50e52d37543015814421444600a)
* Fixed sporadic accuracy issues in bf16 depthwise convolution backpropagation on processors with Intel AVX-512 instruction set (01840442c5e045c7311a4144625bf01322f9e942)
* Fixed primitive creation issue for fp16 pooling backpropagation on Intel GPUs (e4737d908b56e2ca5a56d580266e7f65505c4b0d)
* Fixed failure for subgraphs with int8 matmul operation with experimental Graph Compiler on processors with Intel AMX instruction set (5ebde2e2ad2788fd373da15a2a9073526849ed0d)
* Fixed assert in experimental Graph Compiler on Windows (f53fbd164dab47c73e4c56c97f4bdd0546e47ed3, fd903aebe6917535b49c8e16598d1912ad42b09b)
* Fixed incorrect results for subgraphs with shuffle operation with experimental Graph Compiler (aef502394d43f7aae388487eee05549d28470ae4)
* Improved performance of subgraphs involving int8 matmul with experimental Graph Compiler on processors with Intel AMX support (0ca5bc557e4d1e090aeca659cfbe68a8c57ef168)
* Fixed page fault in fp16 matmul primitive on Intel Data Center GPU Max Series (5587f0820c2cc5b1eca159a7b78e8ae38ce7d7d6)
* Fixed incorrect results in dp32 deconvolution with Arm Compute Library on AArch64 processors (b7694a00a26cfe3f0d9d9b36d16edac91bfdd65b)
* Fixed performance regression in deconvolution on processors with Intel AVX2 instruction set (6f452e2ff782255ae57f91ddfaa142752de21a42)

This is a patch release containing the following changes to v3.4:
* Fixed an issue with caching and serialization of primitives in deterministic mode (7ed604a1e5688022a59444059e53a6a7967f679a)
* Introduced memory descriptor serialization API (4cad420e673f4cd49568ea7c4dd6a55e6f55794e, 929a27ae0412a0851629da70916eee360a39baac, 9b848c859a6b1d046dd63cf20f817aa9428fb483)
* Fixed incorrect results in fp64 convolution and deconvolution on Intel GPUs based on Xe-LPG architecture (ebe77b566bb1cd273e9bda99cc62063b7c2a7e45, 0b399ac42740a9c6ed458aacafdb31ce16205cbd, d748d642d7871608e09f5cee5d964ddcfc8a42ef, 9f4f3d510ddc9d639db052302be579621d46bb1f, 21a8caebb34a85074f3f8a5cef35ed85532a5bbe)
* Fixed incorrect results in reorder with large sizes on Intel CPUs and GPUs (69a111e6d835f8632ea571f3ea0e273b22488d37, 4b7236134bde1c1a71859a844eae860a71670b97, 74a343bf66a1c8f113fa8e025391aba5015c6e48)
* Reduced creation time for deconvolution primitive on Intel CPUs (bec487e4ae16b3e88382adf9574e9c62cc76d1bd, 1eab00586881f4fb6966a16f71216528ec549c11)
* Fixed performance regression in deconvolution on Intel CPUs (fbe5b97c966696a3f5be2240c0eb4592ed548036, 1dd3c6af03addefcf92ac45eddeb8becf63d6a6e)
* Removed dangling symblols from static builds (e92c4041b12e55837452327c3ebd9411dbc2e861, 6f5621aed75226b93f07879fafa6fb799a36f042)
* Fixed crash during platform detection on some AArch64-based systems (406a0798c1c5b939726a892ad5a96e20298396ca)
* Fixed performance regression in int8 deconvolution on Intel CPUs (7e50e152f21a79978b8910260e042b43941b601c)
* Fixed handling of zero points for matmul in verbose logs converter (15c791686f94291eddda7a2e24835ba1113c530a)

Performance Optimizations

* Intel Architecture Processors:
* Improved performance for 4th generation Intel Xeon Scalable processors (formerly Sapphire Rapids).
* Improved performance for the future Intel Xeon Scalable processors (code-named Sierra Forest and Granite Rapids). These optimizations are now included by default on compatible processors.
* Improved RNN primitive performance with LBR_GRU cell.
* Improved softmax performance on processors with Intel AVX2 or Intel AVX-512 instruction set support.
* Improved fp32 inner product performance on processors with Intel AVX2 instruction set support.
* Improved fp32, fp16, bf16 matmul primitive performance on processors with Intel AVX-512 and Intel AMX instruction set support.
* Improved int8 matmul performance with transposed A tensor.
* Improved performance of resampling primitive on processors with Intel AVX2 instruction set support.
* Improved performance of int8 convolution with post-ops.
* Optimized batch matmul with binary post-op and broadcast mask `1` and `14`.
* Improved the Scaled Dot Product Attention (SDPA) subgraph performance with Graph API.
* Improved performance of subgraphs including `matmul` and `add` operations and mixed int8 and bfloat16 data types with Graph API.
* **[experimental]** Improved performance of `reduction`, `softmax` and `layernorm` operations with experimental Graph Compiler backend.
* **[experimental]** Improved performance for llama2 MLP subgraph with experimental Graph Compiler backend.

* Intel Graphics Products:
* Introduced initial optimizations for Processor Graphics based on Xe2 architecture.
* Improved performance for the Intel Data Center GPU Max Series (formerly Ponte Vecchio).
* Improved performance for Intel Arc graphics (formerly Alchemist and DG2) and the Intel Data Center GPU Flex Series (formerly Arctic Sound).
* Improved matmul performance for cases relevant to Large Language Models (LLMs) and Transformer-like models.
* Improved convolution performance for cases relevant to the Stable Diffusion model.
* Improved RNN primitive performance.
* Improved pooling forward propagation performance.
* Improved batched matmul performance for cases with 5 dimensions or more.

* AArch64-based Processors:
* Added an option to build oneDNN with macOS Accelerate library to improve performance on Apple silicon.
* Improved reorder primitive performance with Compute Library for the Arm architecture (ACL).
* Improved bf16 inner product product primitive performance with ACL.

Functionality
* Introduced [GPT-Q support](https://github.com/igorsafo/oneDNN/tree/rfcs-gpt-quantization/rfcs/20231108-gpt-quantization) to improve Large Language Models (LLMs) performance with compressed weights. Optimized implementation is available for Intel Graphics Products and support [matmul with int8 weight compression](https://oneapi-src.github.io/oneDNN/page_weights_decompression_matmul_cpp.html#doxid-weights-decompression-matmul-cpp).
* Introduced [fp8 data type](https://oneapi-src.github.io/oneDNN/dev_guide_data_types.html) support in primitives and Graph API. Optimized implementation is available for Intel Data Center GPU Max Series (formerly Ponte Vecchio).
* Introduced support for fp16 and bf16 scale and shift arguments for layer normalization. Optimized implementation is available for Intel Graphics Products.
* **[experimental]** Introduced unstructured sparsity support for processors with Intel AMX support relying on VCOMPRESS/VPEXPAND instructions.
* Intel Graphics Products
* Introduced support for Intel Data Center GPU Max 1550VG
* Introduced PReLU post-op support for inner product and matmul primitives.

Usability
* Added opt-in [deterministic mode](https://oneapi-src.github.io/oneDNN/dev_guide_attributes_deterministic.html) support. Deterministic mode guarantees that results are bitwise identical between runs in a fixed environment.
* Introduced [accumulation mode](https://oneapi-src.github.io/oneDNN/dev_guide_attributes_accumulation_mode.html) control.
* Extended oneDNN verbose diagnostics with information on dispatching decisions in convolution and matmul implementations.
* Extended verbose diagnostics for Graph API with information for operation schema check results and pattern matching results.
* Reduced RNN primitive memory consumption on GPUs.
* Added examples demonstrating use of oneDNN Graph API in eager mode use cases.
* Extended tensor constructor in Graph API to support memory allocation and management by the library.
* Introduced new API and environment variable to manage [Graph API constant tensor cache capacity](https://oneapi-src.github.io/oneDNN/dev_guide_constant_tensor_cache.html).
* Improved the efficiency of pattern matching in Graph API by optimizing pattern registration, reducing pattern numbers, and skipping patterns more wisely.
* Changed default optimization flags for AArch64 builds to `-mcpu=generic` to improve portability.

Validation
* Improved benchdnn performance by optimizing bottlenecks in validation code.
* Introduced `--num-streams` knob in benchdnn to support benchmarking in multi-stream scenarios.

Known Limitations
* Intel Datacenter GPU Flex Series driver for Windows has an issue resulting in program hangs or crashes when oneDNN primitives are created concurrently.
* int8 concat primitive may produce incorrect results on integrated GPUs with current GPU driver.
* fp32 pooling primitive may produce incorrect results in rare conditions on Intel Datacenter GPU Max Series with current GPU driver.
* reorder primitive causes segmentation fault for prime sizes exceeding 2^31 on Intel CPUs.
* fp64 convolution and deconvolution produces incorrect results on integrated graphics in future Intel Core processors (code name Arrow Lake)
* int8 matmul primitive creation with fp32 bias fails on Intel GPU Flex Series and Intel Arc Graphics.

Breaking Changes
* Updated minimal supported ACL version to 23.11 (was 23.02.1).

Thanks to these Contributors
This release contains contributions from the project core team as well as Alexander Grund Flamefire, David Svantesson davsva01, Fadi Arafeh fadara01, Hugh Delaney hdelan, Ilya Lavrenov ilya-lavrenov, Jacob Kahn jacobkahn, Nathan John Sircombe nSircombe, Renato Barros Arantes renato-arantes, Sergey Shalnov shssf, Sunita Nadampalli snadampal, and Svetlozar Georgiev sgeor255. We would also like to thank everyone who asked questions and reported issues.