Xformers

Fixed
- fMHA/CUTLASS: The current CUDA stream is now used by the kernel [facebookresearch/xformers491]
- fMHA/CUTLASS: Improve overall performance

Added
- SwiGLU: Added `xformers.ops.SwiGLU` and its functional counterpart (`xformers.ops.swiglu`) [facebookresearch/xformers490]
- fMHA: Possible to combine CUTLASS's forward with flash-attention's backward pass [facebookresearch/xformers469] - improves performance on A100 for K = 128
- fMHA: Add custom `xformers.ops.unbind` operator to avoid a cat in the attention block [facebookresearch/xformers458]

Added
- fMHA: Added CUTLASS-based kernel for `xformers.ops.memory_efficient_attention`. This kernel is automatically depending on the inputs, and works on any GPU after P100 [facebookresearch/xformers362]

Fixed
- Removed duplicated biases in the FusedMLP layers [facebookresearch/xformers317]
- Rotary embeddings respecting input types [facebookresearch/xformers326]
- Poolformer style instantiating useless projection layers [facebookresearch/xformers349]
- Fix layer position not being properly tracked, causing extra layernorms for programmatic xformers [facebookresearch/xformers348]
- Pass use_triton flag to LayerNorm module [facebookresearch/xformers336]

Added
- Four blocksparsity layouts from DeepSpeed [facebookresearch/xformers320]
- Support several initialization options [facebookresearch/xformers312]
- Conv2DFeedforward feedforward part [facebookresearch/xformers321]
- VisualAttention [facebookresearch/xformers329]
- Automatic blocksparse for causal attention [facebookresearch/xformers334]
- Better hierarchical transformer generation [facebookresearch/xformers345]
- Fused operations with AOTAutograd/NVFuser, integration into MLP [facebookresearch/xformers357]
- Refactor LRA code to use Pytorch Lightning [facebookresearch/xformers343]

Fixed
- Fix some torchscriptability [facebookresearch/xformers246]
- Fix FourierMix being compatible with AMP [facebookresearch/xformers258]
- Better asserts on QKV dimensions [facebookresearch/xformers264]
- Better perfs for FusedMLP and FusedLinearLayer [facebookresearch/xformers283]
- Deepnorm init missing self-attention [facebookresearch/xformers284]

Added
- Simplicial Embeddings [facebookresearch/xformers259]
- Mem efficient attention, FW pass [facebookresearch/xformers267]
- MHA benchmark
- MLP benchmark
- Move all triton kernels to triton v2 [facebookresearch/xformers272]
- Mem efficient attention, BW pass [facebookresearch/xformers281]
- Metaformer support [facebookresearch/xformers294]

Fixed
- Expose bias flag for feedforwards, same default as Timm [facebookresearch/xformers220]
- Update eps value for layernorm, same default as torch [facebookresearch/xformers221]
- PreNorm bugfix, only one input was normalized [facebookresearch/xformers233]
- Fix bug where embedding dimensions that did not match model dim would lead to a crash [facebookresearch/xformers244]

Added
- Add DeepNet (DeepNorm) residual path and init [facebookresearch/xformers227]

Added
- Compositional Attention [facebookresearch/xformers41]
- Experimental Ragged attention [facebookresearch/xformers189]
- Mixture of Experts [facebookresearch/xformers181]
- BlockSparseTensor [facebookresearch/xformers202]
- Nd-tensor support for triton softmax [facebookresearch/xformers210]

Fixed
- Bugfix Favor, single feature map [facebookresearch/xformers183]
- Sanity check blocksparse settings [facebookresearch/xformers207]
- Fixed some picklability [facebookresearch/xformers204]