Xformers

Fixed
- Much faster fused dropout [facebookresearch/xformers164]
- Fused dropout repeatability [facebookresearch/xformers173]

Added
- Embedding weight tying option [facebookresearch/xformers172]

Fixed
- Dropout setting not properly passed in many attentions [facebookresearch/xformers123]

Fixed
- Fix self attention optimization not being triggered, broken residual path [facebookresearch/xformers119]
- Improve speed by not using contiguous Tensors when not needed [facebookresearch/xformers119]

Added
- Attention mask wrapper [facebookresearch/xformers113]
- ViT comparison benchmark [facebookresearch/xformers117]

Fixed
- Homogenizing the masks, additive or bool [facebookresearch/xformers79][facebookresearch/xformers85][facebookresearch/xformers86]
- Fix causality flag not being respected [facebookresearch/xformers103]
- Enabling FusedLayerNorm by default in the factory if Triton is available
- Fixing Favor with fp16
- Fixing Favor trainability

Added
- Fused dropout/bias/activation layer [facebookresearch/xformers58]
- Fused layernorm used by default in the factory [facebookresearch/xformers92]

Fixed
- Nystrom causal attention [facebookresearch/xformers75]

Fixed
- More robust blocksparse [facebookresearch/xformers24]

Added
- Rotary embeddings [facebookresearch/xformers32]
- More flexible layernorm [facebookresearch/xformers50]