Optimum

* Add OpenVINO Tokenizers by apaniukov 513

* Introduce the OpenVINO quantization configuration by AlexKoff88 https://github.com/huggingface/optimum-intel/pull/538


* Enable model OpenVINO export by echarlaix in https://github.com/huggingface/optimum-intel/pull/557

python
from diffusers import StableDiffusionPipeline
from optimum.exporters.openvino import export_from_model

model_id = "runwayml/stable-diffusion-v1-5"
model = StableDiffusionPipeline.from_pretrained(model_id)

export_from_model(model, output="ov_model", task="stable-diffusion")

* Update optimum-intel required version by echarlaix in https://github.com/huggingface/optimum/pull/1521
* Swin2sr onnx by baskrahmer in https://github.com/huggingface/optimum/pull/1492
* Fix Falcon ONNX export with alibi by fxmarty in https://github.com/huggingface/optimum/pull/1524
* Fix whisper v3 ONNX export by fxmarty in https://github.com/huggingface/optimum/pull/1525
* Add new fusion argument to fix compatibility with onnxruntime v1.16.2 by echarlaix in https://github.com/huggingface/optimum/pull/1535
* Add depth-estimation w/ DPT+GLPN by xenova in https://github.com/huggingface/optimum/pull/1529

The codebase is fully validated for the latest version of Habana SDK, SynapseAI v1.14.0.

- Upgrade to SynapseAI 1.14 https://github.com/huggingface/optimum-habana/pull/664 regisss


Stable Diffusion XL

SDXL is now supported and optimized for Gaudi.

- Stable Diffusion XL for Gaudi 619 dsocek
- Update for SDXL Turbo support 634 atakaha


Textual inversion fine-tuning

An example of textual-inversion fine-tuning has been added.

- Add Textual Inversion fine-tuning script 243 regisss


TRL

The 🤗 [TRL library](https://github.com/huggingface/trl) is now supported on Gaudi for performing DPO and SFT.

- Add DPO and SFT of TRL support in Gaudi and example 601
- Restructure example/trl/stack_llama_2 for generic DPO 635 libinta
- Add DPO of TRL in README.md 652 libinta
- Add seed in DPO for reproduce the training result 646 sywangyi


Full bf16 evaluation

Full bf16 evaluation inside the trainer can now be performed like in Transformers.

- Adding support for bf16_full_eval 610 bhargaveede


Text-generation pipeline

A text-generation pipeline fully optimized for Gaudi has been added.

- Text-Generation Pipeline Example 526 sjagtap1803


Model optimizations

- Enhances llama performance by removing the 'cast_f32_to_bf16' operation 564 kalyanjk
- Refactoring LLama Attention and mlp layers 589 bgoldberg-habana
- Support for FlashAttention in Llama2 584 wszczurekhabana
- Integrate Habana flash attention to Llama2-70B finetune 596 mandy-li
- Enabling T5ForConditionalGeneration Inference using static shapes 425 bhargaveede
- Avoid falcon perf drop from PR607 when BS=1 schoi-habana
- Enable fused rmsnorm in bf16 for llama 621 puneeshkhanna
- Flash attention enhancement of repeatKV 626 puneeshkhanna
- Update repeat KV llama logic for better TP-4 performance 639 puneeshkhanna
- Falcon changes for v1.14.0 release 654 schoi-habana


TGI

TGI on Gaudi has been moved to a dedicated repo: https://github.com/huggingface/tgi-gaudi

- Update tokenizer for tgi 572 hsubramony
- Remove redundant requirements 575 hsubramony
- Change next_token_chooser to HeterogeneousNextTokenChooser for TGI 574 yeonsily
- Remove TGI folder from Optimum Habana 597 regisss


Various fixes

- Fix messed up README for llama2-70b 571 mandy-li
- Fix Diffusers tests 570 ssarkar2
- Fix fp8 command in text-generation README 586 regisss
- Fix wav2vec inference bug 588 skaulintel
- Fix hash_with_views error 587 bgoldberg-habana
- Add dataset disposal of b-mc2/sql-create-context for codegen and fix zero3 lora save issue 552 sywangyi
- Fix gptj training issue 594 BaihuiJin
- Fix DataLoaderDispatcher issue in Gaudi 600 sywangyi
- Fix for Falcon error from PR 587 608 schoi-habana
- Falcon graph compilation error fix for when bs>1 607 regisss
- Fix crash if gaudi_config is not passed to GaudiTrainer 613 sywangyi
- Fix flash attention output for llama for padded batched inputs 623 puneeshkhanna
- Fix backward error in DDP when running reward model finetune in RLHF 507 sywangyi
- Fix dpo graph compile error in evaluation 630 sywangyi
- Fix error in run_image_classification.py 631 regisss
- Fix RLHF llama rewarding modeling backward issue 612 sywangyi
- Fix SD example so that custom bf16 ops can be used 642 regisss
- Fix SD2 test 647 regisss
- Fix typo in README 656 yeonsily
- Fix error in PR654 661 schoi-habana
- Fix compile error for torch_cmpile for llama 662 jiminha
- Fix SDXL test 666 regisss


Others

- Remove red crosses in model table 577 regisss
- Misc changes for transformers tests 581 ankurneog
- Remove delete_doc_comment workflows 582 regisss
- Pin PEFT for the languge-modeling example 591 regisss
- Remove workarounds to have causal_mask in uint8 for GPT2, GPT-J and CodeGen 592 regisss
- Change Synapse validated version in README 603 regisss
- Dyn prompt afterrefactor 543 ssarkar2
- In peft, only the trainable parameters need to be saved 576 sywangyi
- Add inheritance in Diffusers pipelines 611 regisss
- Update generation config to enable flash attention for inference 609 puneeshkhanna
- Remove setting of PT_HPU_LAZY_MODE=2 in training_args.py 625 vivekgoe
- Remove hpu:X notation untill fully supported by bridge 637 hsubramony
- Add use_flash_attention to Llama2-70B finetuning command in README 640 mandy-li
- Enable master_port selecting for DeepSpeed and MPI 641 yangulei
- Enabling Graphs in Wav2Vec AC training 622 bhargaveede
- Add changes to support FSDP 598 vivekgoe
- Run Llama2 with torch.compile on Gaudi2 616 kausikmaiti
- Hqt 648 bgoldberg-habana

IPEX models


python
from optimum.intel import IPEXModelForCausalLM
from transformers import AutoTokenizer, pipeline

model_id = "Intel/q8_starcoder"
model = IPEXModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
results = pipe("He's a dreadful magician and")



* Add IPEX models by echarlaix in 516 / 534 / 536

* Add IPEX models by ofirzaf in 542 / 543 / 544





Fixes
* Fix position_ids initialization for first inference of stateful models by eaidova in https://github.com/huggingface/optimum-intel/pull/532
* Relax requirements to have registered normalized config for decoder models 537 by eaidova in https://github.com/huggingface/optimum-intel/pull/537

Patch release for `transformers==4.34.1` compatibility. We will do a release next week for `transformers==4.35` compatibility and new features. Please bear with us!

* Falcon BetterTransformer requires transformers>=4.34 by fxmarty https://github.com/huggingface/optimum/pull/1431
* Fix arg in bettertransformer llama attention by SunMarc 1421
* Update Transformers dependency for Habana extra by regisss 1508
* temporarily pin to transformers<4.35 by fxmarty https://github.com/huggingface/optimum/commit/616931019b9bd7546918a48d475a07efb92f51b1

* Fix provider availability check on ORT 1.16.0 release by fxmarty in https://github.com/huggingface/optimum/pull/1403
* Fix ONNX Runtime quantization compatibility for onnxruntime v1.16.0 by echarlaix in https://github.com/huggingface/optimum/pull/1405