Llama
The Llama 3.1 models are released by Meta and come in three flavours: 8B, 70B, and 405B.
To get an overview of Llama 3.1, please visit the [Hugging Face announcement blog post](https://huggingface.co/blog/llama31).
We release a [repository of llama recipes](https://github.com/huggingface/huggingface-llama-recipes) to showcase usage for inference, total and partial fine-tuning of the different variants.
Chameleon
The Chameleon model was proposed in [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818v1) by META AI Chameleon Team. Chameleon is a Vision-Language Model that use vector quantization to tokenize images which enables the model to generate multimodal output. The model takes images and texts as input, including an interleaved format, and generates textual response.
* Chameleon: add model by zucchini-nlp in 31534
ZoeDepth
The ZoeDepth model was proposed in [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://arxiv.org/abs/2302.12288) by Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller. ZoeDepth extends the [DPT](https://huggingface.co/docs/transformers/main/en/model_doc/dpt) framework for metric (also called absolute) depth estimation. ZoeDepth is pre-trained on 12 datasets using relative depth and fine-tuned on two domains (NYU and KITTI) using metric depth. A lightweight head is used with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier.
* Add ZoeDepth by NielsRogge in 30136
Hiera
Hiera was proposed in [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://arxiv.org/abs/2306.00989) by Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer
The paper introduces “Hiera,” a hierarchical Vision Transformer that simplifies the architecture of modern hierarchical vision transformers by removing unnecessary components without compromising on accuracy or efficiency. Unlike traditional transformers that add complex vision-specific components to improve supervised classification performance, Hiera demonstrates that such additions, often termed “bells-and-whistles,” are not essential for high accuracy. By leveraging a strong visual pretext task (MAE) for pretraining, Hiera retains simplicity and achieves superior accuracy and speed both in inference and training across various image and video recognition tasks. The approach suggests that spatial biases required for vision tasks can be effectively learned through proper pretraining, eliminating the need for added architectural complexity.
* Adding hiera by Namangarg110 in 30356
Agents
Our ReactAgent has a specific way to return its final output: it calls the tool final_answer, added to the user-defined toolbox upon agent initialization, with the answer as the tool argument. We found that even for a one-shot agent like CodeAgent, using a specific final_answer tools helps the llm_engine find what to return: so we generalized the final_answer tool for all agents.
* Adds final answer tool for all agents by aymeric-roucher in 31703
Now if your code-based agent (like ReactCodeAgent) defines a function at step 1, it will remember the function definition indefinitely. This means your agent can create its own tools for later re-use!
* Code agent: allow function persistence between steps by aymeric-roucher in 31769
This is a transformative PR: it allows the agent to regularly run a specific step for planning its actions in advance. This gets activated if you set an int for planning_interval upon agent initialization. At step 0, a first plan will be done. At later steps (like steps 3, 6, 9 if you set planning_interval=3 ), this plan will be updated by the agent depending on the history of previous steps. More detail soon!
* Agents planning by aymeric-roucher in 31702
Notable changes to the codebase
A significant RoPE refactor was done to make it model agnostic and more easily adaptable to any architecture.
It is only applied to Llama for now but will be applied to all models using RoPE over the coming days.
* Llama: RoPE refactor by gante in 32135
Breaking changes
TextGenerationPipeline and tokenizer kwargs
🚨🚨 This PR changes the code to rely on the tokenizer's defaults when these flags are unset. This means some models using `TextGenerationPipeline` previously did not add a `<bos>` by default, which (negatively) impacted their performance. In practice, this is a breaking change.
Example of a script changed as a result of this PR:
py
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-9b-it", torch_dtype=torch.bfloat16, device_map="auto")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe("Foo bar"))
* 🚨🚨 TextGenerationPipeline: rely on the tokenizer default kwargs by gante in 31747
Bugfixes and improvements
* Fix post gemma merge by ArthurZucker in 31660
* Fix float out of range in owlvit and owlv2 when using FP16 or lower precision by aliencaocao in 31657
* [docs] Llama3 by stevhliu in 31662
* [HybridCache] Fix `get_seq_length` method by sanchit-gandhi in 31661
* don't zero out the attention_mask when using sliding window with flash attention by winglian in 31670
* Fix Gemma2 4d attention mask by hiyouga in 31674
* Fix return_dict in encodec by jla524 in 31646
* add gather_use_object arguments by SangbumChoi in 31514
* Gemma capping is a must for big models by ArthurZucker in 31698
* Add French version of run scripts tutorial by jadechoghari in 31483
* dependencies: `keras-nlp<0.14` pin by gante in 31684
* remove incorrect urls pointing to the llava repository by BiliBraker in 31107
* Move some test files (`tets/test_xxx_utils.py`) to `tests/utils` by ydshieh in 31730
* Fix mistral ONNX export by fxmarty in 31696
* [whisper] static kv cache by sanchit-gandhi in 31166
* Make tool JSON schemas consistent by Rocketknight1 in 31756
* Fix documentation for Gemma2. by jbornschein in 31682
* fix assisted decoding by jiqing-feng in 31401
* Requires for torch.tensor before casting by echarlaix in 31755
* handle (processor_class, None) returned by ModelPatterns by molbap in 31753
* Gemma 2: Update slow tests by gante in 31759
* Add ignore_errors=True to trainer.py rmtree in _inner_training_loop by njbrake in 31668
* [fix bug] logits's shape different from label's shape in preprocess_logits_for_metrics by wiserxin in 31447
* Fix RT-DETR cache for generate_anchors by qubvel in 31671
* Fix RT-DETR weights initialization by qubvel in 31724
* `pytest_num_workers=4` for some CircleCI jobs by ydshieh in 31764
* Fix Gemma2 types by hiyouga in 31779
* Add torch_empty_cache_steps to TrainingArguments by aliencaocao in 31546
* Fix ClapProcessor to merge feature_extractor output into the returned BatchEncoding by mxkopy in 31767
* Fix serialization for offloaded model by SunMarc in 31727
* Make tensor device correct when ACCELERATE_TORCH_DEVICE is defined by kiszk in 31751
* Exclude torch.compile time from metrics computation by zxd1997066 in 31443
* Update CometCallback to allow reusing of the running experiment by Lothiraldan in 31366
* Fix gemma tests by ydshieh in 31794
* Add training support for SigLIP by aliencaocao in 31495
* Repeating an important warning in the chat template docs by Rocketknight1 in 31796
* Allow FP16 or other precision inference for Pipelines by aliencaocao in 31342
* Fix galore lr display with schedulers by vasqu in 31710
* Fix Wav2Vec2 Fairseq conversion (weight norm state dict keys) by gau-nernst in 31714
* Depth Anything: update conversion script for V2 by pcuenca in 31522
* Fix Seq2SeqTrainer crash when BatchEncoding data is None by iohub in 31418
* Bump certifi from 2023.7.22 to 2024.7.4 in /examples/research_projects/decision_transformer by dependabot[bot] in 31813
* Add FA2 and `sdpa` support for SigLIP by qubvel in 31499
* Bump transformers from 4.26.1 to 4.38.0 in /examples/tensorflow/language-modeling-tpu by dependabot[bot] in 31837
* Bump certifi from 2023.7.22 to 2024.7.4 in /examples/research_projects/lxmert by dependabot[bot] in 31838
* Fix typos by omahs in 31819
* transformers.fx.symbolic_trace supports inputs_embeds by fxmarty in 31574
* Avoid failure `TFBlipModelTest::test_pipeline_image_to_text` by ydshieh in 31827
* Fix incorrect accelerator device handling for MPS in `TrainingArguments` by andstor in 31812
* Mamba & RecurrentGemma: enable strict signature by gante in 31549
* Deprecate `vocab_size` in other two VLMs by zucchini-nlp in 31681
* FX symbolic_trace: do not test decoder_inputs_embeds by fxmarty in 31840
* [Grounding DINO] Add processor to auto mapping by NielsRogge in 31845
* chore: remove duplicate words by hattizai in 31853
* save_pretrained: use tqdm when saving checkpoint shards from offloaded params by kallewoof in 31856
* Test loading generation config with safetensor weights by gante in 31550
* docs: typo in tf qa example by chen-keinan in 31864
* Generate: Add new decoding strategy "DoLa" in `.generate()` by voidism in 29619
* Fix `_init_weights` for `ResNetPreTrainedModel` by ydshieh in 31851
* Update depth estimation task guide by merveenoyan in 31860
* Bump zipp from 3.7.0 to 3.19.1 in /examples/research_projects/decision_transformer by dependabot[bot] in 31871
* Add return type annotation to PreTrainedModel.from_pretrained by mauvilsa in 31869
* Revert "Fix `_init_weights` for `ResNetPreTrainedModel`" by ydshieh in 31868
* Bump certifi from 2023.7.22 to 2024.7.4 in /examples/research_projects/visual_bert by dependabot[bot] in 31872
* add warning when using gradient_checkpointing with FSDP full shard by yundai424 in 31578
* Add conversion for interleave llava by zucchini-nlp in 31858
* remove duplicate words in msg by yukionfire in 31876
* Fix file type checks in data splits for contrastive training example script by npyoung in 31720
* Fix failed tests in 31851 by ydshieh in 31879
* fix: Removed `duplicate` field definitions in some classes by Sai-Suraj-27 in 31888
* Push sharded checkpoint to hub when `push_to_hub=True` in `TrainingArguments` by SunMarc in 31808
* [RT-DETR] Add resources by NielsRogge in 31815
* Modify `warnings` in a `with` block to avoid flaky tests by ydshieh in 31893
* Add a condition for nested_detach by haikuoxin in 31855
* InstructBlipVideo: Update docstring by zucchini-nlp in 31886
* Fixes to alternating SWA layers in Gemma2 by turboderp in 31775
* Processor accepts any kwargs by zucchini-nlp in 31889
* [`ConvertSlow`] make sure the order is preserved for addedtokens by ArthurZucker in 31902
* [`Gemma2`] Support FA2 softcapping by ArthurZucker in 31887
* Fix missing methods for Fuyu by Isotr0py in 31880
* fix: Fixed the `1st argument` name in classmethods by Sai-Suraj-27 in 31907
* add gather_use_object arguments II by SangbumChoi in 31799
* Add warning message for beta and gamma parameters by OmarManzoor in 31654
* Fix fx tests with inputs_embeds by fxmarty in 31862
* Refactor flash attention implementation in transformers by ArthurZucker in 31446
* Generate: fix `SlidingWindowCache.reset()` by gante in 31917
* 🚨 fix(SigLip): remove spurious exclusion of first vision output token by transmissions11 in 30952
* Allow `Trainer.get_optimizer_cls_and_kwargs` to be overridden by apoorvkh in 31875
* [Bug Fix] fix qa pipeline tensor to numpy by jiqing-feng in 31585
* Docker: TF pin on the consistency job by gante in 31928
* fix prompt strip to support tensors and np arrays by AvivSham in 27818
* Fix `GenerationMixin.generate` compatibility with pytorch profiler by fxmarty in 31935
* Generate: remove deprecated code due to `Cache` and `cache_position` being default by gante in 31898
* Generate: v4.42 deprecations 🧹🧹 by gante in 31956
* Whisper: move to tensor cpu before converting to np array at decode time by gante in 31954
* fix: Removed a wrong key-word argument in `sigmoid_focal_loss()` function call by Sai-Suraj-27 in 31951
* Generate: handle `logits_warper` update in models with custom generate fn by gante in 31957
* fix: Fixed the arguments in `create_repo()` function call by Sai-Suraj-27 in 31947
* Notify new docker images built for circleci by ydshieh in 31701
* Avoid race condition by ydshieh in 31973
* Masking: remove flakiness from test by gante in 31939
* Generate: doc nits by gante in 31982
* Fix the incorrect permutation of gguf by PenutChen in 31788
* Cambricon MLUs support SDPA and flash_attn by huismiling in 31102
* Speedup model init on CPU (by 10x+ for llama-3-8B as one example) by muellerzr in 31771
* [tests] fix deepspeed zero3 config for `test_stage3_nvme_offload` by faaany in 31881
* Fix bad test about slower init by muellerzr in 32002
* Tests: remove cuda versions when the result is the same 🧹🧹 by gante in 31955
* Bug report update by gante in 31983
* add flash-attn deterministic option to flash-attn>=2.4.1 by junrae6454 in 31961
* fix: Fixed incorrect dictionary assignment in `src/transformers/__init__.py` by Sai-Suraj-27 in 31993
* Bug report update -- round 2 by gante in 32006
* Fix gather when collecting 'num_input_tokens_seen' by CodeCreator in 31974
* Fix if else and *actually* enable superfast init by muellerzr in 32007
* SpeechEncoderDecoder doesn't support param buffer assignments by muellerzr in 32009
* Fix tests skip by qubvel in 32012
* Fixed `log messages` that are resulting in TypeError due to too many arguments by Sai-Suraj-27 in 32017
* Fix typo in classification function selection logic to improve code consistency by moses in 32031
* doc: fix broken BEiT and DiNAT model links on Backbone page by dvrogozh in 32029
* Pass missing arguments to `SeamlessM4Tv2ConformerEncoderLayer.forward()` when gradient checkpointing is enabled by anferico in 31945
* Add language to word timestamps for Whisper by robinderat in 31572
* Add `sdpa` and FA2 for CLIP by qubvel in 31940
* unpin `numpy<2.0` by ydshieh in 32018
* Chameleon: minor fixes after shipping by zucchini-nlp in 32037
* Bump scikit-learn from 1.0.2 to 1.5.0 in /examples/research_projects/decision_transformer by dependabot[bot] in 31458
* Bump scikit-learn from 1.1.2 to 1.5.0 in /examples/research_projects/codeparrot/examples by dependabot[bot] in 32052
* [mistral] Support passing `head_dim` through config (and do not require `head_dim * num_heads == hidden_size`) by xenova in 32050
* Add torch.compile Support For Mamba by zhenglongjiepheonix in 31247
* fix: Removed `duplicate entries` in a dictionary by Sai-Suraj-27 in 32041
* docs: Fixed 2 links in the docs along with some minor fixes by Sai-Suraj-27 in 32058
* Llava: add default chat templates by zucchini-nlp in 31691
* [Chameleon, Hiera] Improve docs by NielsRogge in 32038
* Incorrect Whisper long-form decoding timestamps by kamilakesbi in 32003
* [mistral] Fix FA2 attention reshape for Mistral Nemo by xenova in 32065
* VideoLLaVa: fix chat format in docs by zucchini-nlp in 32083
* Fix progress callback deepcopy by fozziethebeat in 32070
* Fixes to chameleon docs by merveenoyan in 32078
* Add image-text-to-text task guide by merveenoyan in 31777
* Support generating with fallback for short form audio in Whisper by kamilakesbi in 30984
* Disable quick init for deepspeed by muellerzr in 32066
* Chameleon: not supported with fast load by zucchini-nlp in 32091
* Fix tests after `huggingface_hub` 0.24 by Wauplin in 32054
* Fix shard order by b-chu in 32023
* Generate: store special token tensors under a unique variable name by gante in 31980
* fix: Replaced deprecated `mktemp()` function by Sai-Suraj-27 in 32123
* Mention model_info.id instead of model_info.modelId by Wauplin in 32106
* [generate] fix eos/pad id check on mps devices by sanchit-gandhi in 31695
* Fix failing test with race condition by Rocketknight1 in 32140
* Update `ko/_toctree.yml` and remove `custom_tools.md` to reflect latest changes by jungnerd in 31969
* fix: Fixed raising `TypeError` instead of `ValueError` for invalid type by Sai-Suraj-27 in 32111
* [RoBERTa] Minor clarifications to model doc by bt2513 in 31949
* Return assistant generated tokens mask in apply_chat_template by yonigottesman in 30650
* Don't default to other weights file when use_safetensors=True by amyeroberts in 31874
* set warning level to info for special tokens have been added by ArthurZucker in 32138
* Add new quant method by SunMarc in 32047
* Add llama3-llava-next-8b to llava_next conversion script by jamt9000 in 31395
* LLaVaNeXT: pad on right if training by zucchini-nlp in 32134
* Remove `trust_remote_code` when loading Libri Dummy by sanchit-gandhi in 31748
* [modelling] remove un-necessary transpose for fa2 attention by sanchit-gandhi in 31749
* Fix mask creations of `GPTNeoX` and `GPT2` by vasqu in 31944
* Add method to retrieve used chat template by KonradSzafer in 32032
* Add YaRN and Dynamic-YaRN RoPE Scaling Methods by mig-mfreitas in 30910
* Disable quick init for TapasPreTrainedModel by daniellok-db in 32149
* Modify resize_token_embeddings to ensure output type is same as input by bayllama in 31979
* gguf conversion add_prefix_space=None for llama3 by itazap in 31937
* Fix flash attention speed issue by Cyrilvallez in 32028
* Fix video batching to videollava by merveenoyan in 32139
* Added mamba.py backend by alxndrTL in 30139
* Rename Phi-3 rope scaling type by garg-amit in 31436
* Revert "Incorrect Whisper long-form decoding timestamps " by sanchit-gandhi in 32148
* Fix typing to be compatible with later py versions by amyeroberts in 32155
* feat(cache): StaticCache uses index_copy_ to avoid useless copy by tengomucho in 31857
* Added additional kwarg for successful running of optuna hyperparameter search by DeF0017 in 31924
* Enhancing SFT Training Efficiency Using Packing and FlashAttention2 with Position IDs by RhuiDih in 31629
Significant community contributions
The following contributors have made significant changes to the library over the last release:
* aliencaocao
* Fix float out of range in owlvit and owlv2 when using FP16 or lower precision (31657)
* Add torch_empty_cache_steps to TrainingArguments (31546)
* Add training support for SigLIP (31495)
* Allow FP16 or other precision inference for Pipelines (31342)
* voidism
* Generate: Add new decoding strategy "DoLa" in `.generate()` (29619)
* Namangarg110
* Adding hiera (30356)