Transformers

This let us reorganize the example scripts completely for a cleaner codebase.

The main features of the Trainer are:
- Same user-facing API for PyTorch and TF 2
- Support for CPU, GPU, Multi-GPU, and TPU
- Easier than ever to share your fine-tuned models

**The TFTrainer was largely contributed by awesome community member jplu!** 🔥 🔥

A few additional features of the example scripts are:
- Generate argparsers from type hints on dataclasses
- Can load arguments from json files
- Logging through TensorBoard and wandb

Documentation for the Trainer is still work-in-progress, please consider contributing improvements.

TPU Support

- Both the TensorFlow and PyTorch trainers have TPU support (jplu, LysandreJik, julien-c). An additional utility is added so that the TPU scripts may be launched in a similar manner to `torch.distributed`.
- This was built with the support of jysohn23, member of the Google TPU team

---

Multilingual BART (sshleifer)

New BART checkpoint converted: this adds `mbart-en-ro model`, a BART variant finetuned on english-romanian translation.

Improved support for `huggingface/tokenizers`

- Additional tests and support has been added to `huggingface/tokenizers` tokenizers. (mfuntowicz, thomwolf)
- TensorFlow models work out-of-the-box with the new tokenizers (LysandreJik)

Decoder caching for T5 (patrickvonplaten)

Auto-regressive decoding for T5 has been greatly sped up by storing past key/value states. Work done on both PyTorch and TensorFlow.

Breaking change

This introduces a breaking change, in that it increases the default output length of T5Model and T5ForConditionalGeneration from 4 to 5 (including the past_key_value_states).

Encoder-Decoder enhancements

- Apply Encoder Decoder 1.5GB memory savings to TF as well (patrickvonplaten, translation of same work on PyTorch models by sshleifer)
- BART Summarization fine-tuning script now works for T5 as well (sshleifer)
- Clean Encoder-Decoder models with Bart/T5-like API and add generate possibility (patrickvonplaten)

Additional model architectures

Question Answering support for Albert and Roberta in TF with (Pierrci):
- Question Answering support for Albert and Roberta in TF
- TFAlbertForQuestionAnswering

Pipelines

- The question answering pipeline now handles impossible answers (bryant1410)
- Remove tqdm logging (mfuntowicz)
- Sentiment analysis pipeline can now handle more than two sequences (xxbidiao)
- Rewritten batch support in pipelines (mfuntowicz)

Text Generation pipeline (enzoampil)

Implements a text generation pipeline, `GenerationPipeline`, which works on any `ModelWithLMHead` head.

Fixes and improvements

- Clean the generate testing functions (patrickvonplaten)
- Notebooks updated in the documentation (LysandreJik)
- Fix RoBERTa/XLNet Pad Token in run_multiple_choice.py (ethanjperez)
- Fixed RoBERTa conversion script (myleott)
- Speedup torch summarization tests (sshleifer)
- Optimize causal mask using torch.where (Akababa)
- Improved benchmarking utils (patrickvonplaten)
- Fixed edge case for bert tokenization (patrickvonplaten)
- SummarizationDataset cleanup (sshleifer)
- BART: Replace config.output_past with use_cache kwarg (sshleifer)
- Better documentation for Summarization and Translation pipeline (julien-c)
- Additional documentation for model cards (julien-c)
- Fix force_download of files on Windows (calpt)
- Fix shuffling issue for distributed training (elk-cloner)
- Shift labels internally within TransfoXLLMHeadModel when called with labels (TevenLeScao)
- Remove `output_past` everywhere and replace by `use_cache` argument (patrickvonplaten)
- Added unit test for run_bart_sum (sshleifer)
- Cleaner code by factorizating a few methods back in the `PreTrainedModel` (sshleifer)
- [Bert] remove hard-coded pad token id (patrickvonplaten)
- Clean pipelines test and remove unnecessary code (patrickvonplaten)
- JITting is not compatible with PyTorch/XLA or any other frameworks that requires serialization. The JITted methods were removed (LysandreJik)
- Change newstest2013 to newstest2014 and clean up (patrickvonplaten)
- Factor out tensor conversion method in `PretrainedTokenizer` (sshleifer)
- Remove tanh torch warnings (aryanshomray)
- Fix token_type_id in BERT question-answering example (siboehm)
- Add CircleCI workflow to build docs for preview (harupy)
- Higher tolerance for past testing in T5 and TF T5 (patrickvonplaten)
- XLM tokenizer should encode with bos token (LysandreJik)
- XLM tokenizer should encode with bos token (patrickvonplaten)
- fix summarization do_predict (sshleifer)
- Encode to max length of input not max length of tokenizer for batch input (patrickvonplaten)
- Add `qas_id` to SquadResult and SquadExample (jarednielsen)
- Fix bug in run_*.py scripts: double wrap into DataParallel during eval (and-kul)
- Fix torchhub integration (julien-c)
- Fix TFAlbertForSequenceClassification classifier dropout probability (jarednielsen)
- Change uses of pow(x, 3) to pow(x, 3.0) (mneilly-et)
- Shuffle train subset for summarization example (Colanim)
- Removed the boto3 dependency (julien-c)
- Add dialogpt training tips (patrickvonplaten)
- Generation can now start with an empty prompt (patrickvonplaten)
- GPT-2 is now traceable (jazzcook15)
- Add known 3rd party to setup.cfg; removes local/circle ci isort discrepancy. (sshleifer)
- Allow a more backward compatible behavior of max_len_single_sentence and max_len_sentences_pair (thomwolf)
- Now using CDN urls for weights (julien-c)
- [Fix common tests on GPU] send model, ids to torch_device (sshleifer)
- Fix TF input docstrings to refer to tf.Tensor rather than torch.Float (jarednielsen)
- Additional metadata to traing arguments (parmarsuraj99)
- [ci] Load pretrained models into the default (long-lived) cache (julien-c)
- add timeout_decorator to tests (sshleifer)
- Added XLM-R to the multilingual section in the documentation (stefan-it)
- Better `num_labels` in configuration objects
- Updated pytorch lightning scripts (williamFalcon)
- Tests now pass with torch 1.5.0 (LysandreJik)
- Ensure fast tokenizer can construct single-element tensor without pad token (mfuntowicz)

Trainer & TFTrainer (julien-c)

ELECTRA Model (LysandreJik)

ELECTRA is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the SQuAD 2.0 dataset.

This release comes with 6 ELECTRA checkpoints:

- `google/electra-small-discriminator`
- `google/electra-small-generator`
- `google/electra-base-discriminator`
- `google/electra-base-generator`
- `google/electra-large-discriminator`
- `google/electra-large-generator`

Related:

- [Paper](https://openreview.net/pdf?id=r1xMH1BtvB)
- [Original implementation in TF 1.x](https://github.com/google-research/electra)
- 📊 [**Model cards** on huggingface.co](https://huggingface.co/models?search=electra)
- [Docs](https://huggingface.co/transformers/model_doc/electra.html)

Thanks to the author clarkkev for his help during the implementation.

Thanks to community members hfl-rc stefan-it shoarora for already sharing more fine-tuned Electra variants!

Bad word filters in `generate` (patrickvonplaten)

The `generate` method now has a bad word filter.

Fixes and improvements

- Decoder input ids are not necessary for T5 training anymore (patrickvonplaten)
- Update encoder and decoder on set_input_embedding for BART (sshleifer)
- Using loaded checkpoint with --do_predict (instead of random init) for Pytorch-lightning scripts (ethanjperez)
- Clean summarization and translation example testing files for T5 and Bart (patrickvonplaten)
- Cleaner examples (julien-c)
- Extensive testing for T5 model (patrickvonplaten)
- Force models outputs to always have batch_size as their first dim (patrickvonplaten)
- Fix for continuing training in some scripts (xeb)
- Resizing embedding matrix before sending it to the optimizer (ngarneau)
- BertJapaneseTokenizer accept options for mecab (tamuhey)
- Speed up GELU computation with torch.jit (mryab)
- fix argument order of update_mems fn in TF version (patrickvonplaten, dmytyar)
- Split generate test function into beam search, no beam search (patrickvonplaten)

T5 Model (patrickvonplaten, thomwolf )

T5 is a powerful encoder-decoder model that formats every NLP problem into a text-to-text format. It achieves state of the art results on a variety of NLP tasks (Summarization, Question-Answering, ...).

Five sets of pre-trained weights (pre-trained on a multi-task mixture of unsupervised and supervised tasks) are released. In ascending order from 60 million parameters to 11 billion parameters:

`t5-small`, `t5-base`, `t5-large`, `t5-3b`, `t5-11b`

T5 can now be used with the translation and summarization pipeline.

Related:
- [paper](https://arxiv.org/pdf/1910.10683.pdf)
- [official code](https://github.com/google-research/text-to-text-transfer-transformer)
- model available in Hugging Face's [community models](https://huggingface.co/models?search=t5)
- [docs](https://huggingface.co/transformers/model_doc/t5.html)

Big thanks to the original authors, especially craffel who helped answer our questions, reviewed PRs and tested T5 extensively.

New BART checkpoint: `bart-large-xsum` (sshleifer)

These weights are from BART finetuned on the XSum abstractive summarization challenge, which encourages shorter (more abstractive) summaries. It achieves state of the art.

BART summarization example with pytorch-lightning (acarrera94)

New example: BART for summarization, using Pytorch-lightning. Trains on CNN/DM and evaluates.

Translation pipeline (patrickvonplaten)

A new pipeline is available, leveraging the T5 model. The T5 model was added to the summarization pipeline as well.

Memory improvements with BART (sshleifer)

In an effort to have the same memory footprint and same computing power necessary to run inference on BART, several improvements have been made on the model:

- Remove the LM head and use the embedding matrix instead (~200MB)
- Call encoder before expanding input_ids (~1GB)
- SelfAttention only returns weights if config.output_attentions (~500MB)
- Two separate, smaller decoder attention masks (~500MB)
- drop columns that are exclusively pad_token_id from input_ids in `evaluate_cnn` example.

TensorFlow models may now be serialized (gthb)

Supports JSON serialization of Keras layers by overriding get_config, so that they can be sent to Tensorboard to display a conceptual graph of the model. TensorFlow models may now be saved using `model.save`, as other Keras models.

New model: XLMForTokenClassification (sakares)

A new head was added to XLM: `XLMForTokenClassification`.

New Model: BART (added by sshleifer)
Bart is one of the first Seq2Seq models in the library, and achieves state of the art results on text generation tasks, like abstractive summarization.
Three sets of pretrained weights are released:
- `bart-large`: the pretrained base model
- `bart-large-cnn`: the base model finetuned on the CNN/Daily Mail Abstractive Summarization Task
- `bart-large-mnli`: the base model finetuned on the MNLI classification task.


Related:
- [paper](https://arxiv.org/abs/1910.13461)
- model pages are at [https://huggingface.co/facebook](https://huggingface.co/facebook)
- [docs](https://huggingface.co/transformers/model_doc/bart.html)
- [blogpost](https://sshleifer.github.io/blog_v2/jupyter/2020/03/12/bart.html)

Big thanks to the original authors, especially Mike Lewis, Yinhan Liu, Naman Goyal who helped answer our questions.

Model sharing CLI: support for organizations

The huggingface API for model upload now supports [organisations](https://huggingface.co/organizations).

Notebooks (mfuntowicz)

A few beginner-oriented notebooks were added to the library, aiming at demystifying the two libraries huggingface/transformers and huggingface/tokenizers. Contributors are welcome to contribute links to their notebooks as well.

[pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning) examples (srush)

Examples leveraging pytorch-lightning were added, led by srush.
The first example that was added is the [NER example](https://github.com/huggingface/transformers/tree/master/examples/ner).
The second example is a lightning GLUE example, added by nateraw.

New model architectures: CamembertForQuestionAnswering,

- `CamembertForQuestionAnswering` was added to the library and to the SQuAD script maximeilluin
- `AlbertForTokenClassification` was added to the library and to the NER example marma

Multiple fixes were done on the fast tokenizers to make them entirely compatible with the python tokenizers (mfuntowicz)

Most of these fixes were done in the patch 2.5.1. Fast tokenizers should now have the exact same API as the python ones, with some additional functionalities.

Docker images (mfuntowicz)

Docker images for transformers were added.

Generation overhaul (patrickvonplaten)

- Special token IDs logic were improved in run_generation and in corresponding tests.
- Slow tests for generation were added for pre-trained LM models
- Greedy generation when doing beam search
- Sampling when doing beam search
- Generate functionality was added to TF2: with beam search, greedy generation and sampling.
- Integration tests were added
- `no_repeat_ngram_size` kwarg to avoid redundant generations (sshleifer)

Encoding methods now output only model-specific inputs

Models such as DistilBERT and RoBERTa do not make use of token type IDs. These inputs are not returned by the encoding methods anymore, except if explicitly mentioned during the tokenizer initialization.

Pipelines support summarization (sshleifer)
- The default architecture is `bart-large-cnn`, with the generation parameters published in the paper.

Models may now re-use the cache every time without prompting S3 (BramVanroy)

Previously all attempts to load a model from a pre-trained checkpoint would check that the S3 etag corresponds to the one hosted locally. This has been updated so that an argument `local_files_only` prevents this, which can be useful when a firewall is involved.

Usage examples for common tasks (LysandreJik)

In a continuing effort to onboard new users (new to the lib or new to NLP in general), some usage examples were added to the documentation. These usage examples showcase how to do inference on several tasks:

- NER
- Sequence classification
- Question Answering
- Causal Language Modeling
- Masked Language Modeling

Test suite on GPU (julien-c)

CI now runs on GPU. PyTorch and TensorFlow.

Padding token ID needs to be set in order to pad (patrickvonplaten)

Older tokenizers could pad even when no padding token was defined, which has been updated in this version to match the expected behavior, which is the FastTokenizers' behavior: add a pad token or raise an error when trying to batch without one.

Python >= 3.6

We're now dropping Python 3.5 support.

Community additions/bug-fixes/improvements

- Added a warning when using `add_special_tokens` with the fast tokenizer methods of encoding (LysandreJik)
- `encode_plus` was modified and tested to have the exact same behaviour as `encode`, but batches input
- Cleanup DistilBERT code (guillaume-be)
- Only use `F.gelu` for torch >= 1.4.0 (sshleifer)
- Added a `get_vocab` method to tokenizers, which can be used to retrieve all the vocabulary from the tokenizers. (joeddav)
- Correct behaviour of `special_tokens_mask` when `add_special_tokens=False` (LysandreJik)
- Removed untested `Model2LSTM` and `Model2Model` which was not working
- kwargs were passed to both model and configuration in AutoModels, which made the model crash (LysandreJik)
- Correct transfo-xl tokenization regarding punctions (patrickvonplaten)
- Better docstrings for XLNet (patrickvonplaten)
- Better operations for TPU support (srush)
- XLM-R tokenizer is now tested and bug-free (LysandreJik)
- XLM-R model and tokenizer now have integration tests (patrickvonplaten)
- Better documentation for tokenizers and pipelines (LysandreJik)
- All tests (slow and non-slow) now pass (julien-c, LysandreJik, patrickvonplaten, sshleifer, thomwolf)
- Correct attention mask with GPT-2 when using past (patrickvonplaten)
- fix n_gpu count when no_cuda flag is activated in all examples (VictorSanh)
- Test TF GPT2 for correct behaviour regarding the past and attn mask variable (patrickvonplaten)
- Fixed bug where some missing keys would not be identified (LysandreJik)
- Correct `num_labels` initialization (LysandreJik)
- Model special tokens were added to the pretrained configurations (patrickvonplaten)
- QA models for XLNet, XLM and FlauBERT are now set to their "simple" architectures when using the pipeline.
- GPT-2 XL was added to TensorFlow (patrickvonplaten)
- NER PL example updated (shubhamagarwal92)
- Improved Error message when loading config/model with .from_pretrained() (patrickvonplaten, julien-c)
- Cleaner special token initialization in modeling_xxx.py (patrickvonplaten)
- Fixed the learning rate scheduler placement in the `run_ner.py` script erip
- Use AutoModels in examples (julien-c, lifefeel)

AutoTokenizer

AutoTokenizer has been put back to False by default so as to not have a breaking change between 2.4.x and 2.5.x

Fast tokenizers

Bug fixes

Slow tokenizers

Bug fixes related to `batch_encode_plus`