Beir

This is a small release update! We made the following changes in the release of the beir package:

1. Now train dense retrievers (SBERT) models using Margin-MSE loss
We have added a new loss, Margin-MSE, which learns a pairwise score using hard negatives and positives. Thanks to kwang2049 for the motivation, we have now added the loss to the beir repo. The loss is most effective with a Knowledge distillation setup using a powerful teacher model. For more details, we would suggest you refer to the paper by Hofstätter et al., https://arxiv.org/abs/2010.02666.

Margin-MSE Loss function: https://github.com/UKPLab/beir/blob/main/beir/losses/margin_mse_loss.py
Train (SOTA) SBERT model using Margin-MSE: https://github.com/UKPLab/beir/blob/main/examples/retrieval/training/train_msmarco_v3_margin_MSE.py

1. New script to utilize docT5query in parallel with multiple GPUs!
- Thanks to joshdevins, we have a new script to utilize multiple GPUs in parallel to generate multiple queries for passages using a question generation model faster. Check it out [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/sparse/evaluate_anserini_docT5query_parallel.py)].
- You can now pass your custom GPU device if CUDA recognizable devices are not present for question generation.

2. PQ Hashing with OPQ Rotation and Scalar Quantizer from Faiss!
- Now you can utilize ``OPQ`` rotation before using PQ hashing and Scalar Quantizer for ``fp16`` faiss search instead of original ``fp32``.

3. Top-k Accuracy Metric which is commonly used in the DPR repository by facebook!
- [DPR](https://github.com/facebookresearch/DPR) repository evaluates retrieval models using the ``top-k retriever accuracy``. This would allow evaluating top-k accuracy using the BEIR repository!
python
top_k_accuracy = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="top_k_accuracy")

4. Sorting of corpus documents by text length before encoding using a dense retriever!
- We now sort the corpus documents by longest size first, This has two advantages:
1. Why Sort? Similar lengths of texts are now encoded within a single batch, this would help speed up the corpus encoding process.
2. Why Sort longest to smallest? max GPU memory required can be found out in the beginning, so if OOM occurs it will occur in the beginning.

5. FEVER dataset training qrels, problems with doc-ids with special characters now fixed!
- There were issues with training qrels in the FEVER dataset. The doc-ids with special characters, for eg. ``Zlatan_Ibrahimović`` or ``Beyoncé`` had the wrong special characters present in ``qrels/train.tsv``. These were manually fixed by myself and now there are no more similar issues present in the corpus.
- New md5hash for the ``fever.zip`` dataset: ``5a818580227bfb4b35bb6fa46d9b6c03``.

FAISS Indexes and Search Integration

- FAISS Indexes can be created and used for evaluation using the BEIR repository. We have added support to ``Flat-IP``, ``HNSW``, ``PQ``, ``PCAMatrix``, and ``BinaryFlat`` Indexes.
- Faiss indexes use various compression algorithms useful for reducing Index memory sizes or improving retrieval speed.
- You can also save your corpus embeddings as a faiss index, which wasn't possible with the exact search originally.
- Check out how to evaluate dense retrieval using a faiss index [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_faiss_dense.py)] and dimension reduction using PCA [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_dim_reduction.py)].

Multilingual Datasets and Evaluation
- Thanks to julian-risch, we have added our first multilingual dataset to the BEIR repository - GermanQuAD (German SQuAD dataset).
- We have changed Elasticsearch now to allow evaluation on languages apart from English, check it out [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_multilingual_bm25.py)].
- We also have added a DPR model class which lets you load DPR models from Huggingface Repo, you can use this Class now for evaluation let's say the GermanDPR model [[link](https://huggingface.co/deepset/gbert-base-germandpr-ctx_encoder)].

DeepCT evaluation
- We have transformed the original DeepCT code to be able to use tensorflow (tf) >v2.0 and now hosted the latest repo [[here](https://github.com/NThakur20/DeepCT)].
- Using the hosted code, we are now able to use DeepCT for evaluation in BEIR using Anserini Retrieval, check [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/sparse/evaluate_deepct.py)].

Training Latest MSMARCO v3 Models
- From the [SentenceTransformers](https://github.com/UKPLab/sentence-transformers) repository, we have integrated the latest training code for MSMARCO on custom manually provided hard negatives. This provides the state-of-the-art SBERT models trained on MSMARCO, check [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/training/train_msmarco_v3.py)].

Using Multiple-GPU for question-generation
- A big challenge was to use multiple GPUs for the generation of questions much faster. We have included Process-pools to generate questions much faster and now using multiple GPUs also in parallel, check [[here](https://github.com/UKPLab/beir/blob/main/examples/generation/query_gen_multi_gpu.py)].

Integration of Binary Passage Retrievers (BPR)
- BPR (ACL'21, [link](https://arxiv.org/abs/2106.00882)) is now integrated within the BEIR benchmark. Now you can easily train a state-of-the-art BPR model on MSMARCO using the loss function described in the original paper, check [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/training/train_msmarco_v3_bpr.py)].
- You can also evaluate BPR now easily now in a zero-shot evaluation fashion, check [[here](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_bpr.py)].
- We would soon open-source the BPR public models trained on MSMARCO.