Mteb

Documentation

* docs: Update figure (1027)

* Updated figure colors

* Updated figure with current numbers

* Added reasoning as retrieval to figure ([`0acf259`](https://github.com/embeddings-benchmark/mteb/commit/0acf2590b9b6059fc5dbe66c31b1fc5f313efdd6))

* docs: Added result tables (996)

* Added utility function for downloading models results

* Added result tables ([`f809cad`](https://github.com/embeddings-benchmark/mteb/commit/f809cadcbaa259f8bbfc5d3fe98de217023883f1))

* docs: Add Mariya&39;s points (1016)

* Create 1004.jsonl (1011)

add points for working on introduction

* Create 784.jsonl (1012)

add points for working on Appendix B

* rename files

* fix: Fixed subset warning for later versions of polars

---------

Co-authored-by: Mariya Hendriksen <35101262+mariyahendriksenusers.noreply.github.com> ([`4e1f43b`](https://github.com/embeddings-benchmark/mteb/commit/4e1f43b1d56843fe36637068d64e5a09e63964fd))

* docs: Update points.md (1010)

Added an author, fixed alignment issues and some typos
Co-authored-by: dokato <raymon92gmail.com> ([`1d1f7aa`](https://github.com/embeddings-benchmark/mteb/commit/1d1f7aadc71b89444837ea59bd142107789f648a))

Fix

* fix: Fix broken clustering tasks (1026)

* fix: Added imenes fix

* fix: found and fixes two other mistakes for MLSUMClustering (one caused by python 3.8 and the other in the downsampling function) ([`ca9e81d`](https://github.com/embeddings-benchmark/mteb/commit/ca9e81de8e950eef823e31e6799fe9e3fa5767dd))

Unknown

* LLM2Vec multi-gpu bug fix (1020)

* map device to device_map

* lint ([`3a64ed9`](https://github.com/embeddings-benchmark/mteb/commit/3a64ed9ffe87a8673b6c5ebe05bb0c095a2d1698))

* Update points table ([`b3a58d2`](https://github.com/embeddings-benchmark/mteb/commit/b3a58d2ab40cfee130aba2784ba9a067a929ad4f))

* add SFR embed model (1002)

* add sfr model
* add instruction to encode ([`1e25fe0`](https://github.com/embeddings-benchmark/mteb/commit/1e25fe04cd024b3032c80c1479f07f661a788d3f))

Fix

* fix: Add more russian models (1000)

* add models meta

* add models

* fix models versions

* fix models versions

* add models

* add points

* lint

* add to points table ([`d97310d`](https://github.com/embeddings-benchmark/mteb/commit/d97310d6bc090d4e7bb0fde93e52b9241044bc7e))

Unknown

* Update points table ([`17262fe`](https://github.com/embeddings-benchmark/mteb/commit/17262fe895c3f1cc3b6119c42754294d2a3a1851))

Fix

* fix: Added utility function for downloading and loading models results (995)

Added utility function for downloading models results ([`abef8dc`](https://github.com/embeddings-benchmark/mteb/commit/abef8dc2366d56c790a99ed205b97fc81d0894c2))

Unknown

* Update points table ([`17308b3`](https://github.com/embeddings-benchmark/mteb/commit/17308b3d0634a2baae373586f30f34f83bbee7ec))

* [WIP] RAR-b second PR & task instruction for IR tasks in general (972)

* Reval with instruction ARC-C test

* fix formatting issue

* add default instructions to lookup

* add points for both PRs

* add points for both PRs & info ([`4500e0d`](https://github.com/embeddings-benchmark/mteb/commit/4500e0d1157f1bc291dade1f558480fe620c5b9c))

* Update tasks table ([`be5bf69`](https://github.com/embeddings-benchmark/mteb/commit/be5bf694479ed26d30ba554363c80ad4033fb84c))

Fix

* fix: Update DanFever to use avoid using overwrite=True and ignore identical ids (994)

* fix: Updated DanFever to use avoid using overwrite=True and ignore identical ids

* fix: Updated logging, adding utility helper when specifying the wrong task and added --overwrite flag to CLI

* fix: revert changes to danFEVER

* fixed spelling of superseeded_by > superseded_by ([`56ba2d7`](https://github.com/embeddings-benchmark/mteb/commit/56ba2d7657976180ef95d83eec9743de851c5542))

Unknown

* Update points table ([`976d921`](https://github.com/embeddings-benchmark/mteb/commit/976d921a0600cbe4ef42ba357ca4f3d39e491221))

* Fix identical query-corpus ids in retrieval tasks (918)

* Fix bug: identical query-corpus ids

* Add results

* Make device argument optional in Encoder

* add points ([`7c50e59`](https://github.com/embeddings-benchmark/mteb/commit/7c50e59caf055bcd3cb064ddd9476a10fd2c3b08))

Fix

* fix: convert scores in meta to be in 0-100 instead of 0-1 (993)

fix: convert scores in meta to be expressed in 0-100 instead of 0-1 ([`870a2d9`](https://github.com/embeddings-benchmark/mteb/commit/870a2d9417127dfacc18f42943af46a97c2b90e0))

Unknown

* Update tasks table ([`694bbd9`](https://github.com/embeddings-benchmark/mteb/commit/694bbd9b3f2191c8e97d7ae66031808f180dd507))

* Update numbers in retrieval statistics (988)

* example

* fix validation error

* add retrieval task info

* lint

* remove task_name key

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`10c3fbf`](https://github.com/embeddings-benchmark/mteb/commit/10c3fbf25590e97f0c26f6068d261a222d59ad33))

Fix

* fix: Add Nomic and Multilingual Cohere (987)

* Add nomic and cohere models

* format

* Ensure that models run on a sample task and reproduce results on SEB

* Added normalization to encode

* fix naming variable

* Update mteb/evaluation/evaluators/model_encode.py

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`e398f20`](https://github.com/embeddings-benchmark/mteb/commit/e398f2012c189da1a88d18e9b9bc0e87882c675d))

Unknown

* Add BGE large en v1.5 (986)

* add models
* make lint
* correct naming
* does not need as Jina works out of the box with ST
* pop prompt_name
* add normalize_embeddings=True
* make lint ([`d5e8bd1`](https://github.com/embeddings-benchmark/mteb/commit/d5e8bd114f8e76da6cd0eefa8ec43ee1b0218978))