Mteb

Fix

* fix: Add model implementations and script for running the models (845)

* delete existing model results

* fix: Added models implementations and script for running the models

Note there is still a few things missing due to a *very* poor internet connection

* Added loader to metadata obj

* Update revision

* Add e5 models

* Update mteb/models/e5_models.py

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

* Minor update on print

* tested that everything works

* formatted

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`b331c34`](https://github.com/embeddings-benchmark/mteb/commit/b331c340e8f48e5530c9c71003b252769a55cacb))

Fix

* fix: formatted ([`8d3fc1b`](https://github.com/embeddings-benchmark/mteb/commit/8d3fc1b0c6efa8e6cd31db6ab14884b080fb53ab))

Unknown

* Add abstention metrics to retrieval and reranking tasks (854)

* add abstention as retrieval/reranking metric

* add reviewer points

* update docstrings

* Update docs/mmteb/points/854.jsonl

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`53c477f`](https://github.com/embeddings-benchmark/mteb/commit/53c477fd0604f5afbf114a50c54bfe259a071be8))

Fix

* fix: Add Russian tasks (RU-MTEB) (815)

* add ru-mteb tasks

* add results for new tasks

* downsample classifcation tasks & remove validation splits

* update clustering tasks to fit size limit

* remove mmarco dataset

* minor changes

* add points

* add list of tasks to benchmarks ([`e9d61bb`](https://github.com/embeddings-benchmark/mteb/commit/e9d61bba729c0c50f4baa0aec3280ed94b116a96))

Unknown

* Update tasks table ([`eebf973`](https://github.com/embeddings-benchmark/mteb/commit/eebf9735fd165bad71695bb354b153d0625968c7))

* Update points table ([`cebf1fc`](https://github.com/embeddings-benchmark/mteb/commit/cebf1fc12c46c8e1764eb1d1abfd84195591371c))

Fix

* fix: Use model revision in results folder (842)

* use model revision in results folder
* make lint
* tests (not dataset missing ones) passing
* load specified model revision
* check for model.revision first
* use no_revision_available
* make lint
* add revisions to test dir
* points
* make lint ([`2c6065b`](https://github.com/embeddings-benchmark/mteb/commit/2c6065b28e5212deecc6af973ca97d0c56d16264))

Unknown

* Update points table ([`c2e3d30`](https://github.com/embeddings-benchmark/mteb/commit/c2e3d30f098c729fa6ef063ca8ea83c7b8590bc4))

* update affiliation (855) ([`5fa2aee`](https://github.com/embeddings-benchmark/mteb/commit/5fa2aee8252ff83c0404e0a05e3b150832c30a67))

* Update tasks table ([`4bded5a`](https://github.com/embeddings-benchmark/mteb/commit/4bded5a653a806d9068cfb9f246c86d75c3e4c55))

* Update points table ([`f40c8a8`](https://github.com/embeddings-benchmark/mteb/commit/f40c8a861e493578368e92b912260958d0f9f4e1))

* Add Norwegian and Swedish to WikipediaRerankingMultilingual and update points (796)

* first proper upload of wikipedia-retrieval dataset

* update license and README of dataset

* fix test split and add WikipediaRetrievalDE task

* add WikipediaRerankingDE task

* add Bengali tasks

* multilingual reranking dataset

* add Multilingual Reranking

* add Retrieval tasks

* update metadata for Reranking task

* run make lint

* fix metadata validation errors

* delete German and Bengali Reranking tasks

* fix more task metadata, tests passing now

* add retrieval results

* WIP: reranking with multilingual dataset

* undo changes to run script

* update points and contributor info

* subcall MultilingualTask for reranking task and add reranking results

* WIP: make retrieval a multilingual dataset, too

* WIP: first run of WikipediaRetrievalMultilinugal

* add WikipediaRetrievalMultilinugal task and results

* delete language specific retrieval tasks and results

* update points and add Openreview IDs

* make lint

* remove debugging print statement

* add Norwegian and Swedish to ellamind/wikipedia-2023-11-reranking-multilingual dataset

* update reranking dataset revision and add new reranking results

* add missing multilingual-e5-small result on WikipediaRetrievalMultilingual

* update points for adding Norwegian and Swedish to WikipediaRerankingMultilingual

* make lint

* fix merge conflict in reranking results ([`06e4844`](https://github.com/embeddings-benchmark/mteb/commit/06e484441ab5f4765779862ccbdcba244569ffd2))

Fix

* fix: Find missing dataset revisions (844)

fix missing dataset revisions ([`c9b4c0c`](https://github.com/embeddings-benchmark/mteb/commit/c9b4c0c425c67f6219c38761f72ab50fed356880))

Fix

* fix: add model meta to create reproducible workflow (807)

* replace get_tasks as default filtering.

The intention here is to:

1) move complexity away from the MTEB object
2) ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
3) deprecate filtering in MTEB going forward (only with a warning atm.)
4) doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)

* add model meta to create reproducible workflow

- Add outline for model meta object
- Added a single model as a an example
- Added test for the reproducible workflow

The intention is that a reproducible workflow should then look like:


assuming the same mteb and sent. trf. version

model_meta = mteb.get_model(model_name)
task = mteb.get_task(task_name)

model = model_meta.load_model() load model either using custom loader or sentence transformer (with revision)

eval = MTEB(tasks=[task])
eval.run(model, output_folder=&34;tests/results&34;, overwrite_results=True)


For running models we can the simply have tasks like:

1) implement model
2) ensures that it runs on all tasks types

Running the models then become simple:


eval = MTEB(tasks=mteb.get_tasks())
for mdl_name in models:
model_meta = mteb.get_model(mdl_name)
mdl = model_meta.load_model()
eval.run(mteb.get_model(mdl)


We can start with this already now e.g. on classification tasks.

* import ISO_LANGUAGE from languages

* fix import

* Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>

* format

* Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>

* Updated based on suggestions from review

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`0319105`](https://github.com/embeddings-benchmark/mteb/commit/0319105734444de0626c068a3284832d96233dac))

* fix: Updated CLI to use new task filter (826)

* replace get_tasks as default filtering.

The intention here is to:

1) move complexity away from the MTEB object
2) ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
3) deprecate filtering in MTEB going forward (only with a warning atm.)
4) doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)

* tests passing

* Added corrections from review

* Updated CLI

* docs: Added points

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`fb5fec8`](https://github.com/embeddings-benchmark/mteb/commit/fb5fec8b763c107fbcc9bdc853a64d6d8a8d0043))

Unknown

* Update points table ([`f926216`](https://github.com/embeddings-benchmark/mteb/commit/f926216f8427ee514196d200caa089a16a22db48))

* Update tasks table ([`d560c31`](https://github.com/embeddings-benchmark/mteb/commit/d560c31d3d10d6e13cf41d4f3aabff9ef4d37cec))

* Update points table ([`84e6856`](https://github.com/embeddings-benchmark/mteb/commit/84e6856cfb9f251c102a77394fe26eaaa1c01624))

* Add MLQuestions dataset (799)

* mlquestions load script

* more metadata

* add to init

* baseline model results

* add points

* complete metadata

* lint

* Update points and metadata

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* clarification of period in comments

* minor fix

* linting

* Fix validation error

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`3a14885`](https://github.com/embeddings-benchmark/mteb/commit/3a14885b8ea0406f9ae0edf7b550bfadb37fcb4e))