Mteb

Fix

* fix: Use model revision in results folder (842)

* use model revision in results folder
* make lint
* tests (not dataset missing ones) passing
* load specified model revision
* check for model.revision first
* use no_revision_available
* make lint
* add revisions to test dir
* points
* make lint ([`2c6065b`](https://github.com/embeddings-benchmark/mteb/commit/2c6065b28e5212deecc6af973ca97d0c56d16264))

Unknown

* Update points table ([`c2e3d30`](https://github.com/embeddings-benchmark/mteb/commit/c2e3d30f098c729fa6ef063ca8ea83c7b8590bc4))

* update affiliation (855) ([`5fa2aee`](https://github.com/embeddings-benchmark/mteb/commit/5fa2aee8252ff83c0404e0a05e3b150832c30a67))

* Update tasks table ([`4bded5a`](https://github.com/embeddings-benchmark/mteb/commit/4bded5a653a806d9068cfb9f246c86d75c3e4c55))

* Update points table ([`f40c8a8`](https://github.com/embeddings-benchmark/mteb/commit/f40c8a861e493578368e92b912260958d0f9f4e1))

* Add Norwegian and Swedish to WikipediaRerankingMultilingual and update points (796)

* first proper upload of wikipedia-retrieval dataset

* update license and README of dataset

* fix test split and add WikipediaRetrievalDE task

* add WikipediaRerankingDE task

* add Bengali tasks

* multilingual reranking dataset

* add Multilingual Reranking

* add Retrieval tasks

* update metadata for Reranking task

* run make lint

* fix metadata validation errors

* delete German and Bengali Reranking tasks

* fix more task metadata, tests passing now

* add retrieval results

* WIP: reranking with multilingual dataset

* undo changes to run script

* update points and contributor info

* subcall MultilingualTask for reranking task and add reranking results

* WIP: make retrieval a multilingual dataset, too

* WIP: first run of WikipediaRetrievalMultilinugal

* add WikipediaRetrievalMultilinugal task and results

* delete language specific retrieval tasks and results

* update points and add Openreview IDs

* make lint

* remove debugging print statement

* add Norwegian and Swedish to ellamind/wikipedia-2023-11-reranking-multilingual dataset

* update reranking dataset revision and add new reranking results

* add missing multilingual-e5-small result on WikipediaRetrievalMultilingual

* update points for adding Norwegian and Swedish to WikipediaRerankingMultilingual

* make lint

* fix merge conflict in reranking results ([`06e4844`](https://github.com/embeddings-benchmark/mteb/commit/06e484441ab5f4765779862ccbdcba244569ffd2))

Fix

* fix: Find missing dataset revisions (844)

fix missing dataset revisions ([`c9b4c0c`](https://github.com/embeddings-benchmark/mteb/commit/c9b4c0c425c67f6219c38761f72ab50fed356880))

Fix

* fix: add model meta to create reproducible workflow (807)

* replace get_tasks as default filtering.

The intention here is to:

1) move complexity away from the MTEB object
2) ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
3) deprecate filtering in MTEB going forward (only with a warning atm.)
4) doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)

* add model meta to create reproducible workflow

- Add outline for model meta object
- Added a single model as a an example
- Added test for the reproducible workflow

The intention is that a reproducible workflow should then look like:


assuming the same mteb and sent. trf. version

model_meta = mteb.get_model(model_name)
task = mteb.get_task(task_name)

model = model_meta.load_model() load model either using custom loader or sentence transformer (with revision)

eval = MTEB(tasks=[task])
eval.run(model, output_folder=&34;tests/results&34;, overwrite_results=True)


For running models we can the simply have tasks like:

1) implement model
2) ensures that it runs on all tasks types

Running the models then become simple:


eval = MTEB(tasks=mteb.get_tasks())
for mdl_name in models:
model_meta = mteb.get_model(mdl_name)
mdl = model_meta.load_model()
eval.run(mteb.get_model(mdl)


We can start with this already now e.g. on classification tasks.

* import ISO_LANGUAGE from languages

* fix import

* Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>

* format

* Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>

* Updated based on suggestions from review

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`0319105`](https://github.com/embeddings-benchmark/mteb/commit/0319105734444de0626c068a3284832d96233dac))

* fix: Updated CLI to use new task filter (826)

* replace get_tasks as default filtering.

The intention here is to:

1) move complexity away from the MTEB object
2) ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
3) deprecate filtering in MTEB going forward (only with a warning atm.)
4) doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)

* tests passing

* Added corrections from review

* Updated CLI

* docs: Added points

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`fb5fec8`](https://github.com/embeddings-benchmark/mteb/commit/fb5fec8b763c107fbcc9bdc853a64d6d8a8d0043))

Unknown

* Update points table ([`f926216`](https://github.com/embeddings-benchmark/mteb/commit/f926216f8427ee514196d200caa089a16a22db48))

* Update tasks table ([`d560c31`](https://github.com/embeddings-benchmark/mteb/commit/d560c31d3d10d6e13cf41d4f3aabff9ef4d37cec))

* Update points table ([`84e6856`](https://github.com/embeddings-benchmark/mteb/commit/84e6856cfb9f251c102a77394fe26eaaa1c01624))

* Add MLQuestions dataset (799)

* mlquestions load script

* more metadata

* add to init

* baseline model results

* add points

* complete metadata

* lint

* Update points and metadata

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* clarification of period in comments

* minor fix

* linting

* Fix validation error

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`3a14885`](https://github.com/embeddings-benchmark/mteb/commit/3a14885b8ea0406f9ae0edf7b550bfadb37fcb4e))

Fix

* fix: Convert blurbs to fast for s2s and p2p (832)

* add BlurbsClusteringS2S.v2

* add BlurbsClusteringP2P.v

* make lint

* points

* update n_samples ([`a00fdba`](https://github.com/embeddings-benchmark/mteb/commit/a00fdba44d1fadd0be449b7f988dd08145cd5b87))

Unknown

* Update tasks table ([`04d5494`](https://github.com/embeddings-benchmark/mteb/commit/04d549462482f96066b4a10f828094f3da076d92))

* Update points table ([`4b31692`](https://github.com/embeddings-benchmark/mteb/commit/4b316922499d927672a518f1d44dbf8196ea547e))

Fix

* fix: (1) Add `StatcanDialogueDatasetRetrieval` (2) Fix `DRESModel.encode_conversations` to allow list of dictionaries (779)

* WIP

* Update metadata based on reviewer requested changes

* Finalize metadata

* add statcandialoguedatasetretrival to mteb.tasks.retrieval

* Convert query from JSON string to dictionary (parsed using `json`)

* Fix: DRESModel to allow conversations composed of list of dictionaries, alongside lists of strings (e.g. Topicoqa). Also fix batch_size parameter in encode_conversations

* Add baseline results

* Change revision to hash

* Add points

* Fix incorrect object reference ([`7943ff0`](https://github.com/embeddings-benchmark/mteb/commit/7943ff05d4ec4d4c4da81a5a6c50fd298de907dd))

Unknown

* Update tasks table ([`04f6cdc`](https://github.com/embeddings-benchmark/mteb/commit/04f6cdc8022a1514a55ce20fb38cd11431c11842))

* Update points table ([`3a96c69`](https://github.com/embeddings-benchmark/mteb/commit/3a96c6955e29c0a7a9d7e61e25f89fdd15269d7c))

* Update points table ([`6563218`](https://github.com/embeddings-benchmark/mteb/commit/656321807a3bf132753920d6f076266925613d6f))

* Add fast clustering version to existing files (827)

* Add fast clustering to existing file

* add points ([`a0ebf87`](https://github.com/embeddings-benchmark/mteb/commit/a0ebf87da18c6e94b27b855f136dc3a13a0d7a5d))

Fix

* fix: rename *fast to *v2 for clustering task (825)

* rename *fast to *v2 for clustering task

* added correction from review ([`365c0d4`](https://github.com/embeddings-benchmark/mteb/commit/365c0d4c67162c9d3a50fcd5c38d3dae37a96eac))

Unknown

* Update tasks table ([`e3641ea`](https://github.com/embeddings-benchmark/mteb/commit/e3641ea91c677c44851f2e7fe2680dc495fbc54f))