Mteb

Documentation

* docs: Add contribution (688)

* add MIRACLFrReranking dataset

* remove old fr file

* add contribution

* fix missing french data

* update to individual contributor

---------

Co-authored-by: Shreeya Dhakal <shreeyadhakalShreeyas-Mac-mini.local>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`e3d230a`](https://github.com/embeddings-benchmark/mteb/commit/e3d230a5a4da54d85314d40c0643ffe0c0ac02de))

* docs: Update points.md (687) ([`85c7858`](https://github.com/embeddings-benchmark/mteb/commit/85c7858ceb47acbf943bb8b96a6c845fb0affe7b))

* docs: Adding contributor information (683)

* chore: adding myself as a contributor

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* chore: add openreview username

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

---------

Signed-off-by: jupyterjazz <saba.sturuajina.ai> ([`93a1248`](https://github.com/embeddings-benchmark/mteb/commit/93a1248559d23efb49d572d28ae92a91a81511ae))

* docs: Update points system (666)

* Update readme.md

* Update docs/mmteb/readme.md

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>

* Added upper bound on points along with validator

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`d731c98`](https://github.com/embeddings-benchmark/mteb/commit/d731c98ed38ae1299acd27495adfade37c068dd3))

Fix

* fix: Double assignemnt in RomanianReviewsSentiment (692)

* Fix a double assignment in the RomanianReviewsSentiment class.

Signed-off-by: mr.Shu <mrshu.io>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`b4155b5`](https://github.com/embeddings-benchmark/mteb/commit/b4155b555ddf4f77ecab45775f6b3c0289bd4bb9))

Unknown

* Update tasks table ([`05f94b1`](https://github.com/embeddings-benchmark/mteb/commit/05f94b127fbabb138523f593ed52df231de19ff8))

* Update points table ([`34d666d`](https://github.com/embeddings-benchmark/mteb/commit/34d666db71720c29e4610e65f53e01d9ea635071))

* Adding Klue-NLI dataset (675)

* Adding Klue-NLI dataset
* adding points ([`57c3579`](https://github.com/embeddings-benchmark/mteb/commit/57c35792d83f398ac4f2ac712ce996b82ac0b89f))

* Update tasks table ([`c63d408`](https://github.com/embeddings-benchmark/mteb/commit/c63d408037c82b4f9c994e758ea037b14ef3a7be))

* Update points table ([`f01ad7d`](https://github.com/embeddings-benchmark/mteb/commit/f01ad7d88e616a53f3c9be686c60d3da29f4eef0))

* Add classification datasets (eng, ron, swe) (673)

* add swedish reviews

* add poem sentiment

* add romanian reviews

* updates

* add points

* update date

* update date

* update points

---------

Co-authored-by: Márton Kardos <power.up1163gmail.com> ([`01caff0`](https://github.com/embeddings-benchmark/mteb/commit/01caff0edd67b269fdff36f095d09456b33025b5))

* Update points table ([`681b530`](https://github.com/embeddings-benchmark/mteb/commit/681b53019b473704e41dfa1e9875f8ee7929efc2))

* Merge branch &39;main&39; of https://github.com/embeddings-benchmark/mteb ([`779c5bc`](https://github.com/embeddings-benchmark/mteb/commit/779c5bcd6a02f688404260337c1088a095e37d56))

* Update tasks table ([`8e7173e`](https://github.com/embeddings-benchmark/mteb/commit/8e7173e44e2bbf3e674d6f0b462825271afc3245))

* Update points table ([`7090005`](https://github.com/embeddings-benchmark/mteb/commit/709000511a5b9c3a26af171da3f95cf5eb291767))

* Hierarchical clustering (624)

* Added Hierarchical clustering abstask (naive implementation)

* Added SNL as a hierarchical clustering task

* Added Hierarchical clustering as valid task type

* Added results on SNL hierarchical

* fix: Ran linting and fixed metadata

* Turned fast clustering to hierarchical

* Merged hierarchical clustering into fast clustering

* fix: Fixed indentation in clustering fast

* SNLHierarchicalCLustering now superseeds SNL

* fix: Clustering model is now initiated at each level

* Added results for hierarchical clustering with bootstrapping

* Ran linting

* Removed HierarchicalClustering from task types

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* SNL task type is not Clustering

* Updated docstring for fast clustering

* Ran linting

* Made max_depth None by default

* Set max depth to 5 on SNL clustering

* Ran linting

* Fix: clustering now wraps labels that are not hierarchical in lists

* Reran SwednClustering

* Reran linting

* Added points

* fix: Corrected number of samples and mean length in SNLHierarchicalClustering

* Split SNL clustering to S2S and P2P tasks

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`74a19a7`](https://github.com/embeddings-benchmark/mteb/commit/74a19a7a534865d89bc830afccc616f376327d29))

* Update tasks table ([`2e2979e`](https://github.com/embeddings-benchmark/mteb/commit/2e2979ee357267fec110cbf0d53f3a7d2577c444))

* Update points table ([`7ff5b0d`](https://github.com/embeddings-benchmark/mteb/commit/7ff5b0d224f92103cadb836ab877d03ab4e1c43c))

* Add Belebele Retrieval (636)

* feat: belebele retrieval

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* feat: support langs

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* docs: adjust description

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* chore: results

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* refactor: change num_samples and remove answers

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* chore: update results

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* refactor: update avg length

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* chore: points

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* refactor: apply suggestions

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* fix: jsonl

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* style: linting

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

* chore: points

Signed-off-by: jupyterjazz <saba.sturuajina.ai>

---------

Signed-off-by: jupyterjazz <saba.sturuajina.ai>
Co-authored-by: Isaac Chung <chungisaac1217gmail.com>
Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com> ([`4a2b9db`](https://github.com/embeddings-benchmark/mteb/commit/4a2b9db43987f26df77c940f25e980bf459144c4))

Documentation

* docs: remove prompt kwargs for example (681) ([`df490cf`](https://github.com/embeddings-benchmark/mteb/commit/df490cfc16164c2b63e46c8cfe25b01a84e1153d))

Feature

* feat: Standardize MTEB results (658)

* remove misplaced file

* Added MTEBResults and langscript filter

* Ensure standard format for classification

* format

* fixed failing tests and added central task registry

* Added changes from review

* fix: reformatted according to wishes in PR

* fix: Refactored out get_main_score

* Added points

* format ([`7166c31`](https://github.com/embeddings-benchmark/mteb/commit/7166c317c1748b4b11772fe59b8410d5f53aa0f0))

Fix

* fix: Two Korean classification datasets added (670)

* two korean classification datasets added

* kor init points added

* Update docs/mmteb/points/670.jsonl

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Kor datasets use self.stratified_subsampling

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`7c2299c`](https://github.com/embeddings-benchmark/mteb/commit/7c2299c9d0b41fb00978b97807bcbb4d946ef105))

Unknown

* Update points table ([`bc7ed5e`](https://github.com/embeddings-benchmark/mteb/commit/bc7ed5e32e51614f7b6184e889075e154a4aae5c))

* Update points table ([`fd241ba`](https://github.com/embeddings-benchmark/mteb/commit/fd241baa360259c21a9e184649b6e5a03803159e))

* Update tasks table ([`30c94a7`](https://github.com/embeddings-benchmark/mteb/commit/30c94a7c80273d2f8d5297efd0ab82383f8d405d))

* Update points table ([`c9ea24b`](https://github.com/embeddings-benchmark/mteb/commit/c9ea24b03cb2086a47ecebb992de922b683187fc))

* Adding the RTE3 dataset (672)

* Adding the RTE3 dataset

* fixing metadata

* adding points ([`eea0537`](https://github.com/embeddings-benchmark/mteb/commit/eea0537a50de7fcba504dd1772e28f6ca354b8f9))

Fix

* fix: Add cyrillic turkic lang classification (659)

* add MIRACLFrReranking dataset

* remove old fr file

* Add Cyrillic Turkic Lang Classification

* update metadata and add scores

* Update mteb/tasks/Classification/multilingual/CyrillicTurkicLangClassification.py

---------

Co-authored-by: Shreeya Dhakal <shreeyadhakalShreeyas-Mac-mini.local>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`11b4888`](https://github.com/embeddings-benchmark/mteb/commit/11b4888aba0709b071512a5a042ad1d1043c06d1))

Unknown

* Update tasks table ([`8132af9`](https://github.com/embeddings-benchmark/mteb/commit/8132af985ac4c2b1cd9cd879e5d33b9ef331f0d4))

* Update points table ([`6402ceb`](https://github.com/embeddings-benchmark/mteb/commit/6402ceb7f82753597285be88f3e65e4bbc815c00))

Fix

* fix: ensure that task.languages return a list of languages (671)

* fix: ensure that task.languages return a list of languages

* docs: added points

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`2a8f8f5`](https://github.com/embeddings-benchmark/mteb/commit/2a8f8f58b14f071df9c05fe6b861d76e86e899c5))

Unknown

* Update points table ([`4dd940f`](https://github.com/embeddings-benchmark/mteb/commit/4dd940f295a4999b285113ff027ba3d353e1fc05))

* Update tasks table ([`071ea70`](https://github.com/embeddings-benchmark/mteb/commit/071ea70b548c7fdced60c50463c2a115d8b62ce7))

* Update points table ([`78299fb`](https://github.com/embeddings-benchmark/mteb/commit/78299fb8cc28e5a762c59fd72928d351e9c7e2bf))

* added nusax-senti dataset (662)

* added nusax-senti dataset

* evaluated NusaX-Senti

* added points

* made changes in NusaX-senti based on comments

* linted correctly

---------

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com> ([`59092c3`](https://github.com/embeddings-benchmark/mteb/commit/59092c33e46f884acd0feee781a096020ead23e1))

Fix

* fix: Add Marathi news classification (504)

* first commit for Telugu News Classification

* revert to original main

* add first push

* add dataset

* add results and points

* complete adding points ([`d93488a`](https://github.com/embeddings-benchmark/mteb/commit/d93488a6fe461eb00e3deb47397908af9a467805))

Unknown

* Update tasks table ([`600dbd0`](https://github.com/embeddings-benchmark/mteb/commit/600dbd0026172eb6bccbb5210baf16953b753f5c))

* Update points table ([`73422b7`](https://github.com/embeddings-benchmark/mteb/commit/73422b7118bb2de4e2011eb364ac7b9064bab25b))

* Update tasks table ([`dc73123`](https://github.com/embeddings-benchmark/mteb/commit/dc731232a876a4588072b92cf1160266937281ce))

* Update points table ([`54192c7`](https://github.com/embeddings-benchmark/mteb/commit/54192c7299ff33193eb5a608d1aeef878ae00f7a))

* Multilabel classification (440)

* Added Multilabel kNN classification evaluator

* Added Multilabel classification AbsTask

* Added MultiLabelClassification Task type to TaskMetadata

* bugfix

* Removed all references to metadata_dict from Multilabel classification

* Added Eurlex (wip)

* Made MultiLabelClassification more efficient by moving the embedding step outside the evaluator and encoding every possible training sentence before running the evaluation.

* fix: changed itertools.chain to itertools.chain.from_iter

* fix: Fixed validation and import on MultiEURLEX

* Removed MultioutputClassifier, because kNN can already do that

* fix: multilabels are not turned into an array

* Ran linting

* Added points for PR (2+23*4 for eurlex, 10 for new task type)

* fix: Fixed undersampling for training set in Multitask classification

* fix: sped up sampling by using select() instead of indexing

* fix: removed duplicate code for selecting train sentences

* Added n_samples and avg_length to MultiEURLEX

* Added MultiEURLEX results for paraphrase-multilingual-MiniLM-L12

* Added EURLEX results for multilingual-e5-small

* Changed evaluation in multilabel classification to use MLPClassifier

* Limited evaluation to test split in EURLEX

* multilabel classification now subsamples test set, and the neural network is smaller.

* Multilabel classification now allows tasks to define the samples per label for training

* Removed unused code

* Moved subsampling to before encoding

* Made subsampling error tolerant

* Made sure all labels are represented in the training set

* Revert &34;Made sure all labels are represented in the training set&34;

This reverts commit 96312c7ca55b2870995c9b69ab4b88eeaf92fe79.

* Reran EURLEX

* EURLEX only evaluates on test set, not validation set

* Made KNeighbours the default classifier in MultiLabelClassification, made switching out classifiers more flexible

* Added results for EURLEX ([`2aa0c67`](https://github.com/embeddings-benchmark/mteb/commit/2aa0c67b05acd9dadb9b1731f8a8bb28de58702f))

Fix

* fix: mmteb | Arabic Retrieval Task | SadeemQuestionRetrieval (643)

* create a new directory for Arabic Retrieval tasks

* Push SadeemQuestionRetrieval dataset

* Push SadeemQuestionRetrieval baseline results

* remove invalid comments

* update SadeemQuestionRetrieval metadata

* update SadeemQuestionRetrieval metadata

* add points to the PR

* update points

* apply lint ([`f2d6c1a`](https://github.com/embeddings-benchmark/mteb/commit/f2d6c1a6cb6cb13edffad6a7735148da806e972d))

Unknown

* Update points table ([`1198ba1`](https://github.com/embeddings-benchmark/mteb/commit/1198ba1963709e1763d1cdd5db58f5c5689ca089))

* Update points table ([`a3082f4`](https://github.com/embeddings-benchmark/mteb/commit/a3082f4ea519c954d824e82de2a7aa04e6dc1dba))

* Small typo fix from 649 (668)

Fixed naming 649.json to 649.jsonl ([`f58b7f5`](https://github.com/embeddings-benchmark/mteb/commit/f58b7f5614980b6822ac39de19337562d14ae722))

* Update tasks table ([`3765683`](https://github.com/embeddings-benchmark/mteb/commit/376568330d79367f1330c3faf7ba3f6533cbd41f))

* Added XNLI V2.0 Greek, Turkish, Russian support (649)

* Added xnli_greek

* Added also XNLI Turkish and XNLI Russian whilst at it

* Added description and fixed text creation method of previous commit

* updated descriptions

* Switched to the multilingual setup, removed previous independent files. as requested

* changed to abstart task, MTEB dataset

* Added points

* small fix

* fixed points

* Apply suggestions from code review

used original xnli2.0-multi-pair

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com>

---------

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com>
Co-authored-by: Orion Weller <31665361+orionwusers.noreply.github.com> ([`22eae1b`](https://github.com/embeddings-benchmark/mteb/commit/22eae1bb756cdbd1ec17b01ad59629829c5d36b1))

* Delete mteb/results directory (664) ([`8296c17`](https://github.com/embeddings-benchmark/mteb/commit/8296c17071c82f7f9713ad61ddacd9bfdd27faf3))

* Update tasks table ([`0cf33d7`](https://github.com/embeddings-benchmark/mteb/commit/0cf33d73b1f3ff5be1d3689f2aa8abbbe4454c99))

* Update points table ([`5a8a78d`](https://github.com/embeddings-benchmark/mteb/commit/5a8a78d56871d9aa6ddfaf168bd2fcae65580829))

* fix: Add LegalBench datasets - 8 (648)

* Add LegalBench datasets

* Add points

* Update docs/mmteb/points/648.jsonl

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Reformulate Diversity datasets; Update description for Function of Decision Section dataset

* Fix linting

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`10a0354`](https://github.com/embeddings-benchmark/mteb/commit/10a03544b9cd7b6839bc3c725dcddea27f2cdccf))