Mteb

Fix

* fix: E5 instruct now listed as sbert compatible (2475)

Fixes 1442 ([`6c8c8d2`](https://github.com/embeddings-benchmark/mteb/commit/6c8c8d240d710cdf662d0b8d095d85966affaeb5))

Unknown

* suppress logging warnings on leaderboard (2406)

* supress logging warnings

* remove loggers

* return blocks

* rename function

* fix gme models

* add server name

* update after merge

* fix ruff ([`e837b09`](https://github.com/embeddings-benchmark/mteb/commit/e837b093e256a105ba13aa77bd0706ba364a10c7))

Ci

* ci: cache `~/.cache/huggingface` (2464)

ci: cache ~/.cache/huggingface

Co-authored-by: sam021313 <40773225+sam021313users.noreply.github.com> ([`d11934f`](https://github.com/embeddings-benchmark/mteb/commit/d11934fd03655011527045afca02adacac0a8d0d))

Fix

* fix: add nb_sbert model (2339)

* add_nb_sbert_model

* Update nb_sbert.py

added n_parameters and release_date

* Update mteb/models/nb_sbert.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* Update nb_sbert.py

fix: make lint

* added nb_sbert to overview.py + ran make lint

* Update nb_sbert.py

Fix error: Input should be a valid date or datetime, month value is outside expected range of 1-12

---------

Co-authored-by: Roman Solomatin <samoed.romangmail.com> ([`c617598`](https://github.com/embeddings-benchmark/mteb/commit/c61759807fa8b32d5969598c127139ff38f96062))

* fix: Adds family of NeuML/pubmedbert-base-embedding models (2443)

* feat: added pubmedbert model2vec models

* fix: attribute model_name

* fix: fixed commit hash for pubmed_bert model2vec models

* fix: changes requested in PR 2443 ([`f293d8b`](https://github.com/embeddings-benchmark/mteb/commit/f293d8bf65470b0e431c39f18082fe3845fe233e))

Unknown

* Update tasks table ([`5b567bf`](https://github.com/embeddings-benchmark/mteb/commit/5b567bf9d39e5ebffb2d947849b1eee3a80a446a))

* [MIEB] &34;capability measured&34;-Abstask 1-1 matching refactor [3/3]: reimplement ImageCoDe (2468)

* reimplement ImageCoDe with ImageTextPairClassification

* add missing stats file ([`8799126`](https://github.com/embeddings-benchmark/mteb/commit/8799126c8e2f294326e1840832159aea980327ef))

* leaderboard fix (2456) ([`35a8a5b`](https://github.com/embeddings-benchmark/mteb/commit/35a8a5b3ae2606bc3a64ab50319a877f497f888d))

* add ops_moa_models (2439)

* add ops_moa_models

* add custom implementations

* Simplify custom implementation and format the code

* support SentenceTransformers

* add training datasets

* Update mteb/models/ops_moa_models.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* update training_datasets

---------

Co-authored-by: kunka.xgw <kunka.xgwtaobao.com>
Co-authored-by: Roman Solomatin <samoed.romangmail.com> ([`61d3c6c`](https://github.com/embeddings-benchmark/mteb/commit/61d3c6c29ddafb633c1ab112d88a8a4c4ce893f8))

* Add Background Gradients in Summary and Task Table (2392)

* Add Background Gradients in Summary and Task Table

* Remove warnings and add light green cmap

* Address comments

* Separate styling function

* address comments

* added comments ([`5af5547`](https://github.com/embeddings-benchmark/mteb/commit/5af55479ea10f27ee3e10a4a5be5bbaec49f60e7))

* Feat/searchmap preview (2420)

* Added meta information about SearchMap_Preview model to the model_dir

* Added meta information about SearchMap_Preview model to the model_dir

* updated revision name

* Device loading and cuda cache cleaning step left out

* removed task instructions since it&39;s not necessary

* changed sentence transformer loader to mteb default loader and passed instructions s model prompts

* Included searchmap to the models overview page

* Included searchmap to the models overview page

* added meta data information about where model was adpated from

* Update mteb/models/searchmap_models.py

* fix lint

* lint

---------

Co-authored-by: Roman Solomatin <samoed.romangmail.com>
Co-authored-by: Roman Solomatin <36135455+Samoedusers.noreply.github.com> ([`43adb0c`](https://github.com/embeddings-benchmark/mteb/commit/43adb0cd2a0cc611edc85c11df120629a7963de1))

* Error while evaluating MIRACLRetrievalHardNegatives: &39;trust_remote_code&39; (2445)

Fixes 2444 ([`dadafbe`](https://github.com/embeddings-benchmark/mteb/commit/dadafbe50a214ef099b15550746f61927323093d))

* Update tasks table ([`19dc625`](https://github.com/embeddings-benchmark/mteb/commit/19dc625583a0206ae68779830cebb7390158bfcb))

* [MIEB] &34;capability measured&34;-Abstask 1-1 matching refactor [2/3]: reimplement r-Oxford and r-Paris (2442)

* MutipleChoiceEvaluationMixin; reimplement r-Oxford and r-Paris; rerun stats

* modify benchmark list

* fix citation ([`65446e5`](https://github.com/embeddings-benchmark/mteb/commit/65446e5d9b9580021ae52bdb25480ee1438191dc))

Documentation

* docs: typos; Standardize spacing; Chronological order (2436)

* Fix typos; add chrono order

* Fix spacing ([`0db0a20`](https://github.com/embeddings-benchmark/mteb/commit/0db0a20d48d422d3f3db25d50b73ada2525ccc19))

Fix

* fix: Add model specific dependencies in pyproject.toml (2424)

* Add model specific dependencies in pyproject.toml

* Update documentation ([`8a024be`](https://github.com/embeddings-benchmark/mteb/commit/8a024be2e69a4f42c8cc281a5af21e08d69ff0ff))

Unknown

* Added Memory Usage column on leaderboard (2428) ([`d3eab6f`](https://github.com/embeddings-benchmark/mteb/commit/d3eab6f1f483e4e5b9e1e871289397c8e114f713))

* add richinfoai models (2427)

* add richinfoai models

add richinfoai models

* format codes by linter

format codes by linter ([`98ab0ef`](https://github.com/embeddings-benchmark/mteb/commit/98ab0ef357a1233441b13f4137f3a7a9d5c28cc4))

* Update speed dependencies with new setuptools release (2429) ([`731c4fc`](https://github.com/embeddings-benchmark/mteb/commit/731c4fccf82f5841d30a50210b6945255d5dea58))

* Update tasks table ([`55c542b`](https://github.com/embeddings-benchmark/mteb/commit/55c542be5cd577159a74c0172e470daf16821d25))

* add __init__.py Clustering > kor folder, And edit __init__.py in Clustering folder (2422)

* add PatentFnBClustering.py

* do make lint and revise

* rollback Makefile

* Update mteb/tasks/Clustering/kor/PatentFnBClustering.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* klue_mrc_domain

* make lint

* klue_modified_clustering_dataset

* clustering & kor folder add __init.py

* clustering & kor folder add __init__.py

* task.py roll-back

* correct text_creation to sample_creation & delete form in MetaData

* correct task_subtype in TaskMetaData

* delete space

* edit metadata

* edit task_subtypes

---------

Co-authored-by: Roman Solomatin <samoed.romangmail.com> ([`39cee62`](https://github.com/embeddings-benchmark/mteb/commit/39cee62dc20c60f78143e261aae0a03784b06b17))

* HOTFIX: pin setuptools (2423)

* pin setuptools

* pin setuptools

* pin setuptools in makefile

* try ci

* fix ci

* remove speed from installs ([`071741d`](https://github.com/embeddings-benchmark/mteb/commit/071741d201643f5e2537cb4ac20545032914d5f4))

Fix

* fix: Add VDR Multilingual Dataset (2408)

* Added VDR Multilingual Dataset

* address comments

* make lint

* Formated Dataset for retrieval

* Update mteb/tasks/Retrieval/multilingual/VdrMultilingualRetrieval.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* Update mteb/tasks/Retrieval/multilingual/VdrMultilingualRetrieval.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* make lint

* corrected date

* fix dataset building

* move to image folder

---------

Co-authored-by: Roman Solomatin <samoed.romangmail.com>
Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`9d9b0b4`](https://github.com/embeddings-benchmark/mteb/commit/9d9b0b4329f29351b4c6dc596845205a97893744))

Unknown

* Update tasks table ([`34edcd5`](https://github.com/embeddings-benchmark/mteb/commit/34edcd552e9d4a055d794cdef5c985701b572892))

Fix

* fix: Add option to remove benchmark from leaderboard (2417)

fix: Add option to remove leaderboard from leaderboard

fixes 2413

This only removed the benchmark from the leaderboard but keep it in MTEB. ([`e8faf3f`](https://github.com/embeddings-benchmark/mteb/commit/e8faf3fb1c132d17caba4da5b6f570739f005a91))

Unknown

* Update tasks table ([`065159d`](https://github.com/embeddings-benchmark/mteb/commit/065159d999dacc2dce87beb5f876d2cc9ebcfc13))

* [MIEB] &34;capability measured&34;-Abstask 1-1 matching refactor [1/3]: reimplement CV-Bench (2414)

* refactor CV-Bench

* reimplement CV Bench

* remove abstask/evaluator/tests for Any2TextMultipleChoice

* rerun descriptive stats ([`2833138`](https://github.com/embeddings-benchmark/mteb/commit/283313845fdde1b07b33b280e74b7602bdcf79f3))

Fix

* fix: Add validation to model_name in `ModelMeta` (2404)

* add test for name validation

* upd docs

* upd cohere name

* fix tests

* fix name for average_word_embeddings_komninos

* fix name for average_word_embeddings_komninos

* fix reranker test

* fix reranker test ([`095851f`](https://github.com/embeddings-benchmark/mteb/commit/095851f66fbb87560fd677b04332b57293c3da61))

Unknown

* Min torchvision>0.2.1 (2410)

matching torch>1.0.0 ([`146a893`](https://github.com/embeddings-benchmark/mteb/commit/146a893c3fdcb0833fc1bd625ed567d6c2dba93d))

* remove Arabic_Triplet_Matryoshka_V2.py (2405) ([`811dbf6`](https://github.com/embeddings-benchmark/mteb/commit/811dbf6e16070731940c84de67ffb5bdbc256dcb))