Mteb

Fix

* fix: Add Retrieval SK Quad dataset for Slovak search evaluation (1276)

* Add Retrieval SK Quad dataset for Slovak search evaluation

This commit introduces the Retrieval SK Quad dataset, designed to assess Slovak search performance. The dataset is derived from SK-QuAD and includes questions with their best answers categorized post-annotation. This addition provides a significant resource for advancing Slovak language search evaluation and supporting further research and development.

* Add Retrieval SK Quad dataset for Slovak search evaluation 2

Added the requested changes on the SKQuadRetrieval.py file

* add task to init

* add missing task metadata

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`fc53498`](https://github.com/embeddings-benchmark/mteb/commit/fc534980b27d3909eaa06943e60480fde41d926e))

Unknown

* Update tasks table ([`95f012a`](https://github.com/embeddings-benchmark/mteb/commit/95f012ac7a43159e93f911053e9628b8e5c25436))

Feature

* feat: Use prompts instead of encode_corpus and encode_queries (1278)

* add prompt per task type

* fix prompt

* upd test

* lint

* fix test

* fix DeprecatedSummarizationEvaluator

* fix prompts

* add test

* lint

* logger info

* use task type only in model_encode

* lint

* update interface

* add prompt types to docs

* fix test

* mock tasks

* mock task registry

* remove last task_type

* fix tests

* lint

* fix test

* fix

* use wrapper and new prompts

* fix tests

* lint

* fix test

* remove conftest

* validate task to prompt_name

* override model prompts

* task to prompt name optional

* fix tests

* fix models

* remove task_to_prompt_name

* remove from mteb __init__

* update docs

* load existing model prompts if model_prompts is None

* fix

* lint

* change wrapper loader

* add wrapper class

* lint

* add wrapper file

* update logging

* upd logging

* refactor reranking

* lint

* remove prints ([`2a61821`](https://github.com/embeddings-benchmark/mteb/commit/2a61821d9294eb5b0cb053e1e676c199f23be12b))

Unknown

* Leaderboard (1235)

* Add leaderboard dev

* Renamed MTEBResults to TaskResult

* Moved model and model meta loading utilities into overview.py

* Added get_model_metas to retrieve filtered metadata for models

* Restructured results object and made it into a class instead of a dict

* Added utilities for filtering models on BenchmarkResults objects

* Added to_table utility function to BenchmarkResults

* Added serialization utilities to BenchmarkResults

* Attempted fixing tests

* Added get_model_metas to __init__

* Added get_benchmarks to __init__ and made it return all benchmarks by default

* Added get_benchmarks to __init__

* Made tasks hashable

* Added task filtering based on task objects on BenchmarkResults

* Added BenchmarkResults to __init__

* Added additional arguments to get_scores on two classes

* Made get_scores smarter on BenchmarkResult

* Added basic multilingual benchmark

* Modified benchmark to be able to easily access results

* Added useful properties and filtering functions to BenchmarkResults

* Added minimal functioning example

* Added smarter table, task-list updating and tried fixing dropdown scrolling

* Made restrict_results into a private function

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Removed old leaderboard scripts

* Hardcoded max and min model size

* Removed redundant utils file

* Ran linting

* added leaderboard dependencies as optional

* Fixed union type error on Python 3.9

* Removed references to Dict in task aggregation

* Fixed name errors in _restrict_task_results

* Fixed _restrict_task_results

* Made hf_subsets={&39;default&39;} when the task is monolingual in _restric_task_results

* Task dropdown now gets filtered based on the other criteria

* Ran linting again

* Introduced hotfix for reranking test

* Added BenchmarkResults to __all__ in __init__

* Fixed validate_and_filter_scores method, and replaced _restric_task_results with it

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`094f922`](https://github.com/embeddings-benchmark/mteb/commit/094f9225973268fedae55b322eb9af7bb0ae2110))

Fix

* fix: Remove non-existent eval split of CMNLI (1294)

fix eval_splits of CMNLI ([`5b4b262`](https://github.com/embeddings-benchmark/mteb/commit/5b4b262555d8c9d55aec4d178b68be33616c145d))

Fix

* fix: Add metadata dict to QBQTC in C-MTEB (1292)

* fix QBQTC in C-MTEB

* make lint

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`4a88a1d`](https://github.com/embeddings-benchmark/mteb/commit/4a88a1d8b515efae58a06b687681a04598ae8db2))

Fix

* fix: Allow numpy<2.0.0 (1291) ([`60cef98`](https://github.com/embeddings-benchmark/mteb/commit/60cef9834045c372861d4700ac87b5e9753e94a3))

Documentation

* docs: Fix a link in the README (1289)

* Fix a link in the README

And fix some typos.

* Update README.md ([`c1ebd6e`](https://github.com/embeddings-benchmark/mteb/commit/c1ebd6e84acdc790dd3190e854e1a46539c2c902))

* docs: points for paper writing (1286)

* Create 1004.jsonl

* Create 1006.jsonl

* Update docs/mmteb/points/1004.jsonl

* Update docs/mmteb/points/1006.jsonl

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`426f4a1`](https://github.com/embeddings-benchmark/mteb/commit/426f4a136f579f264f3e60a6919ac7d68d265107))

Fix

* fix: Update benchmarks (1288)

* make benchmark var name uppercase

* update touche to v3

* add MIRACLRetrievalHardNegatives to multilingual

* add mteb(indic)

* add eu benchmark ([`f55a888`](https://github.com/embeddings-benchmark/mteb/commit/f55a888b8244798b0bd5763b8a09481d051fe935))

Unknown

* Update points table ([`1123788`](https://github.com/embeddings-benchmark/mteb/commit/112378832ecb22f29169e4de283c1de1242b1576))

* Update points table ([`b1a6cae`](https://github.com/embeddings-benchmark/mteb/commit/b1a6caef3aa2f4733d890b3ef1057478676dca34))

* Update points table ([`c77ad6e`](https://github.com/embeddings-benchmark/mteb/commit/c77ad6ebeead9c5632a0804d47d1d16839f4f785))

* Update points table ([`73e829d`](https://github.com/embeddings-benchmark/mteb/commit/73e829dc59d326c1ee30171853d4eb5e972b6183))