Mteb

Fix

* fix: Re-upload dataset to hub to avoid using script upload (1322)

* fix dataset upload

* add linting ([`f00a262`](https://github.com/embeddings-benchmark/mteb/commit/f00a2622821eeec68e191561ca9f2b346f0a5dc6))

Unknown

* Update tasks table ([`e5b6c12`](https://github.com/embeddings-benchmark/mteb/commit/e5b6c12b2578c5421dce46df259c6327342b7681))

Fix

* fix: remove duplicate multilingual ([`2f14519`](https://github.com/embeddings-benchmark/mteb/commit/2f1451955da42070bf6aea4c317c4bc3da755a38))

Fix

* fix: Add Slovak Hate Speech and Offensive Language Dataset (1274)

* Add Slovak Hate Speech and Offensive Language
Dataset

This commit introduces the Slovak Hate Speech and Offensive Language Database to MTEB. The dataset includes posts from a social network, annotated by humans for hate speech and offensive content. Additionally, the corresponding task has been added to the tasks.md table to reflect this update.

* Add Slovak Hate Speech and Offensive Language Dataset
- Updated __init__.py to include the new SlovakHateSpeechClassification task.
- Modified SlovakHateSpeechClassification.py as per review suggestions to enhance functionality and readability.

* Did requested changes:
- Updated __init__.py to include the new SlovakHateSpeechClassification task.
- Modified SlovakHateSpeechClassification.py as per review suggestions to enhance functionality and readability.

* resolve linting issues by running `make lint` ([`f3d8014`](https://github.com/embeddings-benchmark/mteb/commit/f3d8014fc91dfdf400ab7713683afe4cf785cabf))

Unknown

* WIP: Leaderboard UI improvements (1312)

* Fixed typos in task_results

* Fixed typos in task_results

* Added Tailwind, reorganized layout and fixed scrolling

* Ran linting ([`bd5ee9e`](https://github.com/embeddings-benchmark/mteb/commit/bd5ee9eca947a656f6ed0d83971615d53a85475a))

* Update tasks table ([`0d86753`](https://github.com/embeddings-benchmark/mteb/commit/0d8675338a0c99a606daa19bbc2a35d904b5c93f))

Fix

* fix: Add Retrieval SK Quad dataset for Slovak search evaluation (1276)

* Add Retrieval SK Quad dataset for Slovak search evaluation

This commit introduces the Retrieval SK Quad dataset, designed to assess Slovak search performance. The dataset is derived from SK-QuAD and includes questions with their best answers categorized post-annotation. This addition provides a significant resource for advancing Slovak language search evaluation and supporting further research and development.

* Add Retrieval SK Quad dataset for Slovak search evaluation 2

Added the requested changes on the SKQuadRetrieval.py file

* add task to init

* add missing task metadata

---------

Co-authored-by: Isaac Chung <chungisaac1217gmail.com> ([`fc53498`](https://github.com/embeddings-benchmark/mteb/commit/fc534980b27d3909eaa06943e60480fde41d926e))

Unknown

* Update tasks table ([`95f012a`](https://github.com/embeddings-benchmark/mteb/commit/95f012ac7a43159e93f911053e9628b8e5c25436))

Feature

* feat: Use prompts instead of encode_corpus and encode_queries (1278)

* add prompt per task type

* fix prompt

* upd test

* lint

* fix test

* fix DeprecatedSummarizationEvaluator

* fix prompts

* add test

* lint

* logger info

* use task type only in model_encode

* lint

* update interface

* add prompt types to docs

* fix test

* mock tasks

* mock task registry

* remove last task_type

* fix tests

* lint

* fix test

* fix

* use wrapper and new prompts

* fix tests

* lint

* fix test

* remove conftest

* validate task to prompt_name

* override model prompts

* task to prompt name optional

* fix tests

* fix models

* remove task_to_prompt_name

* remove from mteb __init__

* update docs

* load existing model prompts if model_prompts is None

* fix

* lint

* change wrapper loader

* add wrapper class

* lint

* add wrapper file

* update logging

* upd logging

* refactor reranking

* lint

* remove prints ([`2a61821`](https://github.com/embeddings-benchmark/mteb/commit/2a61821d9294eb5b0cb053e1e676c199f23be12b))

Unknown

* Leaderboard (1235)

* Add leaderboard dev

* Renamed MTEBResults to TaskResult

* Moved model and model meta loading utilities into overview.py

* Added get_model_metas to retrieve filtered metadata for models

* Restructured results object and made it into a class instead of a dict

* Added utilities for filtering models on BenchmarkResults objects

* Added to_table utility function to BenchmarkResults

* Added serialization utilities to BenchmarkResults

* Attempted fixing tests

* Added get_model_metas to __init__

* Added get_benchmarks to __init__ and made it return all benchmarks by default

* Added get_benchmarks to __init__

* Made tasks hashable

* Added task filtering based on task objects on BenchmarkResults

* Added BenchmarkResults to __init__

* Added additional arguments to get_scores on two classes

* Made get_scores smarter on BenchmarkResult

* Added basic multilingual benchmark

* Modified benchmark to be able to easily access results

* Added useful properties and filtering functions to BenchmarkResults

* Added minimal functioning example

* Added smarter table, task-list updating and tried fixing dropdown scrolling

* Made restrict_results into a private function

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Removed old leaderboard scripts

* Hardcoded max and min model size

* Removed redundant utils file

* Ran linting

* added leaderboard dependencies as optional

* Fixed union type error on Python 3.9

* Removed references to Dict in task aggregation

* Fixed name errors in _restrict_task_results

* Fixed _restrict_task_results

* Made hf_subsets={&39;default&39;} when the task is monolingual in _restric_task_results

* Task dropdown now gets filtered based on the other criteria

* Ran linting again

* Introduced hotfix for reranking test

* Added BenchmarkResults to __all__ in __init__

* Fixed validate_and_filter_scores method, and replaced _restric_task_results with it

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`094f922`](https://github.com/embeddings-benchmark/mteb/commit/094f9225973268fedae55b322eb9af7bb0ae2110))

Fix

* fix: Remove non-existent eval split of CMNLI (1294)

fix eval_splits of CMNLI ([`5b4b262`](https://github.com/embeddings-benchmark/mteb/commit/5b4b262555d8c9d55aec4d178b68be33616c145d))