Mteb

Fix

* fix: Ensure that summarization only calculates the metric across the statistic. (1157)

* fix: Ensure that summarization only calculates the metric across the statistic.

Fixes 1156

* Added versioning to datasets ([`e57efad`](https://github.com/embeddings-benchmark/mteb/commit/e57efad4655460506111cecbfba9144d813424d4))

Unknown

* Update tasks table ([`b47a81e`](https://github.com/embeddings-benchmark/mteb/commit/b47a81ed6c4b330988854ca63d68783f282acbb1))

Documentation

* docs: format notebooks and upgrade ruff (1164) ([`4e98381`](https://github.com/embeddings-benchmark/mteb/commit/4e98381993fa1c1e55ceeb4e1ba6f5d99d4f0192))

Fix

* fix: load_data in MintakaRetrieval (1155)

fix: add trust remote code ([`1a1ea2e`](https://github.com/embeddings-benchmark/mteb/commit/1a1ea2eaf9eded2435a613ac12203bbb288d4838))

Unknown

* Fix: test wrappers and export all tasks results in tests (1116)

* test wrappers and export all tasks results

* lint

* change to monkeypatch

* lint

* add Mxbai

* Update tests/test_benchmark/mock_models.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* remove create meta test

* remove gritlm models

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`7f3ad0e`](https://github.com/embeddings-benchmark/mteb/commit/7f3ad0e8c4929a518f0815d51193ea7f7b1acb4a))

* Update ruMTEB tasks list (1163)

* update Russian benchmark

* add detailed instructions

* add language filter

* fix lint ([`f619201`](https://github.com/embeddings-benchmark/mteb/commit/f619201a9ab91d453d7f500a0db6f8f3789a5840))

Documentation

* docs: Updating the contributor information (1161)

Update points.md

Hey, this just adds some &34;contributor information&34; about my coworker and myself ([`1325a6a`](https://github.com/embeddings-benchmark/mteb/commit/1325a6a922bc474d2e19b9c307fe30e7bd2a2579))

Fix

* fix: Added Task selection and aggregation (1140)

* task metric selection initial commit (891)

Co-authored-by: vaibhavad <vaibhavadlakha95gmail.com>

* Add changes to PR

* save

* save

* Added significant rank

* clean up

* save attemtps

* add scores plot

* update scores plot

* latent models

* Added utility function for downloading models results

* Added result tables

* Added bench construction

* format

* Added correction citations

* format

* sav

* sav

* sav

* added task selection for EU languages

* remove token

* clean up branch

* clean up branch

* merge

* minor

* minor

* fix metadata

* fix: Move benchmarks from script to benchmarks.py

- A few of the benchmarks weren&39;t implemented in the benchmark list. The best choice seem to move them benchmarks and remove the script (there were already a few issues). To ensure that they are maintained with the rest.
- Removed trivial (e.g. running one task) or outdated examples. Removing them ensure that we have less to maintain.
- moved mteb specific script into a mteb-folder.

* format

* fix: Convert arbitrary imports to absolute imports.

This avoids loading in classes as modules (see PR ...)

* Added example

* Final correction to notebook

* format

* reorder scripts

* remove additional files

* remove plots

* fixed loading

* Added corrections

---------

Co-authored-by: vaibhavad <vaibhavadlakha95gmail.com> ([`57f4fb2`](https://github.com/embeddings-benchmark/mteb/commit/57f4fb254391285cb93a82228abf61baafc2b9ba))

Unknown

* Update tasks table ([`b879133`](https://github.com/embeddings-benchmark/mteb/commit/b87913368736854fd43a47a3b34b44cf3be7218e))

* Update tasks table ([`1e5cdaf`](https://github.com/embeddings-benchmark/mteb/commit/1e5cdafc3cd625d50aa5008a0a1b00110cbff3d1))

* Update points table ([`35a17c5`](https://github.com/embeddings-benchmark/mteb/commit/35a17c551f1098ff0df35896e0de28f2ac598b76))

* LitSearchRetrieval added (1150)

* LitSearchRetrieval added

* points added

* fixed descriptive_stats in LitSearchRetrieval

* fix lint

* metadata fixed for LitSearchRetrieval

* metadata fixed for LitSearchRetrieval again

* metadata fixed for LitSearchRetrieval again

* points 1150 updated

* metadata fixed for LitSearchRetrieval again

* lint fix ([`a6458aa`](https://github.com/embeddings-benchmark/mteb/commit/a6458aa283c7253a6347f78d3043b30fecc3e2e8))

Fix

* fix: removed test set for AFQMC with no gold labels (1153)

The test set of AFQMC is hidden so has removed it from the default suite. This influences the C-MTEB. Who is the best person to contact for this? (Muennighoff) ([`14cff3e`](https://github.com/embeddings-benchmark/mteb/commit/14cff3e7830150e28b45098a008048189c7d1e1e))

Unknown

* Fix: llm2vec for retrieval (1152)

* fix: remove kwargs LLM2Vec doesn&39;t have

* fix: linter error

* fix: an item automatically added by linter ([`d1794cd`](https://github.com/embeddings-benchmark/mteb/commit/d1794cd2edaf9236699d6d1ccba4cbc15b5f8708))

Feature

* feat: Cleaned up scripts folder (1144)

* fix: Move benchmarks from script to benchmarks.py

- A few of the benchmarks weren&39;t implemented in the benchmark list. The best choice seem to move them benchmarks and remove the script (there were already a few issues). To ensure that they are maintained with the rest.
- Removed trivial (e.g. running one task) or outdated examples. Removing them ensure that we have less to maintain.
- moved mteb specific script into a mteb-folder.

* format

* fix: Convert arbitrary imports to absolute imports. (1145)

This avoids loading in classes as modules (see PR ...)

* restructe scripts folder ([`ebe6def`](https://github.com/embeddings-benchmark/mteb/commit/ebe6defb6942e7e98edcece6eb8be9820ce7899a))

Fix

* fix: Remove unused tests (1148)

These tests are no longer used since results moved to the results repo. ([`28f592f`](https://github.com/embeddings-benchmark/mteb/commit/28f592f2289b31d8de5e49b76a49e19f94810ebc))

Unknown

* Update tasks table ([`8560d9f`](https://github.com/embeddings-benchmark/mteb/commit/8560d9fb4bec43e584540b0d1642b02cfd449c44))

* Missing import for SadeemQuestionRetrieval (1146)

Fixes 1143 ([`3b6e27d`](https://github.com/embeddings-benchmark/mteb/commit/3b6e27d6c0c58c2624c2f3cb40e30f8f20fea39d))

* Simplify models (1118)

* Merge

* Adapt

* Simplify

* Check for rev again

* Rmv cmmnt

* Simplify

* simplify

* Rmv comment

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Use logging; change try except; add info

* Lint

* Rmv results

* Update rev

* Simplify models; Allow instructions

* Jobs

* Fix merge

* Format

* Adapt models

* Fix task types

* Update

* Fix syntax

* Simplify

* Add comparison

* Format

* Fix double comment

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Move example

* Format

* Rmv outdated instructions

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`79c1a42`](https://github.com/embeddings-benchmark/mteb/commit/79c1a420f62716c2dfd1284185d5f35c630e0f47))