Mteb

Major changes to SummEval:
- https://github.com/embeddings-benchmark/mteb/pull/99: Batched evaluation making it significantly faster
- https://github.com/embeddings-benchmark/mteb/pull/99: Standardized interface to call encode with List[str] ; Previously it called model.encode with str, which lead to some wrong scores; All scores on the [leaderboard](https://huggingface.co/spaces/mteb/leaderboard) have already been fixed
- https://github.com/embeddings-benchmark/mteb/pull/97: Fixed typos

Other:
- A new version of the paper has been released with cleaner plots & some additional scores 😊
- Lots of cool models have been added to the leaderboard: https://huggingface.co/spaces/mteb/leaderboard
- MTEB has been accepted to the EACL 2023 conference

There have been several problems with the GPU parallelism employed by the beir package for Retrieval tasks, such as [here](https://github.com/embeddings-benchmark/mteb/issues/85) & [here](https://github.com/embeddings-benchmark/mteb/issues/89). This patch release rolls back the support for GPU parallelism with beir. Encoding will still automatically use GPUs, but only a single one. We hope the issue in beir will be fixed soon, so we can enable GPU parallelism for Retrieval tasks in MTEB again! 🤗

- We've released our paper 🤗 https://arxiv.org/abs/2210.07316
- Added some useful scripts for merging results & SLURM scripts in this repository: https://github.com/embeddings-benchmark/mtebscripts

Ci

* ci: renamed test job and workflow (282)

ci: Added tests ([`6675bb8`](https://github.com/embeddings-benchmark/mteb/commit/6675bb8668ff17ca8cf3cce2703f3ebf17795bfc))

Documentation

* docs: typos in readme (268) ([`aa9234c`](https://github.com/embeddings-benchmark/mteb/commit/aa9234cc24f6dd3408961895d092ee019551fab2))

* docs: add dataset schemas (255)

* docs: update AbsTaskClassification.py document schema for classification task

* update AbsTaskBitextMining.py

* update BornholmskBitextMining.py

* update AbsTaskClustering.py and BlurbsClusteringP2P.py

* update 8 files

* update 9 files

* update AbsTaskReranking.py

* update BlurbsClusteringP2P.py

* update CMTEBPairClassification.py

* update GerDaLIRRetrieval.py

* update 7 files

* update AbsTaskBitextMining.py

* update AbsTaskClassification.py ([`c3ce1ac`](https://github.com/embeddings-benchmark/mteb/commit/c3ce1ac8ac92baf9a7481c30d476b45e3ec36786))

* docs: Add development installation instructions (246)

* docs: Add development installation instructions

* removed unused requirements file

I don&39;t believe this is nec. with the setup.py specifying the same dependencies

* docs: Updated make file with new dependencies

* ci: Update ci to use make commands

This ensure that the user runs exactly what the CI expects

* ci: Avoid specifying tests folder as it causes issuew ith tests

* ci: removed unec. args for test ci

* Added dev install ([`0048878`](https://github.com/embeddings-benchmark/mteb/commit/0048878deba9f57147c3696dcb89ade098c90376))

Feature

* feat: update revision id of wikicitiesclustering task ([`fb90c02`](https://github.com/embeddings-benchmark/mteb/commit/fb90c022e11834ae6605f5bbb0a79af701793a96))

Fix

* fix: dead link in readme ([`ecbb776`](https://github.com/embeddings-benchmark/mteb/commit/ecbb776fba460c531f09e7b0ce986f075f2b665a))

* fix: Added sizes to the metadata (276)

* restructing the readme

* added mmteb

* removed unec. method

* Added docstring to metadata

* Updated outdated examples

* formatting documents

* fix: Updated form to be parsed correctly

* fix: Added sizes to the metadata

this allow for automatic metadata generations

* Updated based on feedback

* Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

* updated based on feedback

* Added suggestion from review

* added correction based on review

* reformatted empty fields to None

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`cd4a012`](https://github.com/embeddings-benchmark/mteb/commit/cd4a012271463b89db7a8ec9ca298a975805988d))

* fix: remove debugging print statement ([`d292d93`](https://github.com/embeddings-benchmark/mteb/commit/d292d937ceedb5d137537b2c25a0f135d1bb91b9))

* fix: pass parallel_retrieval kwarg to use DenseRetrievalParallelExactSearch ([`19b8f66`](https://github.com/embeddings-benchmark/mteb/commit/19b8f6619f07dfd95860f43f9376af230978f447))

* fix: msmarco-v2 uses dev.tsv, not dev1.tsv ([`6908d21`](https://github.com/embeddings-benchmark/mteb/commit/6908d21cfce644140bd70df47df0452c551ee0d0))

* fix: add missing task-langs attribute (152) ([`bc22909`](https://github.com/embeddings-benchmark/mteb/commit/bc22909c49284efb0df1d997ac23806694424a94))

Refactor

* refactor: add metadata basemodel (260)

* refactor: rename description to metadata dict

* refactor: add TaskMetadata and first example

* update 9 files

* update TaskMetadata.py

* update TaskMetadata.py

* update TaskMetadata.py

* update LICENSE, TaskMetadata.py and requirements.dev.txt

* update 151 files

* update 150 files

* update 43 files and delete 1 file

* update 106 files

* update 45 files

* update 6 files

* update 14 files

* Added model results to repo and updated CLI to create consistent folder structure. (254)

* Added model results to repo and updated CLI to create consistent folder structure.

* ci: updated ci to use make install

* Added missing pytest dependencies

* Update README.md

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

* Restructing the readme (262)

* restructing the readme

* removed double specification of versions and moved all setup to pyproject.toml

* correctly use flat-layout for the package

* build(deps): update TaskMetadata.py and pyproject.toml

* update 221 files

* build(deps): update pyproject.toml

* build(deps): update pyproject.toml

* build(deps): update pyproject.toml

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>
Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`dd5d617`](https://github.com/embeddings-benchmark/mteb/commit/dd5d61724e71b2cdba9f9cf7e01fbed1b81cb423))

Unknown

* Ci-fix (289)

* added release pipeline

* v1.3.0

* ci: moved release to the correct folder ([`7f56c1a`](https://github.com/embeddings-benchmark/mteb/commit/7f56c1a7d2eb2fab6eb028291d85054727c650d1))

* v1.3.0

* added release pipeline

* v1.3.0 ([`5e4d10e`](https://github.com/embeddings-benchmark/mteb/commit/5e4d10e5224d21e51d0ffcd87abee42008f2446c))

* tests: speed up tests (283)

update Makefile and test_all_abstasks.py ([`2155bf6`](https://github.com/embeddings-benchmark/mteb/commit/2155bf66c2ea5435744c47b03e3f14b6a5df1813))

* update TaskMetadata.py (281) ([`acfd7d4`](https://github.com/embeddings-benchmark/mteb/commit/acfd7d420fc3aa05d624db43c5b41f85b1a93367))

* Merge branch &39;main&39; of https://github.com/embeddings-benchmark/mteb ([`c9d1a03`](https://github.com/embeddings-benchmark/mteb/commit/c9d1a03c7d8531a0293cbd24e523e904c2be9477))

* Enable ruff ci (279)

* restructing the readme

* added mmteb

* removed unec. method

* Added docstring to metadata

* Updated outdated examples

* formatting documents

* fix: Updated form to be parsed correctly

* fix: Added sizes to the metadata

this allow for automatic metadata generations

* Updated based on feedback

* Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

* updated based on feedback

* Added suggestion from review

* added correction based on review

* reformatted empty fields to None

* CI: Enable linter

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`a16eb07`](https://github.com/embeddings-benchmark/mteb/commit/a16eb07da1d1a6d8380683e9fa11df3244fae87b))

* Added MMTEB (275)

* restructing the readme

* added mmteb

* removed unec. method

* Added docstring to metadata

* Updated outdated examples

* formatting documents

* fix: Updated form to be parsed correctly

* Updated based on feedback

* Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

* updated based on feedback

* Added suggestion from review

* added correction based on review

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`c0dc49a`](https://github.com/embeddings-benchmark/mteb/commit/c0dc49a6b99f4d8136b7ec46c49563d7e1b866db))

* dev: add ruff as suggested extension (274) ([`b08913f`](https://github.com/embeddings-benchmark/mteb/commit/b08913f8616c580f8bbb15bfa808549e2b74912a))

* dev: add isort (271)

* dev: add isort

* dev: add isort ([`845099d`](https://github.com/embeddings-benchmark/mteb/commit/845099d5b49b0757cc4cf23c08c6d7f65627538e))

* dev: run tests on pull request towards any branch ([`13f759a`](https://github.com/embeddings-benchmark/mteb/commit/13f759a62bff085e156e4d115f64604c2dc0f087))

* Merge branch &39;main&39; of https://github.com/embeddings-benchmark/mteb ([`b42abe4`](https://github.com/embeddings-benchmark/mteb/commit/b42abe4e37f96dd0cab898b381d644d507a228a1))

* replaced linter with ruff (265)

* restructing the readme

* removed double specification of versions and moved all setup to pyproject.toml

* correctly use flat-layout for the package

* replaced linter with ruff

* rerun tests

* ci: Added in newer workflow

some of them are disables as they require other issues to be solved

* Update Makefile

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`023e881`](https://github.com/embeddings-benchmark/mteb/commit/023e8817f108a76718fc37f7c8937e000de56786))

* Restructing the readme (262)

* restructing the readme

* removed double specification of versions and moved all setup to pyproject.toml

* correctly use flat-layout for the package ([`769157b`](https://github.com/embeddings-benchmark/mteb/commit/769157b1e49c97ee6ca334a299392392bc3a6523))

* restructing the readme ([`364be7f`](https://github.com/embeddings-benchmark/mteb/commit/364be7f4a26275263c9a82de594e47aaf28a1bcf))

* Added model results to repo and updated CLI to create consistent folder structure. (254)

* Added model results to repo and updated CLI to create consistent folder structure.

* ci: updated ci to use make install

* Added missing pytest dependencies

* Update README.md

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com>

---------

Co-authored-by: Niklas Muennighoff <n.muennighoffgmail.com> ([`8a758bc`](https://github.com/embeddings-benchmark/mteb/commit/8a758bce00a6bc64dd4f0dab98f5bc3e0c683f46))

* dev: add workspace defaults in VSCode (253)

* dev: add black as default formatter in vscode

* Update .vscode/settings.json

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`30e5b9e`](https://github.com/embeddings-benchmark/mteb/commit/30e5b9ebb288d923b3c7e8d0e55728f82a1bc5d8))

* Add Danish Discourse dataset (247)

* misc.

* update ddisco.py

* chore: delete ddisco.py, ddisco.test.tsv and ddisco.train.tsv

* Update mteb/tasks/Classification/DdiscoCohesionClassification.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Update mteb/tasks/Classification/DdiscoCohesionClassification.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Update mteb/tasks/Classification/DdiscoCohesionClassification.py

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com>

* Update mteb/tasks/Classification/DdiscoCohesionClassification.py

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com>

* Update mteb/tasks/Classification/DdiscoCohesionClassification.py

Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com>

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>
Co-authored-by: Imene Kerboua <33312980+imenelydiakerusers.noreply.github.com> ([`d46d0f5`](https://github.com/embeddings-benchmark/mteb/commit/d46d0f5281d5a9036501aca437e9c76480dc8885))

* Update structure of mteb/tasks to mteb/tasks/{type}/{language}  (245)

* Fix structure of mteb/tasks
Fixes 243

* fix: Added missing init files ([`b1c78c1`](https://github.com/embeddings-benchmark/mteb/commit/b1c78c1121fbd488d62d1385d6f000d3f3b46ef4))

* tests: do not run tests on collection (249)

test: update test_all_abstasks.py ([`236614a`](https://github.com/embeddings-benchmark/mteb/commit/236614add5de4e848bc0f1db8ad997f451ae2906))

* Update README.md with correct DRESModel location ([`399edf4`](https://github.com/embeddings-benchmark/mteb/commit/399edf4331cd316b28a832fd37e187d5c5e204f1))

* Fix typo ([`9610378`](https://github.com/embeddings-benchmark/mteb/commit/96103788cd135caecdb439b5d26af38ab6cd33ef))

* Set dev version ([`716f59c`](https://github.com/embeddings-benchmark/mteb/commit/716f59cae9be31a747371497ec6792b23070270c))

- Test release prior to 1.0.0 with minor fixes

Lots of bug fixes across all tasks. 👻
We aim to make this the final version before the initial 1.0.0 release. 🤗