Mteb

Documentation

* docs: Create benchmarks overview table (1245)

* fix get_benchmarks method

* add create benchmark script

* make lint ([`fda9be1`](https://github.com/embeddings-benchmark/mteb/commit/fda9be1c085b5d83be58253e32e4aca8e4a2d594))

* docs: Add MTEB(code) dataset (1237)

* docs: Add MTEB(code) dataset

* Fix linting ([`f808863`](https://github.com/embeddings-benchmark/mteb/commit/f808863f5e393fc472058539fb113efe47e0abf4))

* docs: Update points (1228)

* Fix case

* Fix casing

* Fix case

* Fix case

* Create 971.jsonl

* Update contrib

* Add contributors ([`a636dc2`](https://github.com/embeddings-benchmark/mteb/commit/a636dc28e968e5689f465983cbdad40481893e6f))

Fix

* fix: mrshu&39;s name in `points.md` (1246)

* Use the diacritic character to be inline with Slovak spelling.

Signed-off-by: mr.Shu <mrshu.io> ([`3c06694`](https://github.com/embeddings-benchmark/mteb/commit/3c06694cabb6f6b8d71543aace90f1086cf296e5))

Unknown

* Add contributor (1243) ([`18f7306`](https://github.com/embeddings-benchmark/mteb/commit/18f730696451a5aaa026494cecf288fd5cde9fd0))

* Update of my affiliation (1242)

Update points.md ([`7469afa`](https://github.com/embeddings-benchmark/mteb/commit/7469afabbbc3b40ca85949472267ecf2baa09612))

* Update points table ([`eece6ec`](https://github.com/embeddings-benchmark/mteb/commit/eece6ecdb3248b1fb5a044d33c904f3d48cd6eab))

* Update points table ([`dddf5c2`](https://github.com/embeddings-benchmark/mteb/commit/dddf5c249bd102cd22f46965e38c1ebe0896a33d))

Documentation

* docs: clarify adding a model (1222) ([`25b7a2f`](https://github.com/embeddings-benchmark/mteb/commit/25b7a2fd1c6c1c24a48bf3c2c4d8c00dfa2820b9))

Fix

* fix: Add RepLLaMA style models (1223)

* init commit

* working and reproducing

* lint

* update hashes

* warning

* add pyproject ([`bedcfb3`](https://github.com/embeddings-benchmark/mteb/commit/bedcfb3e3991c0573aebb05add24c4f627e14f92))

Unknown

* Update points table ([`694e563`](https://github.com/embeddings-benchmark/mteb/commit/694e5639c0622af3bf90de5855c1af1ab01d86df))

* Update points table ([`4de39ee`](https://github.com/embeddings-benchmark/mteb/commit/4de39ee917565c8b5da0a101c4769e5b025745e2))

* Update points table ([`b66aaeb`](https://github.com/embeddings-benchmark/mteb/commit/b66aaeb0919707530a41ad983c61413bd6933a53))

Fix

* fix: Allow benchmark to specify eval_splits (1217)

* fix: Allow benchmark to specify eval_splits

This PR allow for benchmarks to specify specific eval. splits. This allow us to fully specify a benchmark within the benchmark object.

To do this it add the following:
- added eval_splits to the Abstask object, which default to metadata.eval_splits
- use the task.eval_splits unless overwritten in mteb.MTEB.run
- added eval_splits arg to mteb.get_tasks, which filter the tasks based on splits
- updated documentation
- renamed the &34;Advanced Usage&34; to &34;Usage Documentation&34; to make it more accicible
- added tests where relevant

* Added correction based on feedback ([`00260b5`](https://github.com/embeddings-benchmark/mteb/commit/00260b5497b4c82583be6383d8b22a3fceb64b54))

Documentation

* docs: Fix broken links in docs (1212)

* Added fixes for broken links in adding_a_dataset and adding_a_model docs.

* Updated link name ([`b1bd941`](https://github.com/embeddings-benchmark/mteb/commit/b1bd9410715aeadf26af34d6845ddd0a7ee3ade8))

Fix

* fix: Ensure that results are returned even when hitting cache (1215)

Fixes 1122 ([`64e01ae`](https://github.com/embeddings-benchmark/mteb/commit/64e01ae9d6fcf125a4ea6516263fa062b2aafeef))

Unknown

* Update tasks table ([`88b4f6e`](https://github.com/embeddings-benchmark/mteb/commit/88b4f6eda695201ee297cf0e6483344cba9a5985))

* Mismatch of the category of AmazonPolarityClassification (1220)

Fixes 1219 ([`4595b19`](https://github.com/embeddings-benchmark/mteb/commit/4595b198a7aa2f297999e32d25cb116d12ad1e7d))

Fix

* fix: Normalize benchmarks no only include task objects and added getter for benchmarks (1208)

* Normalize benchmarks to only include tasks

- Force benchmarks to only include tasks. This fixes a few bugs where benchmarks can reference a task which is not implemented
- implements `mteb.get_benchmark`, which makes it easier to fetch benchmarks
- Added tests + updated docs

A few outstanding issues:

I would like `mteb.MTEB(benchmark)` to always reproduce the benchmark. Currently this is not possible as MTEB(eng) required the split to be specified. A solution it to allow &34;eval_splits) to be specified when initializing a task and then pass it on to the `load_data()`. This way we can write the following:

`mteb.get_tasks(tasks=[...], eval_splits=[&34;test&34;], ...)`

I would also love the aggregation to be a part of the benchmark (such that it is clear how it should be aggregated). This is especially relevant for MTEB(eng) as it average the CQAD datasets before creating the global average. This way we can also create a result object for the benchmark itself. A complimenting solution for this is to allow nested benchmarks.

* fix error in tests

* format

* Added corrections based on review

* added example and formatted ([`f93154f`](https://github.com/embeddings-benchmark/mteb/commit/f93154f465b99bd9737b2ecfd54b3beb491a996d))

Fix

* fix: Normalize licenses including casing, uses of &34;-&34; etc. (1210)

* fix: Normalize licenses including casing, uses of &34;-&34; etc.

* fix tests ([`768c031`](https://github.com/embeddings-benchmark/mteb/commit/768c031d3e1e29e39edcf20dd4f9f1ea6092db50))

* fix: Normalize licenses including casing, uses of &34;-&34; etc. ([`a8f7d80`](https://github.com/embeddings-benchmark/mteb/commit/a8f7d80e20efd97b0c00ef2c028eba830ce1d308))