Mteb

Fix

* fix: Updating paper scripts (1958)

* change reference revisions to align with paper

* Update author list

* Added code for main results table

* updated minor changes

* added external as a &34;no_revision_available&34; case

* revert unintended changes

* format ([`c6829d3`](https://github.com/embeddings-benchmark/mteb/commit/c6829d34d7a324bb1f3754d39dd52756921d6a9f))

* fix: Add climate fever v2 (1873)

* Updated ClimateFEVER dataset with new version

* Adds Fill in the empty metadata.

* Updates the date tuple

* Update class name

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Update domains

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Update task_subtypes

* Update annotations_creators for the first version

* Update date

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Update task subtypes

* Update path

* Update description

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>
Co-authored-by: Mina Parham <minaparhamKeatext.local> ([`8604e07`](https://github.com/embeddings-benchmark/mteb/commit/8604e079fd5bc6adac2d2050713dcceaeee5932d))

* fix: Added gte models (1539)

* fix: Added gte models

* fix: Add mixbai models (1540)

for 1515 ([`76e05dd`](https://github.com/embeddings-benchmark/mteb/commit/76e05ddb0006620eaf0b8850c5fb37bb74b943e1))

Unknown

* Update tasks table ([`11ced79`](https://github.com/embeddings-benchmark/mteb/commit/11ced79009a190f02a285743476745ca40d3d987))

* Update tasks table ([`479fa20`](https://github.com/embeddings-benchmark/mteb/commit/479fa206a9d2a1c3401fccd93786ac493769db56))

* misc: Add VisualSTS descriptive stats (2062)

* add visualsts stats

* add last dataset ([`3cf7b15`](https://github.com/embeddings-benchmark/mteb/commit/3cf7b158047498b516c46197a25c06d0d7004fa1))

* Rename MIEB task classes with duplicated names (2061)

fix class names ([`bef4046`](https://github.com/embeddings-benchmark/mteb/commit/bef4046b51fc24b3e0648d30307efb2ea0d89163))

* Update tasks table ([`e090330`](https://github.com/embeddings-benchmark/mteb/commit/e090330a6bbcf595d13ff682703cad524007b65e))

* misc: Add ZS and multilabel image classification descriptive stats implementation (2059)

* add image clustering descirptive stats and run

* finish off last one

* remove script

* add ImageMultilabelClassificationDescriptiveStatistics

* add VOC2007

* add zeroshot and mnist example ([`20df284`](https://github.com/embeddings-benchmark/mteb/commit/20df284af80d1623c0feb546e70a95d858e32dd8))

* Add giga embeddings (1741)

* add gigaembeddings

* use jasper

* fix name

* create sentence_transformer instruct wrapper

* apply instruction template

* fix jasper

* update meta ([`02d2583`](https://github.com/embeddings-benchmark/mteb/commit/02d258307099782d233c92e7764e490a98c62903))

Fix

* fix: Update embed_dim for jina models (2058)

see https://github.com/embeddings-benchmark/results/pull/117 ([`50b8e7b`](https://github.com/embeddings-benchmark/mteb/commit/50b8e7ba10c9a33d2febbc25be8b69893d0b50e6))

Unknown

* Update tasks table ([`48ef6f4`](https://github.com/embeddings-benchmark/mteb/commit/48ef6f4d2351042b1f5397ea1d2e28f58df2ecc0))

* misc: Add image clustering descriptive stats implementation (2057)

* add image clustering descirptive stats and run
* finish off last one
* remove script ([`eb32719`](https://github.com/embeddings-benchmark/mteb/commit/eb32719210a05ab33cb0d33319b9aeafb6455a91))

Fix

* fix: Add BRIGHT (long) and fix bug in TaskResult.filter_and_validate() (2041)

* fix: Add BRIGHT Long

Fixes 1978

* fix: Add BRIGHT(long)

* fix bug in task results

* updated bright

* updated tests for TaskResults ([`3537223`](https://github.com/embeddings-benchmark/mteb/commit/35372238b1f345ecf1422cb967186d8059213d07))

* fix: Add column descriptions to leaderboard (2039)

* fix: Add column descriptions to leaderboard

* removed existing overlap ([`01fd6fb`](https://github.com/embeddings-benchmark/mteb/commit/01fd6fbb2a7a2f54543a4b2a41ac96fd90cc61b2))

Unknown

* Update tasks table ([`fadba48`](https://github.com/embeddings-benchmark/mteb/commit/fadba483340e12cd1ccf15acd47b038f844f2b9b))

* misc: Add image classification descriptive stats implementation (2045)

* add ImageClassificationDescriptiveStatistics

* add MNIST descriptive stats

* use tuples instead

* add label count and update docstrings

* update MNIST example ([`9556f99`](https://github.com/embeddings-benchmark/mteb/commit/9556f99d6a2a12e665aadfe282708a7401de09a0))

Fix

* fix: add Voyage-code-3 (2040) ([`0b37966`](https://github.com/embeddings-benchmark/mteb/commit/0b3796658768cf4c213f83246112d5dbad9ac341))

Documentation

* docs: Fix README code rendering (2037)

Fix README code rendering ([`1b04130`](https://github.com/embeddings-benchmark/mteb/commit/1b041303bcf56708a0eb9080db6c67099d3fe3f6))

Fix

* fix: Add versioning to MTEB benchmarks (2024)

* Add versioning to MTEB benchmarks

- Following suggestion made in 2001 I added version to MTEB benchmarks
- changed the name of the MTEB(Chinese) to MTEB(cmn, v1). Though we could go for MTEB(Chinese, v1) assuming it is a group (also covering chinese other than mandarin)
- change the name of FaMTEB(fas, beta) to MTEB(fas)
- did a minor formatting of imports due to a circular import error
- moved the Benchmark object out of the file with the benchmarks
- this is still >1000 lines so we could split it up to &34;external_benchmarks&34;, &34;monolingual_benchmarks&34;, &34;domain_specific_benchmarks&34;, &34;multilingual_benchmarks&34;

Fixes 2001

* format

* update tests

* updated for backward compatibility

* ibid

* Update mteb/benchmarks/get_benchmark.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* Update mteb/benchmarks/get_benchmark.py

Co-authored-by: Roman Solomatin <samoed.romangmail.com>

* add missing import

---------

Co-authored-by: Roman Solomatin <samoed.romangmail.com> ([`65f3407`](https://github.com/embeddings-benchmark/mteb/commit/65f3407e91100b6176052e2e5137f8e88c4eaece))

Fix

* fix: Add SONAR metadata and resolve missing models (2014)

* Add SONAR metadata

Add SONAR metadat, but without an implementation

Fixes 1981

* fix: Add SONAR metadata

Fixes 1981

* minor edits

* reduced logging serverity of no model_meta.json

* resolve missing models

by ensuring that &34;Unknown&34; number of parameters is not filtered.

Should resolve:
1979
1976

This seems to have been caused by the restructering of calls on the leaderboard.

* format

* resolve missing models

by ensuring that &34;Unknown&34; number of parameters is not filtered.

Should resolve:
1979
1976

This seems to have been caused by the `MAX_MODEL_SIZE/MIN_MODEL_SIZE` args.

* format

* format

* added memory usage

* fixed None checks

* consistently refer to tasks as tasks not as datasets

adresses 2026

* minor

* removed used arg

* revert fix of not allowing None in model name ([`92b74b6`](https://github.com/embeddings-benchmark/mteb/commit/92b74b66e41e747eb6251ca87c21ebe071f1d83e))

* fix: Added script for generating and saving a local leaderboard (2015)

* Added script for making a local leaderboard

* Updated variable

* Update scripts/make_leaderboard.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com>

* Resolved comments

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsengmail.com> ([`477bea5`](https://github.com/embeddings-benchmark/mteb/commit/477bea5777f39b429e9a2509f5d1344754648ca0))

Unknown

* Add SFR-Embedding-Code-2B_R (2032)

* Add SFR-Embedding-Code-2B_R(2019)

* make lint

* Added HF repo name for adapted_from parameter

* replace instruct_wrapper ([`0caea67`](https://github.com/embeddings-benchmark/mteb/commit/0caea6704ede3e3f1414912db3450fe04eb38711))

* Add FRIDA model meta (2031)

* add FRIDA to models

* update prompts ([`f3c71b8`](https://github.com/embeddings-benchmark/mteb/commit/f3c71b8520c926037b7f846c1dbafeb8c1ada055))