Lighteval

Release Note

This introduced the new extended tasks feature, documentation and many other patches for improved stability.
New tasks are also introduced:
- Big Bench Hard: https://huggingface.co/papers/2210.09261
- AGIEval: https://huggingface.co/papers/2304.06364
- TinyBench:
- MT Bench: https://huggingface.co/papers/2306.05685
- AlGhafa Benchmarking Suite: https://aclanthology.org/2023.arabicnlp-1.21/

MT-Bench marks the introduction of multi-turn prompting as well as llm-as-a-judge metric.

New tasks
* Add BBH by clefourrier in https://github.com/huggingface/lighteval/pull/7, bilgehanertan in https://github.com/huggingface/lighteval/pull/126
* Add AGIEval by clefourrier in https://github.com/huggingface/lighteval/pull/121
* Adding TinyBench by clefourrier in https://github.com/huggingface/lighteval/pull/104
* Adding support for Arabic benchmarks : AlGhafa benchmarking suite by alielfilali01 in https://github.com/huggingface/lighteval/pull/95
* Add mt-bench by NathanHB in https://github.com/huggingface/lighteval/pull/75

Features
* Extended Tasks ! by clefourrier in https://github.com/huggingface/lighteval/pull/101, lewtun in https://github.com/huggingface/lighteval/pull/108, NathanHB in https://github.com/huggingface/lighteval/pull/122, https://github.com/huggingface/lighteval/pull/123
* Added support for launching inference endpoint with different model dtypes by shaltielshmid in https://github.com/huggingface/lighteval/pull/124

Documentation
* Adding LICENSE by clefourrier in https://github.com/huggingface/lighteval/pull/86, NathanHB in https://github.com/huggingface/lighteval/pull/89
* Make it clearer in the README that the leaderboard uses the harness by clefourrier in https://github.com/huggingface/lighteval/pull/94

Small patches
* Update huggingface-hub for compatibility with datasets 2.18 by clefourrier in https://github.com/huggingface/lighteval/pull/84
* Tidy up dependency groups by lewtun in https://github.com/huggingface/lighteval/pull/81
* bump git python by NathanHB in https://github.com/huggingface/lighteval/pull/90
* Sets a max length for the MATH task by clefourrier in https://github.com/huggingface/lighteval/pull/83
* Fix parallel data processing bug by clefourrier in https://github.com/huggingface/lighteval/pull/92
* Change the eos condition for GSM8K by clefourrier in https://github.com/huggingface/lighteval/pull/85
* Fixing rolling loglikelihood management by clefourrier in https://github.com/huggingface/lighteval/pull/78
* Fixes input length management for generative evals by clefourrier in https://github.com/huggingface/lighteval/pull/103
* Reorder addition of instruction in chat template by clefourrier in https://github.com/huggingface/lighteval/pull/111
* Ensure chat models terminate generation with EOS token by lewtun in https://github.com/huggingface/lighteval/pull/115
* Fix push details to hub by NathanHB in https://github.com/huggingface/lighteval/pull/98
* Small fixes to InferenceEndpointModel by shaltielshmid in https://github.com/huggingface/lighteval/pull/112
* Fix import typo autogptq by clefourrier in https://github.com/huggingface/lighteval/pull/116
* Fixed the loglikelihood method in inference endpoints models by clefourrier in https://github.com/huggingface/lighteval/pull/119
* Fix TextGenerationResponse import from hfh by Wauplin in https://github.com/huggingface/lighteval/pull/129
* Do not use deprecated list_files_info by Wauplin in https://github.com/huggingface/lighteval/pull/133
* Update test workflow name to 'Tests' by Wauplin in https://github.com/huggingface/lighteval/pull/134

New Contributors
* shaltielshmid made their first contribution in https://github.com/huggingface/lighteval/pull/112
* bilgehanertan made their first contribution in https://github.com/huggingface/lighteval/pull/126
* Wauplin made their first contribution in https://github.com/huggingface/lighteval/pull/129

**Full Changelog**: https://github.com/huggingface/lighteval/compare/v0.2.0...v0.3.0

Release Note

This release focuses on customization and personalisation: it's now possible to define custom metrics, not just custom tasks, see the README for the full mechanism.
Also includes small fixes to improve stability and new tasks. We made the choice to split community tasks from the main library source to better manage maintenance.

Better community task handling
* New mechanism for evaluation contributions by clefourrier in https://github.com/huggingface/lighteval/pull/47
* Adding the custom metrics system by clefourrier in https://github.com/huggingface/lighteval/pull/65

New tasks
* Add GPQA by clefourrier in https://github.com/huggingface/lighteval/pull/42
* Adding support for Arabic benchmarks : AceGPT benchmarking suite by alielfilali01 in https://github.com/huggingface/lighteval/pull/44
* IFEval by clefourrier in https://github.com/huggingface/lighteval/pull/48

Features
* Add an automatic system to compute average for tasks with subtasks by clefourrier in https://github.com/huggingface/lighteval/pull/41

small patches
* Typos https://github.com/huggingface/lighteval/pull/27, https://github.com/huggingface/lighteval/pull/28, https://github.com/huggingface/lighteval/pull/30, https://github.com/huggingface/lighteval/pull/29, https://github.com/huggingface/lighteval/pull/34,
* Better README https://github.com/huggingface/lighteval/pull/26, https://github.com/huggingface/lighteval/pull/37, https://github.com/huggingface/lighteval/pull/55,
* Patch fix to match with config update/simplification in nanotron by thomwolf in https://github.com/huggingface/lighteval/pull/35
* bump transformers to 4.38 by NathanHB in https://github.com/huggingface/lighteval/pull/46
* Small fix to be able to use extensions of nanotron configs by thomwolf in https://github.com/huggingface/lighteval/pull/58
* Remove the eos token override in the Default Config Task by clefourrier in https://github.com/huggingface/lighteval/pull/54
* Update leaderboard task set by lewtun in https://github.com/huggingface/lighteval/pull/60
* Remove the eos token override in the Default Config Task by clefourrier in https://github.com/huggingface/lighteval/pull/54
* Fixes wikitext prompts + some patches on tg models by clefourrier in https://github.com/huggingface/lighteval/pull/64
* Fix unset generation size by clefourrier in https://github.com/huggingface/lighteval/pull/76
* Update ruff by clefourrier in https://github.com/huggingface/lighteval/pull/71
* Relax sentencepiece version by lewtun in https://github.com/huggingface/lighteval/pull/74
* Better chat template system by clefourrier in https://github.com/huggingface/lighteval/pull/38

:sparkles: Community Contributions
* ledrui made their first contribution in https://github.com/huggingface/lighteval/pull/26
* alielfilali01 made their first contribution in https://github.com/huggingface/lighteval/pull/44
* lewtun made their first contribution in https://github.com/huggingface/lighteval/pull/55

**Full Changelog**: https://github.com/huggingface/lighteval/compare/v0.1.1...v0.2.0

Small patch for PyPi release

Include tasks_table.jsonl in package

Init

LightEval 🌤️
A lightweight LLM evaluation

Context
LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library [datatrove](https://github.com/huggingface/datatrove) and LLM training library [nanotron](https://github.com/huggingface/nanotron).

We're releasing it with the community in the spirit of building in the open.

Note that it is still very much early so don't expect 100% stability ^^'
In case of problems or question, feel free to open an issue!

**Full Changelog**: https://github.com/huggingface/lighteval/commits/v0.1