Lighteval

Release Note

This release focuses on customization and personalisation: it's now possible to define custom metrics, not just custom tasks, see the README for the full mechanism.
Also includes small fixes to improve stability and new tasks. We made the choice to split community tasks from the main library source to better manage maintenance.

Better community task handling
* New mechanism for evaluation contributions by clefourrier in https://github.com/huggingface/lighteval/pull/47
* Adding the custom metrics system by clefourrier in https://github.com/huggingface/lighteval/pull/65

New tasks
* Add GPQA by clefourrier in https://github.com/huggingface/lighteval/pull/42
* Adding support for Arabic benchmarks : AceGPT benchmarking suite by alielfilali01 in https://github.com/huggingface/lighteval/pull/44
* IFEval by clefourrier in https://github.com/huggingface/lighteval/pull/48

Features
* Add an automatic system to compute average for tasks with subtasks by clefourrier in https://github.com/huggingface/lighteval/pull/41

small patches
* Typos https://github.com/huggingface/lighteval/pull/27, https://github.com/huggingface/lighteval/pull/28, https://github.com/huggingface/lighteval/pull/30, https://github.com/huggingface/lighteval/pull/29, https://github.com/huggingface/lighteval/pull/34,
* Better README https://github.com/huggingface/lighteval/pull/26, https://github.com/huggingface/lighteval/pull/37, https://github.com/huggingface/lighteval/pull/55,
* Patch fix to match with config update/simplification in nanotron by thomwolf in https://github.com/huggingface/lighteval/pull/35
* bump transformers to 4.38 by NathanHB in https://github.com/huggingface/lighteval/pull/46
* Small fix to be able to use extensions of nanotron configs by thomwolf in https://github.com/huggingface/lighteval/pull/58
* Remove the eos token override in the Default Config Task by clefourrier in https://github.com/huggingface/lighteval/pull/54
* Update leaderboard task set by lewtun in https://github.com/huggingface/lighteval/pull/60
* Remove the eos token override in the Default Config Task by clefourrier in https://github.com/huggingface/lighteval/pull/54
* Fixes wikitext prompts + some patches on tg models by clefourrier in https://github.com/huggingface/lighteval/pull/64
* Fix unset generation size by clefourrier in https://github.com/huggingface/lighteval/pull/76
* Update ruff by clefourrier in https://github.com/huggingface/lighteval/pull/71
* Relax sentencepiece version by lewtun in https://github.com/huggingface/lighteval/pull/74
* Better chat template system by clefourrier in https://github.com/huggingface/lighteval/pull/38

:sparkles: Community Contributions
* ledrui made their first contribution in https://github.com/huggingface/lighteval/pull/26
* alielfilali01 made their first contribution in https://github.com/huggingface/lighteval/pull/44
* lewtun made their first contribution in https://github.com/huggingface/lighteval/pull/55

**Full Changelog**: https://github.com/huggingface/lighteval/compare/v0.1.1...v0.2.0

Small patch for PyPi release

Include tasks_table.jsonl in package

Init

LightEval 🌤️
A lightweight LLM evaluation

Context
LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library [datatrove](https://github.com/huggingface/datatrove) and LLM training library [nanotron](https://github.com/huggingface/nanotron).

We're releasing it with the community in the spirit of building in the open.

Note that it is still very much early so don't expect 100% stability ^^'
In case of problems or question, feel free to open an issue!

**Full Changelog**: https://github.com/huggingface/lighteval/commits/v0.1