Internlm

We're thrilled to announce OpenCompass v0.2.1, loaded with new datasets, features, and vital fixes. This release is a testament to our ongoing commitment to enhancing user experience and broadening research capabilities.

🌟 **Highlights**:
- **Add Agent and Code datasets**: Diverse new datasets like `GPQA`, `mastermath2024v1`, and more, significantly expanding the scope of OpenCompass.
- **Support Different JudgeLLM Subjective Evaluation**: Providing more choice when choose judgellms.
- **Support Needle in Haystack**: Support Needle in Haystack for longtext evaluation.
- **Add VLLM Evaluation**: We support VLLM inference and evaluation.

Here's what's new:

🚀 New Features:
- 📦 **Dataset Expansion**:
- Added `rwkv-5-3b` model ([666](https://github.com/open-compass/opencompass/pull/666))
- Integration of diverse datasets including `GPQA`, `Creationbench`, and more.
- Support for new datasets like `mastermath2024v1`, `mbpp_plus`, and `sanitized_mbpp` ([744](https://github.com/open-compass/opencompass/pull/744), [#770](https://github.com/open-compass/opencompass/pull/770), [#745](https://github.com/open-compass/opencompass/pull/745))

- 🛠 **Functional Enhancements**:
- Subjective evaluation improvements ([692](https://github.com/open-compass/opencompass/pull/692), [#724](https://github.com/open-compass/opencompass/pull/724))
- Updated python action, slurm, and docker docs ([694](https://github.com/open-compass/opencompass/pull/694), [#718](https://github.com/open-compass/opencompass/pull/718))
- Turbomind API support and Qwen API integration ([693](https://github.com/open-compass/opencompass/pull/693), [#735](https://github.com/open-compass/opencompass/pull/735))

- 📖 **Documentation Updates**:
- Updated contamination, alignmentbench, and other docs for better clarity ([698](https://github.com/open-compass/opencompass/pull/698), [#707](https://github.com/open-compass/opencompass/pull/707))
- Fixed dead links and typos in various documents ([455](https://github.com/open-compass/opencompass/pull/455), [#773](https://github.com/open-compass/opencompass/pull/773), [#774](https://github.com/open-compass/opencompass/pull/774))

🐛 Bug Fixes:
- Addressed various issues including those in alignmentbench, configs, and postprocess scripts.
- Fixed bugs concerning subjective evaluation and EOS string detection.
- Quick fixes for improved performance and reliability.

🎉 Welcome New Contributors:
- A warm welcome to our first-time contributors:
- BBuf, DseidLi, Skyfall-xzz, RunningLeon, zehuichen123, AllentDan, Connor-Shen, Francis-llgg, hzhwcmhf, ChrisLiu6, yanyc428, tpoisonooo, jiangjin1999

🔗 Full Changelog
* add rwkv-5-3b model by BBuf in https://github.com/open-compass/opencompass/pull/666
* [Feature] Add double order of subjective evaluation and removing duplicated response among two models by bittersweet1999 in https://github.com/open-compass/opencompass/pull/692
* [Feat] update python action and slurm by yingfhu in https://github.com/open-compass/opencompass/pull/694
* [Doc] Update contamination docs by Leymore in https://github.com/open-compass/opencompass/pull/698
* alignmentbench infer and judge by bittersweet1999 in https://github.com/open-compass/opencompass/pull/697
* [Fix] Update alignmentbench by tonysy in https://github.com/open-compass/opencompass/pull/704
* removed redundant code in GSM8KDataset.load method. by DseidLi in https://github.com/open-compass/opencompass/pull/700
* [Fix] fix a bug on configs/eval_mixtral_8x7b.py by jingmingzhuo in https://github.com/open-compass/opencompass/pull/706
* [Doc] Update Doc for Alignbench by tonysy in https://github.com/open-compass/opencompass/pull/707
* [Fix] minor fix openai by yingfhu in https://github.com/open-compass/opencompass/pull/711
* Add Judgellms by bittersweet1999 in https://github.com/open-compass/opencompass/pull/710
* [Feat] Update math/agent by yingfhu in https://github.com/open-compass/opencompass/pull/716
* [Docs] update docker docs by yingfhu in https://github.com/open-compass/opencompass/pull/718
* [Fix] Quick fix for max_out_len in subjective evaluation by bittersweet1999 in https://github.com/open-compass/opencompass/pull/719
* [Feature] Support the use of humaneval_plus. by jingmingzhuo in https://github.com/open-compass/opencompass/pull/720
* [Feature] Add reasonbench dataset by Skyfall-xzz in https://github.com/open-compass/opencompass/pull/577
* [Feature] Add abbr for judgemodel in subjective evaluation by bittersweet1999 in https://github.com/open-compass/opencompass/pull/724
* Update configs for evaluating chat models like qwen, baichuan, llama2 using turbomind backend by RunningLeon in https://github.com/open-compass/opencompass/pull/721
* [News] add news for T-Eval by zehuichen123 in https://github.com/open-compass/opencompass/pull/727
* Add NeedleInAHaystack Test Support by DseidLi in https://github.com/open-compass/opencompass/pull/714
* [Fix] Fixed abbr erro of subjective alignbench and size partition by bittersweet1999 in https://github.com/open-compass/opencompass/pull/730
* add turbomind restful api support by AllentDan in https://github.com/open-compass/opencompass/pull/693
* [Fix] Update merge script for non-split settting by tonysy in https://github.com/open-compass/opencompass/pull/733
* [Sync] Sync with internal codes by Leymore in https://github.com/open-compass/opencompass/pull/734
* [Feature] Add InfiniteBench by philipwangOvO in https://github.com/open-compass/opencompass/pull/739
* Update LightllmApi and Fix mmlu bug by helloyongyang in https://github.com/open-compass/opencompass/pull/738
* [Feature] Add other judgelm prompts for Alignbench by bittersweet1999 in https://github.com/open-compass/opencompass/pull/731
* [Feat] support sanitized mbpp dataset by yingfhu in https://github.com/open-compass/opencompass/pull/745
* [Fix] SubSizePartition fix by bittersweet1999 in https://github.com/open-compass/opencompass/pull/746
* add chinese version of humaneval, mbpp by Connor-Shen in https://github.com/open-compass/opencompass/pull/743
* [Fix] fix erro in configs by bittersweet1999 in https://github.com/open-compass/opencompass/pull/750
* [Feature] Add Creationbench Dataset by bittersweet1999 in https://github.com/open-compass/opencompass/pull/753
* [Feat] update code config by yingfhu in https://github.com/open-compass/opencompass/pull/749
* update plot function in tools_needleinahaystack.py by DseidLi in https://github.com/open-compass/opencompass/pull/747
* [Feature] Add new dataset mastermath2024v1 by Francis-llgg in https://github.com/open-compass/opencompass/pull/744
* [Feature] Add GPQA Dataset by Francis-llgg in https://github.com/open-compass/opencompass/pull/729
* change NeedleInAHaystackDataset to dynamic loading by DseidLi in https://github.com/open-compass/opencompass/pull/754
* [Feature] Add support of Qwen API by hzhwcmhf in https://github.com/open-compass/opencompass/pull/735
* [Feature] Support LLaMA2-Accessory by ChrisLiu6 in https://github.com/open-compass/opencompass/pull/732
* [Fix] Fix small bug in alignbench by bittersweet1999 in https://github.com/open-compass/opencompass/pull/764
* [Feature] Add multi_round dataset evaluation by bittersweet1999 in https://github.com/open-compass/opencompass/pull/766
* [Feature] add subject ir dataset by bittersweet1999 in https://github.com/open-compass/opencompass/pull/755
* [Update] Update introduction of CompassBench-2024-Q1 by tonysy in https://github.com/open-compass/opencompass/pull/769
* [Fix] quick fix for postprocess by bittersweet1999 in https://github.com/open-compass/opencompass/pull/771
* Support Mbpp_plus dataset by Connor-Shen in https://github.com/open-compass/opencompass/pull/770
* [Fix] fix typos in drop prompt by yanyc428 in https://github.com/open-compass/opencompass/pull/773
* typo(installation.md): fix unzip commands by tpoisonooo in https://github.com/open-compass/opencompass/pull/774
* Contamination analysis for MMLU, Hellaswag, and ARC_c by liyucheng09 in https://github.com/open-compass/opencompass/pull/699
* [Docs] Update contamination docs by Leymore in https://github.com/open-compass/opencompass/pull/775
* [Feature] *_batch_generate* function, add the MultiTokenEOSCriteria by jiangjin1999 in https://github.com/open-compass/opencompass/pull/772
* [Sync] Sync with internal codes 2023.01.08 by Leymore in https://github.com/open-compass/opencompass/pull/777

**For a full list of updates, visit our** [Full Changelog](https://github.com/open-compass/opencompass/compare/0.2.0...0.2.1).

Thank you to every contributor, old and new. Your dedication is shaping OpenCompass into a more robust and versatile tool. 🙌 🎉

---

Remember to star 🌟 our GitHub repository if OpenCompass aids your research and development! Your support and feedback are crucial for our continuous improvement.

What's Changed
* fix(timeout): larger timeout by JiaoPL in https://github.com/InternLM/InternLM/pull/495
* feat(doc): add GPU memory info for 7B & 20B models by li126com in https://github.com/InternLM/InternLM/pull/507
* feat(model): add rope_base interface by 00INDEX in https://github.com/InternLM/InternLM/pull/512
* Feat(QA): Check loss when swapping micro_num and micro_bsz && Check grad norm by li126com in https://github.com/InternLM/InternLM/pull/510
* Fix(QA): the py name in main is wrong by li126com in https://github.com/InternLM/InternLM/pull/514
* fix/feat: small fix and enhancement by SolenoidWGT in https://github.com/InternLM/InternLM/pull/515
* test(workflow): add workflow for loss test and change trigger event by kkscilife in https://github.com/InternLM/InternLM/pull/513
* fix(ci): fix test model ckpt ci test by SolenoidWGT in https://github.com/InternLM/InternLM/pull/518
* test(workflow): add unit test case by kkscilife in https://github.com/InternLM/InternLM/pull/524
* feat(storage): use multipart upload when using oss by li126com in https://github.com/InternLM/InternLM/pull/520
* Fix (QA checkpoint): fix test_model_checkpoint singleton import by li126com in https://github.com/InternLM/InternLM/pull/526
* fix(model): add IS_SEQUENCE_PARALLEL check for norm module by yingtongxiong in https://github.com/InternLM/InternLM/pull/528
* feat(model): add output embedding tf32 option by JiaoPL in https://github.com/InternLM/InternLM/pull/523
* feat(grad_norm): vocab grad norm profiling by JiaoPL in https://github.com/InternLM/InternLM/pull/519
* fix(data): fix the unpack for type_ids when use_flash_attn=False by yingtongxiong in https://github.com/InternLM/InternLM/pull/516
* fix(storage): unify the name of AK and SK by li126com in https://github.com/InternLM/InternLM/pull/527
* fix(test): fix type_ids unpack bug by SolenoidWGT in https://github.com/InternLM/InternLM/pull/530
* feat(model): support llama model with checkpoint loading by li126com in https://github.com/InternLM/InternLM/pull/532
* fix(metric): add metric dtype control by Pryest in https://github.com/InternLM/InternLM/pull/533
* feat(ckpt): support auto resume in Volc and Ali by li126com in https://github.com/InternLM/InternLM/pull/529
* fix(sequence_parallel): fix norm all-reduce in seq_parallel when not overlaping by yingtongxiong in https://github.com/InternLM/InternLM/pull/534
* fix(pp): fix no-packed dataset load micro batch error by SolenoidWGT in https://github.com/InternLM/InternLM/pull/538
* fix(model): change model_type `LLAMA` to `LLAMA2` by li126com in https://github.com/InternLM/InternLM/pull/539
* fix(moe): fix moe zero mode bug by blankde in https://github.com/InternLM/InternLM/pull/548
* fix(grad_norm): token grad norm with tp by JiaoPL in https://github.com/InternLM/InternLM/pull/547
* test(workflow): change into reserved by kkscilife in https://github.com/InternLM/InternLM/pull/550
* fix(model): add ckpt_type constraint when loading ckpts by li126com in https://github.com/InternLM/InternLM/pull/542
* feat(logger): add tensorboard key value buffer by SolenoidWGT in https://github.com/InternLM/InternLM/pull/549
* fix(metrics): remove redundant cuda memory in metric calculations by SolenoidWGT in https://github.com/InternLM/InternLM/pull/557
* fix(lr_scheduler): fix when resuming lr_scheduler without loading optimizer by gaoyang07 in https://github.com/InternLM/InternLM/pull/565


**Full Changelog**: https://github.com/InternLM/InternLM/compare/v0.2.1dev20231121...v0.2.1dev20240102

TBD

Highlights
* fix the bug that may have grad overflow when total_steps is small
* fix the rotary_emb.inv_freq KeyError in tool convert2hf.py
* add unit test for model

What's Changed

🚀 Features
* feat(core/trainer.py): add more tgs metrics by li126com in https://github.com/InternLM/InternLM/pull/310

🐞 Bug fixes
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError by jiangtann in https://github.com/InternLM/InternLM/pull/299
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 by huangting4201 in https://github.com/InternLM/InternLM/pull/302
* fix(chat): fix stream_chat to return generator by zhjunqin in https://github.com/InternLM/InternLM/pull/123

📚 Documentations
* docs(doc/code-docs): update quickstart usage by huangting4201 in https://github.com/InternLM/InternLM/pull/301
* docs(doc/code-docs): add figure for training docs by zigzagcai in https://github.com/InternLM/InternLM/pull/307

✅ Tests
* tests(tests/test_model): add unit test for model by li126com in https://github.com/InternLM/InternLM/pull/300
* tests(tests/test_solver): add unit test for optimizer by li126com in https://github.com/InternLM/InternLM/pull/303

🌐 Other

Known issues

**Full Changelog**: https://github.com/InternLM/InternLM/compare/v0.2.1dev20230909...v0.2.1dev20230915

What's Changed
* fix(ckpt): fix snapshot none load error and remove file lock by SolenoidWGT in https://github.com/InternLM/InternLM/pull/298


**Full Changelog**: https://github.com/InternLM/InternLM/compare/v0.2.1dev20230908...v0.2.1dev20230909

Highlights
* fix the bug that may have NaN value when overlap gradients' allreduce with backward
* support timeout wrapper and runtime diagnosis
* support readthedocs Chinese version

What's Changed

🚀 Features
* feat(monitor): add light monitor by JiaoPL in https://github.com/InternLM/InternLM/pull/275
* feat(utils): add timeout wrapper by SolenoidWGT in https://github.com/InternLM/InternLM/pull/286
* feat: add runtime diagnosis by sunpengsdu in https://github.com/InternLM/InternLM/pull/297

💥 Improvements
* fix(storage): refactor and fix storage_manager api by SolenoidWGT in https://github.com/InternLM/InternLM/pull/281
* Feat/sync grad use async op by sunpengsdu in https://github.com/InternLM/InternLM/pull/277

🐞 Bug fixes
* fix(doc/code-docs): autodoc shown error by huangting4201 in https://github.com/InternLM/InternLM/pull/265
* fix(eval): no need to check length of valid_dl when using streaming dataset by 00INDEX in https://github.com/InternLM/InternLM/pull/274
* fix/broadcast should not in commu stream by sunpengsdu in https://github.com/InternLM/InternLM/pull/276
* fix(model): set tensor parallel attribute for mlp by yingtongxiong in https://github.com/InternLM/InternLM/pull/271
* feat(ckpt): checkpoint bug fixes and feature enhancements. by SolenoidWGT in https://github.com/InternLM/InternLM/pull/259
* fix(ckpt): fix checkpoint reload bug by SolenoidWGT in https://github.com/InternLM/InternLM/pull/282
* fix(core/context): use dummy mode to generate random numbers in model construction by blankde in https://github.com/InternLM/InternLM/pull/266
* fix(monitor): add alert switch and refactor monitor config by JiaoPL in https://github.com/InternLM/InternLM/pull/285
* fix: fix the bug to do bcast in a stream by sunpengsdu in https://github.com/InternLM/InternLM/pull/294

📚 Documentations
* docs(*): add documentation and reST files for readthedocs by zigzagcai in https://github.com/InternLM/InternLM/pull/272
* docs(doc/code-docs): support zh cn readthedocs by huangting4201 in https://github.com/InternLM/InternLM/pull/289
* docs(fsdp): add training option for fsdp by zaglc in https://github.com/InternLM/InternLM/pull/273
* docs(doc/code-docs): refine profiler docs by zigzagcai in https://github.com/InternLM/InternLM/pull/295

🌐 Other

Known issues

New Contributors
* JiaoPL made their first contribution in https://github.com/InternLM/InternLM/pull/275
* blankde made their first contribution in https://github.com/InternLM/InternLM/pull/266
* zigzagcai made their first contribution in https://github.com/InternLM/InternLM/pull/272
* zaglc made their first contribution in https://github.com/InternLM/InternLM/pull/273

**Full Changelog**: https://github.com/InternLM/InternLM/compare/v0.2.1dev20230901...v0.2.1dev20230908