Ms-swift

2. Web-UI 支持音频和视频。
3. 支持使用 OpenAI API 部署音频和视频模型。
4. 采用新的多模态训练框架。
5. 支持视频模型的推理加速（lmdeploy 和 internvl2 系列）。

新模型：
1. idefics3-8b-llama3
2. llava-hf 72b、110b、llama3-llava
3. deepseek-coder-v2、deepseek-coder-lite-v2、deepseek-v2

What's Changed
* support video deploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1632
* fix infer deploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1665
* Fix bugs by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1664
* Add generation config support for swift eval by Yunnglin in https://github.com/modelscope/ms-swift/pull/1659
* Add a bunch of small features by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1669
* Support lmdeploy&vllm support video by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1668
* [TorchAcc] fix: add mark_step when GA is greater than 1 by baoleai in https://github.com/modelscope/ms-swift/pull/1677
* Support audio and video in web-ui by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1681
* Fix lmdeploy deploy seed by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1678
* add paper link by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1684
* set default writer_batch_size argument in mllm rlhf by hjh0119 in https://github.com/modelscope/ms-swift/pull/1689
* New multimodal framework by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1594
* fix video component by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1692
* refactor internvl2 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1625
* fix model dtype in rlhf by hjh0119 in https://github.com/modelscope/ms-swift/pull/1696
* Fix minicpmv 2.6 & video/plain-text bug by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1698
* Fix internvl2 device map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1701
* Support deepseek-v2 base by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1708
* [TorchAcc] update data bucketing strategy by baoleai in https://github.com/modelscope/ms-swift/pull/1706
* [TorchAcc] consolidate FSDP ckpt after saving instead of during loading by baoleai in https://github.com/modelscope/ms-swift/pull/1709
* fix deploy stream by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1713
* update docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1714
* Fix mllm rlhf by hjh0119 in https://github.com/modelscope/ms-swift/pull/1707
* support idefics3 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1720
* Fix minicpm-v-v2.6 device_map='auto' & agent deploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1733
* Fix kto by hjh0119 in https://github.com/modelscope/ms-swift/pull/1737
* support llava-hf 72b, 110b, llama3-llava-hf by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1741
* Fix glm4v rlhf by hjh0119 in https://github.com/modelscope/ms-swift/pull/1745

New Contributors
* Yunnglin made their first contribution in https://github.com/modelscope/ms-swift/pull/1659

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.3.0...v2.3.1

English Version

Models
1. Support Qwen2.5 coder models

Feature
1. Correct and support the new loss and gradient accumulation algorithm from transformers.trainer

中文版本
模型
1. 支持千问coder系列模型

功能
1. 支持新的transformers loss和GA计算算法，并修正了其中的bug

What's Changed
* fix gptq ovis quant (inputs_embeds) by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2378
* [TorchAcc] fix qwen2 for transformers>=4.45 by baoleai in https://github.com/modelscope/ms-swift/pull/2379
* fix trl transformers==4.46 compat by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2385
* fix deploy stream media_type by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2393
* fix_mplug_owl3_cut_shape by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2394
* fix swift deploy (lmdeploy stream) by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2397
* fix lmdeploy warning & gptq-int4 support by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2401
* support qwen2.5-coder by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2400
* fix qwen_vl npu by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2408
* fix model path by yingdachen in https://github.com/modelscope/ms-swift/pull/2410
* fix qwen_vl dpo by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2411
* Compat transformers 4.46.2 loss by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2413
* Fix molmo infer by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2419
* Fix dataset map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2421
* Support qwen2 5 coder series model by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2422
* fix PPO by hjh0119 in https://github.com/modelscope/ms-swift/pull/2377
* fix docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2425
* compat transformers==4.47 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2426
* fix internvl2 lmdeploy>=0.6 deploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2429


**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.5.2...v2.6.0

New Models:
1. emu3-chat
2. aya-expanse
3. ministral-8b-inst-2410

New Datasets:
1. llava-video-178k
2. moviechat-1k-test

What's Changed
* fix system by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2307
* Support Llava-video-178k dataset, a video dataset by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2309
* update molmo readme by mi804 in https://github.com/modelscope/ms-swift/pull/2313
* fix molmo device_map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2311
* support emu3-chat by mi804 in https://github.com/modelscope/ms-swift/pull/2322
* Support Moviechat-1k-test dataset, a video datatset by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2319
* fix custom vllm eval args by Yunnglin in https://github.com/modelscope/ms-swift/pull/2325
* fix enable_cache by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2332
* feat: Add aya models by Aunali321 in https://github.com/modelscope/ms-swift/pull/2335
* compat transformers 4.46 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2341
* fix streaming by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2350
* fix doc by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2355
* fix internvl2 eval by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2362
* fix transformers>=4.46 loss by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2365
* Support Ministral-8b-inst-2410 by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2371
* fix custom loss by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2374

New Contributors
* Aunali321 made their first contribution in https://github.com/modelscope/ms-swift/pull/2335

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.5.1...v2.5.2

English Version

New Features:
1. Support for RM for LLM and MLLM, as well as PPO for LLM.

New Models:
1. molmo series
2. mplug-owl3 1b/2b
3. llama3.1-nemotron-70b-instruct
4. deepseek-janus

中文版

新特性：
1. 支持LLM和MLLM的RM, 以及LLM的PPO.

新模型：
1. molmo系列
2. mplug-owl3 1b/2b
3. llama3.1-nemotron-70b-instruct
4. deepseek-janus

What's Changed
* support reward modeling and ppo by hjh0119 in https://github.com/modelscope/ms-swift/pull/2093
* fix rescale_image by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2223
* fix deploy timeout by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2230
* Fix qwen2 vl batch size by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2239
* Fix ovis1.6 infer by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2242
* fix publish by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2245
* fix qwen2vl video args by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2251
* Update FAQ by slin000111 in https://github.com/modelscope/ms-swift/pull/2252
* Support molmo series vlm by mi804 in https://github.com/modelscope/ms-swift/pull/2260
* fix sft system by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2262
* support mplug3 1b/2b by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2271
* Fix deploy openai by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2278
* fix vllm ignore suffix by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2287
* fix lora_target_modules in PPO by hjh0119 in https://github.com/modelscope/ms-swift/pull/2274
* fix quant blocks by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2292
* Support Llama3.1-nemotron-70b-inst-hf by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2299
* fix ppo citest by hjh0119 in https://github.com/modelscope/ms-swift/pull/2302
* support deepseek-janus by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2300
* update molmo by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2305

New Contributors
* mi804 made their first contribution in https://github.com/modelscope/ms-swift/pull/2260

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.5.0...v2.5.1

English Version

New Features:
1. Support for GPTQ & AWQ quantization of multimodal LLMs.
2. Support for dynamic addition of gradient checkpointing in the ViT section to reduce memory consumption.
3. Support for multimodal model pre-training.

New Models:
1. llama3.2, llama3.2-vision series
2. got-ocr2
3. llama3.1-omni
5. ovis1.6-gemma2
6. pixtral-12b
7. telechat2-115b
8. mistral-small-inst-2409

New Datasets:
1. egoschema

中文版

新特性：
1. 支持多模态LLM的gptq&awq量化.
2. 支持动态在vit部分增加gradient_checkpointing, 减少显存消耗.
3. 支持多模态模型预训练.

新模型：
1. llama3.2, llama3.2-vision系列
2. got-ocr2
3. llama3.1-omni
4. ovis1.6-gemma2
5. pixtral-12b
6. telechat2-115b
7. mistral-small-inst-2409

新数据集：
1. egoschema

What's Changed
* fix win32 quote by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2065
* Fix yi template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2067
* fix rlhf zero3 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2072
* Update qwen2-vl最佳实践.md by Digital2Slave in https://github.com/modelscope/ms-swift/pull/2058
* fix RLHF & max_length by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2075
* Support Mistral-small-inst-2409 by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2077
* dynamic vit gradient_checkpointing by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2071
* fix qwen2.5 template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2081
* fix multiprocess remove_columns by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2088
* Support for fine-tuning Pixtral-12B. by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2090
* fix vllm tokenizer by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2099
* Fix the issue with media_offset in owl3 when batch_size > 1. by LukeForeverYoung in https://github.com/modelscope/ms-swift/pull/2100
* fix deploy openai compat by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2101
* fix dataset preprocess by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2102
* fix cpu infer device_map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2103
* fix infer device_map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2105
* Support for fine-tuning Llama 3.1 Omni. by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2106
* support vllm & qwen2-vl video by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2110
* Fix qwen2-vl zero2/3 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2114
* fix qwen2-audio by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2116
* [TorchAcc] fix: fix find_labels and can_return_loss by baoleai in https://github.com/modelscope/ms-swift/pull/2120
* support got-ocr2 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2123
* Support for fine-tuning and deployment of the Llama 3.2 series models. by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2130
* Support fine-tuning MLLama. by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2132
* fix not impl bug by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2134
* Compat vllm & qwen2-vl by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2136
* fix requirements by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2137
* fix model_type by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2138
* fix deploy vllm by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2141
* fix docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2142
* Fix VLM lora by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2140
* support mllm pt by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2146
* [TorchAcc] fix: fix save config and additional file for swift and peft by baoleai in https://github.com/modelscope/ms-swift/pull/2149
* update quant_device_map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2154
* fix qwen2-audio by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2157
* fix template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2160
* compat trl==0.11 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2169
* Support for Egoschema, a new video dataset by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2173
* Update FAQ by slin000111 in https://github.com/modelscope/ms-swift/pull/2165
* fix mplug-owl3 infer by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2175
* Support quant mllm by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2177
* update setup.py by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2205
* fix bugs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2207
* support telechat2 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2210
* Support ovis 1.6 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2211

New Contributors
* Digital2Slave made their first contribution in https://github.com/modelscope/ms-swift/pull/2058
* LukeForeverYoung made their first contribution in https://github.com/modelscope/ms-swift/pull/2100

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.4.2...v2.5.0

English Version

New Features:
1. RLHF reconstruction, supporting all integrated multimodal models, compatible with DeepSpeed Zero2/Zero3, and supports lazy_tokenize.
2. Using infer_backend vllm, inference deployment of multimodal large models supports multiple images.

New Models:
1. Qwen2.5 series, Qwen2-vl-72b series (base/instruct/gptq-int4/gptq-int8/awq)
3. Qwen2.5-math, Qwen2.5-coder series (base/instruct)
4. Deepseek-v2.5

New Datasets:
1. longwriter-6k-filtered

中文版

新特性：
1. RLHF重构，支持所有已接入的多模态模型，兼容deepspeed zero2/zero3，支持lazy_tokenize
2. 使用infer_backend vllm，推理部署多模态大模型支持多图.

新模型：
1. qwen2.5系列、qwen2-vl-72b系列（base/instruct/gptq-int4/gptq-int8/awq）
3. qwen2.5-math, qwen2.5-coder系列（base/instruct）
5. deepseek-v2.5

新数据集：
1. longwriter-6k-filtered

What's Changed
* fix model_mapping by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1982
* fix patch by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1997
* fix by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1995
* Support Deepseek 2.5 by DaozeZhang in https://github.com/modelscope/ms-swift/pull/1992
* fix EngineGenerationConfig importError of lmdeploy by irexyc in https://github.com/modelscope/ms-swift/pull/1990
* compat lmdeploy==0.6 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2001
* Fix rlhf ref model by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2003
* Support llava1.6-llama3.1-8b-instruct by DaozeZhang in https://github.com/modelscope/ms-swift/pull/2005
* fix lmdeploy qwen_vl by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2009
* Add FAQ Document by slin000111 in https://github.com/modelscope/ms-swift/pull/2013
* Florence use _post_encode & template support encoder-decoder by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2019
* refactor rlhf by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1975
* update code by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2028
* fix deploy eval kill by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2029
* Fix olora and pissa saving files which will cause the second saving failed by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2032
* fix rlhf & zero3 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2034
* Add longwriter filtered dataset by wangxingjun778 in https://github.com/modelscope/ms-swift/pull/2037
* fix mplug-owl3 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2042
* support multi bbox grounding by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2045
* Fix multi coordinate grounding by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2047
* llama3 tool calling by tastelikefeet in https://github.com/modelscope/ms-swift/pull/2048
* update docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2050
* fix qwen2vl position_ids by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2051
* support qwen2-vl-base by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2052
* Support qwen2.5 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2054
* support qwen2-vl -72b/qwen2.5-math/qwen2.5-coder by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2056
* vllm support mutli image by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2059
* support qwen2.5-coder by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2061
* fix notebook gradio by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2062
* update qwen2-vl docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2063

New Contributors
* irexyc made their first contribution in https://github.com/modelscope/ms-swift/pull/1990
* wangxingjun778 made their first contribution in https://github.com/modelscope/ms-swift/pull/2037

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.4.1...v2.4.2