Ms-swift

English Version
New Features:

English Version

New Features
1. Support for readthedocs documentation site at: https://swift.readthedocs.io/en/latest
2. Support Megatron architecture training for QianWen series models, and added new `pt` command for pretraining. See docs: https://swift.readthedocs.io/en/latest/LLM/Megatron-training.html
3. Support LMDeploy for inference and deployment, improving inference acceleration for multi-modal models. See: https://swift.readthedocs.io/en/latest/Multi-Modal/LmDeploy-inference-acceleration.html
4. Support passing lora target modules via regular expressions
5. Support configuring max_memory usage for each GPU in device_map
6. `export` command supports BitsAndBytes quantization
7. `export` command supports Ollama export: https://swift.readthedocs.io/en/latest/LLM/OLLaMA-Export.html
8. Support Q-GaLore algorithm
9. Support RLHF training for multi-modal models: https://swift.readthedocs.io/en/latest/Multi-Modal/human-preference-alignment-training-documentation.html
10. Support evaluation on 100+ datasets for multi-modal models: https://swift.readthedocs.io/en/latest/LLM/LLM-eval.html
11. Support resizing input images when memory usage is too high for multi-modal models
12. Modified default lora injection for multi-modal model training. Now takes effect on LLM and projector, results are better without significantly increasing training memory.
13. Support PEFT 0.12, and added new tuner: fourierft
14. Support rope-scaling for multi-modal models
15. Support streaming processing of datasets to reduce memory usage, enable with `--streaming`
16. Support vLLM multi-modal inference and deployment
17. Support grounding task for popular multi-modal models.

New Models
1. qwen2-audio series
2. qwen2-math
4. codegeex4
5. internvl2 series
6. llava video
7. xcomposer2.5
8. cogvlm2-video
9. numina-math
10. mistral-nemo
11. llama3.1 series
12. mistral-large
13. gemma-2-2b
14. internlm2.5 1.8b 20b
15. minicpm-v-v2_6-chat

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

New Datasets
1. zhihu-kol and zhihu-kol-filtered
2. SA1B series multi-modal zh datasets

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

中文版本

新功能
1. 支持readthedocs文档库， 地址：https://swift.readthedocs.io/zh-cn/latest
2. 支持千问系列模型的Megatron结构训练，并支持了新的pt命令用于预训练，详见文档：https://swift.readthedocs.io/zh-cn/latest/LLM/Megatron%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
3. 支持LMDeploy的推理和部署，更好地支持了多模态模型的推理加速，详见：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/LmDeploy%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E6%96%87%E6%A1%A3.html
4. 支持以正则表达式方式传入lora target模块
5. 支持配置device_map各GPU用量的max_memory
6. export命令支持BitsAndBytes量化
7. export命令支持Ollama导出：https://swift.readthedocs.io/zh-cn/latest/LLM/OLLAMA%E5%AF%BC%E5%87%BA%E6%96%87%E6%A1%A3.html
8. 支持Q-GaLore算法
9. 支持多模态模型的RLHF训练：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/%E4%BA%BA%E7%B1%BB%E5%81%8F%E5%A5%BD%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
10. 支持多模态模型100+数据集的评测能力：https://swift.readthedocs.io/zh-cn/latest/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.html
11. 支持多模态模型显存占用过高时对输入图片进行缩放
12. 修改了多模态模型训练的默认lora注入，目前对LLM和projector生效，不显著提高训练显存情况下效果更好
13. 支持PEFT0.12，并支持了新的tuner：fourierft
14. 支持多模态模型的rope-scaling
15. 支持数据集的流式处理，降低显存消耗，使用--streaming开启
16. 支持了vLLM的多模态推理部署能力
17. 对部分多模态模型支持了grounding任务

新模型
1. qwen2-audio系列模型
22. qwen2-math
23. codegeex4
24. internvl2系列模型
25. llava video
26. xcomposer2.5
27. cogvlm2-video
28. numina-math
29. mistral-nemo
30. llama3.1系列
31. mistral-large
32. gemma-2-2b
33. internlm2.5 1.8b 20b
34. minicpm-v-v2_6-chat

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

新数据集
1. zhihu-kol和zhihu-kol-filtered数据集
3. SA1B系列中文多模态数据集

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

What's Changed
* fix dependency by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1306
* support codegeex4 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1305
* support internvl2 by hjh0119 in https://github.com/modelscope/ms-swift/pull/1304
* support llava video by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1307
* fix docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1309
* support lr_scheduler_kwargs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1310
* Fix internvl2 template by hjh0119 in https://github.com/modelscope/ms-swift/pull/1308
* Fix bugs by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1311
* support warmup_stable_decay by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1312
* Support xcomposer2.5 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1287
* Fix bugs by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1319
* fix bug by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1320
* fix template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1321
* support cogvlm2-video by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1318
* Fix bugs by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1325
* fix web-ui by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1327
* compatible with trl 0.9.6 by hjh0119 in https://github.com/modelscope/ms-swift/pull/1326
* compat with vllm==0.5.1 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1329
* Update qrcode by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1332
* fix florence model by hjh0119 in https://github.com/modelscope/ms-swift/pull/1334
* Relaxing requirements for trl by hjh0119 in https://github.com/modelscope/ms-swift/pull/1342
* fix xcomposer2.5 device_map by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1343
* support generation_info by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1344
* fix requirements by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1347
* readthedocs by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1345
* fix sequence parallel get labels by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1352
* fix filelock by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1354
* Add pt command by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1356
* fix generation_info efficiency by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1359
* fix sh ddp_backend by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1360
* support LLM & lmdeploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1272
* fix a file path by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1363
* Internvl2 support video by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1366
* fix openai api by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1367
* fix internvl2-40b by hjh0119 in https://github.com/modelscope/ms-swift/pull/1369
* fix vlm deploy lora & agent by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1371
* Support lora regex by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1375
* Fix docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1380
* Fix FSDP; Add training percentage to jsonl logging; Add a web-ui component by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1381
* Support max memory args by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1382
* fix max_memory by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1383
* Fix gpu assert calculation by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1384
* fix dataset_sample & deploy stop_words by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1385
* fix internvl doc by hjh0119 in https://github.com/modelscope/ms-swift/pull/1394
* Fix link by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1397
* fix vllm==0.5.1 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1404
* [TorchAcc] update accelerate API and add llama3-70B by baoleai in https://github.com/modelscope/ms-swift/pull/1400
* Support Ollama and BNB for export by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1407
* Fix glm4v merge lora by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1410
* [TorchAcc] fix model download when using TorchAcc distributed training by baoleai in https://github.com/modelscope/ms-swift/pull/1408
* Support padding left by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1414
* Fix ollama export by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1416
* fix web-ui params by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1417
* fix hub_token by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1420
* Update ms hub token by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1424
* Add numina math model by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1421
* fix internvl template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1433
* Internvl series models update by hjh0119 in https://github.com/modelscope/ms-swift/pull/1426
* fix internvl2 template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1436
* Fix bug and make lazydataset more stable by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1438
* Fix llava-hf by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1439
* [WIP]Support Q-Galore by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1440
* 1. support deepspeed on ui 2. add tools to client_utils by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1446
* fix read csv (float) by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1447
* fix dataset by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1448
* update internvl doc by hjh0119 in https://github.com/modelscope/ms-swift/pull/1449
* Support api key by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1452
* Support mistral nemo series models by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1454
* fix minicpm-v2.5 lora_target_modules by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1455
* Add two datasets by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1459
* Update trl dependency version by hjh0119 in https://github.com/modelscope/ms-swift/pull/1463
* fix bugs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1464
* fix yi1.5 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1465
* Fix yi1.5 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1467
* add activate and deactivate for part tuner by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1470
* support llama3.1 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1475
* support megatron by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1365
* fix megatron by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1476
* Support internvl2 grounding by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1473
* update doc by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1477
* Support alignment algorithm for vision MLLM by hjh0119 in https://github.com/modelscope/ms-swift/pull/1474
* fix doc by hjh0119 in https://github.com/modelscope/ms-swift/pull/1481
* Fix visual cpo by hjh0119 in https://github.com/modelscope/ms-swift/pull/1482
* support llama3.1-quant by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1478
* fix part tuner by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1483
* fix import by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1488
* Fix GLM4V by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1490
* support mistral large by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1485
* fix resume_only_model & zero3 & full by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1497
* Fix resume_from_checkpoint & full by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1498
* fix part tuner by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1495
* fix cogvlm2-video by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1500
* [TorchAcc] add script for qwen2 in torchacc by Zhikaiiii in https://github.com/modelscope/ms-swift/pull/1492
* Fix CI by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1501
* fix vlm template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1503
* fix internvl-4b by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1505
* support zero3 & freeze by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1508
* fix part mix with lora by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1509
* fix docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1511
* Update README.md by ArtificialZeng in https://github.com/modelscope/ms-swift/pull/1516
* fix kto custom data by hjh0119 in https://github.com/modelscope/ms-swift/pull/1515
* Fix KTO doc by hjh0119 in https://github.com/modelscope/ms-swift/pull/1517
* Rescale image by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1512
* fix pretrain dataset by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1518
* fix deepseek-vl template by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1521
* Support exporting of llama3.1, and awq-batch-size by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1522
* support lmdeploy & vlm by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1364
* fix tf 4.43 llava by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1525
* fix llamapro by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1527
* fix template & docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1529
* fix lmdeploy & vlm by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1530
* update doc by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1531
* fix lmdeploy & minicpm-v-2.5 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1534
* fix internvl-phi3 batch infer by hjh0119 in https://github.com/modelscope/ms-swift/pull/1539
* Support SA1B series datasets by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1542
* fix bug in _prepare_inputs by guihonghao in https://github.com/modelscope/ms-swift/pull/1543
* Support lmdeploy infer deploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1541
* add lmdeploy link by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1545
* support lmdeploy & app-ui by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1546
* fix lmdeploy bug by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1550
* support more models by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1552
* fix multi node by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1554
* support lmdeploy awq by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1555
* support quant_policy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1556
* fix xcomposer lora by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1559
* Update docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1558
* fix minicpm-v by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1562
* add result_dir paramerter to InferArgument & fix a minor bug by starxhong in https://github.com/modelscope/ms-swift/pull/1561
* fix some bugs in dpo by hjh0119 in https://github.com/modelscope/ms-swift/pull/1565
* Fix bugs 0801 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1566
* fix dataset copy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1569
* fix qwen-vl-merged lmdeploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1572
* Change multi-modal default lora to llm&projector by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1571
* fix quant by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1573
* fix kto by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1575
* update docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1578
* Fix huge model saving by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1579
* Fix/0802 by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1581
* Peft 0.12 by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1586
* fix bugs in gemma2-2b-it by DaozeZhang in https://github.com/modelscope/ms-swift/pull/1587
* [TorchAcc] Update patch for transformers>=4.41.0 by baoleai in https://github.com/modelscope/ms-swift/pull/1584
* fix agent deployment by hjh0119 in https://github.com/modelscope/ms-swift/pull/1592
* support swift deploy stats by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1593
* Fix megatron convert by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1597
* add gemma-2-2b by DaozeZhang in https://github.com/modelscope/ms-swift/pull/1595
* support max_batch_size by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1599
* update docs by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1600
* support multi modal evaluation by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1540
* support internlm2.5 1.8b 20b by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1551
* support qwen1.5 megatron by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1564
* compat with peft==0.11 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1604
* Fix InternVL2 doc by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1607
* Fix rope scaling by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1610
* Fix/rope by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1612
* support minicpm-v-v2_6-chat by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1609
* Fix InternVL2-LLaMA3 by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1614
* fix rope scaling bug by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1620
* fix florence template by hjh0119 in https://github.com/modelscope/ms-swift/pull/1618
* support internlm_xcomposer2_4khd by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1622
* support vllm & vlm by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1630
* Fix ci by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1634
* Compat transformers 4.44 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1640
* fix xcomposer lora_target_modules by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1645
* fix TypeError: 'NoneType' object is not iterable, when only have video data the image is none by Wondersui in https://github.com/modelscope/ms-swift/pull/1637
* support qwen2-math by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1644
* fix peft patch by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1647
* fix oom test in rlhf by hjh0119 in https://github.com/modelscope/ms-swift/pull/1651
* Fix peft patch by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1650
* [TorchAcc] fix bcast of output_dir by baoleai in https://github.com/modelscope/ms-swift/pull/1652
* fix tp lmdeploy by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1654
* fix transformers==4.33 by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1655
* support qwen2-audio by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1633
* update aishell1 dataset by Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1657
* Add OLLaMA doc by tastelikefeet in https://github.com/modelscope/ms-swift/pull/1660
* Support IterableDataset by hjh0119 in https://github.com/modelscope/ms-swift/pull/1596

New Contributors
* ArtificialZeng made their first contribution in https://github.com/modelscope/ms-swift/pull/1516
* guihonghao made their first contribution in https://github.com/modelscope/ms-swift/pull/1543
* DaozeZhang made their first contribution in https://github.com/modelscope/ms-swift/pull/1587
* Wondersui made their first contribution in https://github.com/modelscope/ms-swift/pull/1637

**Full Changelog**: https://github.com/modelscope/ms-swift/compare/v2.2.0...v2.3.0

New Features:
1. Support for RLHF with multimodal models, including DPO, ORPO, SIMPO, and CPO
2. SWIFT infer, SWIFT deploy support using lmdeploy for inference acceleration.
3. Support the use of Megatron for performing PT and SFT on the Qwen2 series models.
4. Support the grounding task for InternVL2/Qwen-VL-Chat models

New Models:
1. mistral-nemo series, mistral-large
2. llama3.1 series

New Datasets:
1. sa1b-dense-caption, sa1b-paired-caption
2. rlaif-v
3. zhihu-kol, zhihu-kol-filtered

What's Changed
* Support api key by Jintao-Huang in https://github.com/modelscope/swift/pull/1452
* Support mistral nemo series models by tastelikefeet in https://github.com/modelscope/swift/pull/1454
* fix minicpm-v2.5 lora_target_modules by Jintao-Huang in https://github.com/modelscope/swift/pull/1455
* Add two datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1459
* Update trl dependency version by hjh0119 in https://github.com/modelscope/swift/pull/1463
* fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1464
* fix yi1.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1465
* Fix yi1.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1467
* add activate and deactivate for part tuner by tastelikefeet in https://github.com/modelscope/swift/pull/1470
* support llama3.1 by Jintao-Huang in https://github.com/modelscope/swift/pull/1475
* support megatron by Jintao-Huang in https://github.com/modelscope/swift/pull/1365
* fix megatron by Jintao-Huang in https://github.com/modelscope/swift/pull/1476
* Support internvl2 grounding by tastelikefeet in https://github.com/modelscope/swift/pull/1473
* update doc by tastelikefeet in https://github.com/modelscope/swift/pull/1477
* Support alignment algorithm for vision MLLM by hjh0119 in https://github.com/modelscope/swift/pull/1474
* fix doc by hjh0119 in https://github.com/modelscope/swift/pull/1481
* Fix visual cpo by hjh0119 in https://github.com/modelscope/swift/pull/1482
* support llama3.1-quant by Jintao-Huang in https://github.com/modelscope/swift/pull/1478
* fix part tuner by tastelikefeet in https://github.com/modelscope/swift/pull/1483
* fix import by tastelikefeet in https://github.com/modelscope/swift/pull/1488
* Fix GLM4V by tastelikefeet in https://github.com/modelscope/swift/pull/1490
* support mistral large by Jintao-Huang in https://github.com/modelscope/swift/pull/1485
* fix resume_only_model & zero3 & full by Jintao-Huang in https://github.com/modelscope/swift/pull/1497
* Fix resume_from_checkpoint & full by Jintao-Huang in https://github.com/modelscope/swift/pull/1498
* fix part tuner by tastelikefeet in https://github.com/modelscope/swift/pull/1495
* fix cogvlm2-video by Jintao-Huang in https://github.com/modelscope/swift/pull/1500
* [TorchAcc] add script for qwen2 in torchacc by Zhikaiiii in https://github.com/modelscope/swift/pull/1492
* Fix CI by tastelikefeet in https://github.com/modelscope/swift/pull/1501
* fix vlm template by Jintao-Huang in https://github.com/modelscope/swift/pull/1503
* fix internvl-4b by Jintao-Huang in https://github.com/modelscope/swift/pull/1505
* support zero3 & freeze by Jintao-Huang in https://github.com/modelscope/swift/pull/1508
* fix part mix with lora by tastelikefeet in https://github.com/modelscope/swift/pull/1509
* fix docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1511
* Update README.md by ArtificialZeng in https://github.com/modelscope/swift/pull/1516
* fix kto custom data by hjh0119 in https://github.com/modelscope/swift/pull/1515
* Fix KTO doc by hjh0119 in https://github.com/modelscope/swift/pull/1517
* Rescale image by tastelikefeet in https://github.com/modelscope/swift/pull/1512
* fix pretrain dataset by Jintao-Huang in https://github.com/modelscope/swift/pull/1518
* fix deepseek-vl template by Jintao-Huang in https://github.com/modelscope/swift/pull/1521
* Support exporting of llama3.1, and awq-batch-size by tastelikefeet in https://github.com/modelscope/swift/pull/1522
* support lmdeploy & vlm by Jintao-Huang in https://github.com/modelscope/swift/pull/1364
* fix tf 4.43 llava by tastelikefeet in https://github.com/modelscope/swift/pull/1525
* fix llamapro by tastelikefeet in https://github.com/modelscope/swift/pull/1527
* fix template & docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1529
* fix lmdeploy & vlm by Jintao-Huang in https://github.com/modelscope/swift/pull/1530
* update doc by tastelikefeet in https://github.com/modelscope/swift/pull/1531
* fix lmdeploy & minicpm-v-2.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1534
* fix internvl-phi3 batch infer by hjh0119 in https://github.com/modelscope/swift/pull/1539
* Support SA1B series datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1542
* fix bug in _prepare_inputs by guihonghao in https://github.com/modelscope/swift/pull/1543
* Support lmdeploy infer deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1541
* add lmdeploy link by Jintao-Huang in https://github.com/modelscope/swift/pull/1545
* support lmdeploy & app-ui by Jintao-Huang in https://github.com/modelscope/swift/pull/1546
* fix lmdeploy bug by Jintao-Huang in https://github.com/modelscope/swift/pull/1550
* support more models by Jintao-Huang in https://github.com/modelscope/swift/pull/1552
* fix multi node by tastelikefeet in https://github.com/modelscope/swift/pull/1554
* support lmdeploy awq by Jintao-Huang in https://github.com/modelscope/swift/pull/1555
* support quant_policy by Jintao-Huang in https://github.com/modelscope/swift/pull/1556
* fix xcomposer lora by Jintao-Huang in https://github.com/modelscope/swift/pull/1559
* Update docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1558
* fix minicpm-v by Jintao-Huang in https://github.com/modelscope/swift/pull/1562
* add result_dir paramerter to InferArgument & fix a minor bug by starxhong in https://github.com/modelscope/swift/pull/1561
* fix some bugs in dpo by hjh0119 in https://github.com/modelscope/swift/pull/1565
* Fix bugs 0801 by Jintao-Huang in https://github.com/modelscope/swift/pull/1566
* fix dataset copy by Jintao-Huang in https://github.com/modelscope/swift/pull/1569
* fix qwen-vl-merged lmdeploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1572

New Contributors
* ArtificialZeng made their first contribution in https://github.com/modelscope/swift/pull/1516
* guihonghao made their first contribution in https://github.com/modelscope/swift/pull/1543

**Full Changelog**: https://github.com/modelscope/swift/compare/v2.2.3...v2.2.5

New Features:
1. support bnb and ollama export
2. suport Q-Galore

New Models:
1. numina-math-7b

Bug Fix:
1. fix vllm>=0.5.1, TP
2. fix internvl2 template
3. fix glm4v merge-lora

What's Changed
* fix internvl doc by hjh0119 in https://github.com/modelscope/swift/pull/1394
* Fix link by Jintao-Huang in https://github.com/modelscope/swift/pull/1397
* fix vllm==0.5.1 by Jintao-Huang in https://github.com/modelscope/swift/pull/1404
* [TorchAcc] update accelerate API and add llama3-70B by baoleai in https://github.com/modelscope/swift/pull/1400
* Support Ollama and BNB for export by tastelikefeet in https://github.com/modelscope/swift/pull/1407
* Fix glm4v merge lora by Jintao-Huang in https://github.com/modelscope/swift/pull/1410
* [TorchAcc] fix model download when using TorchAcc distributed training by baoleai in https://github.com/modelscope/swift/pull/1408
* Support padding left by tastelikefeet in https://github.com/modelscope/swift/pull/1414
* Fix ollama export by tastelikefeet in https://github.com/modelscope/swift/pull/1416
* fix web-ui params by tastelikefeet in https://github.com/modelscope/swift/pull/1417
* fix hub_token by Jintao-Huang in https://github.com/modelscope/swift/pull/1420
* Update ms hub token by Jintao-Huang in https://github.com/modelscope/swift/pull/1424
* Add numina math model by tastelikefeet in https://github.com/modelscope/swift/pull/1421
* fix internvl template by Jintao-Huang in https://github.com/modelscope/swift/pull/1433
* Internvl series models update by hjh0119 in https://github.com/modelscope/swift/pull/1426
* fix internvl2 template by Jintao-Huang in https://github.com/modelscope/swift/pull/1436
* Fix bug and make lazydataset more stable by tastelikefeet in https://github.com/modelscope/swift/pull/1438
* Fix llava-hf by tastelikefeet in https://github.com/modelscope/swift/pull/1439
* [WIP]Support Q-Galore by tastelikefeet in https://github.com/modelscope/swift/pull/1440
* 1. support deepspeed on ui 2. add tools to client_utils by tastelikefeet in https://github.com/modelscope/swift/pull/1446
* fix read csv (float) by Jintao-Huang in https://github.com/modelscope/swift/pull/1447
* fix dataset by tastelikefeet in https://github.com/modelscope/swift/pull/1448
* update internvl doc by hjh0119 in https://github.com/modelscope/swift/pull/1449


**Full Changelog**: https://github.com/modelscope/swift/compare/v2.2.2...v2.2.3

English Version

Features

1. Support lmdeploy for inference
2. Support training for Internvl2 Video
3. Support argument for LoRA target module in regex format
4. Support RTD website
5. Support argument of device_max_memory to config device_map memory usage.

New Models
1. Support CogVLM2-Video

中文

新功能

1. 支持lmdeploy框架的推理能力
2. 支持InternVL2模型的视频训练能力
3. 支持lora_target_regex参数，用来以正则表达式配置lora模块
4. 支持RTD文档网站
5. 支持device_max_memory参数来配置device_map的显存使用

新模型
1. CogVLM2-Video

What's Changed
* fix bug by tastelikefeet in https://github.com/modelscope/swift/pull/1320
* fix template by Jintao-Huang in https://github.com/modelscope/swift/pull/1321
* support cogvlm2-video by Jintao-Huang in https://github.com/modelscope/swift/pull/1318
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1325
* fix web-ui by tastelikefeet in https://github.com/modelscope/swift/pull/1327
* compatible with trl 0.9.6 by hjh0119 in https://github.com/modelscope/swift/pull/1326
* compat with vllm==0.5.1 by Jintao-Huang in https://github.com/modelscope/swift/pull/1329
* Update qrcode by tastelikefeet in https://github.com/modelscope/swift/pull/1332
* fix florence model by hjh0119 in https://github.com/modelscope/swift/pull/1334
* Relaxing requirements for trl by hjh0119 in https://github.com/modelscope/swift/pull/1342
* fix xcomposer2.5 device_map by Jintao-Huang in https://github.com/modelscope/swift/pull/1343
* support generation_info by Jintao-Huang in https://github.com/modelscope/swift/pull/1344
* fix requirements by Jintao-Huang in https://github.com/modelscope/swift/pull/1347
* readthedocs by tastelikefeet in https://github.com/modelscope/swift/pull/1345
* fix sequence parallel get labels by tastelikefeet in https://github.com/modelscope/swift/pull/1352
* fix filelock by tastelikefeet in https://github.com/modelscope/swift/pull/1354
* Add pt command by tastelikefeet in https://github.com/modelscope/swift/pull/1356
* fix generation_info efficiency by Jintao-Huang in https://github.com/modelscope/swift/pull/1359
* fix sh ddp_backend by Jintao-Huang in https://github.com/modelscope/swift/pull/1360
* support LLM & lmdeploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1272
* fix a file path by tastelikefeet in https://github.com/modelscope/swift/pull/1363
* Internvl2 support video by tastelikefeet in https://github.com/modelscope/swift/pull/1366
* fix openai api by Jintao-Huang in https://github.com/modelscope/swift/pull/1367
* fix internvl2-40b by hjh0119 in https://github.com/modelscope/swift/pull/1369
* fix vlm deploy lora & agent by Jintao-Huang in https://github.com/modelscope/swift/pull/1371
* Support lora regex by tastelikefeet in https://github.com/modelscope/swift/pull/1375
* Fix docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1380
* Fix FSDP; Add training percentage to jsonl logging; Add a web-ui component by tastelikefeet in https://github.com/modelscope/swift/pull/1381
* Support max memory args by tastelikefeet in https://github.com/modelscope/swift/pull/1382
* fix max_memory by tastelikefeet in https://github.com/modelscope/swift/pull/1383
* Fix gpu assert calculation by tastelikefeet in https://github.com/modelscope/swift/pull/1384
* fix dataset_sample & deploy stop_words by Jintao-Huang in https://github.com/modelscope/swift/pull/1385


**Full Changelog**: https://github.com/modelscope/swift/compare/v2.2.1...v2.2.2

English Version
New Features

1. **Multimodal**: Supported a large number of multimodal datasets and restructured the multimodal architecture. Some models now support grounding tasks.
2. **Web-ui**: Added support for RLHF, evaluation, and quantization.
3. **Evaluation Functionality**: Refactored the evaluation functionality, now using OpenCompass internally, supporting over 50 evaluation datasets.
4. **Deployment Functionality**: VLLM infer_backend now supports multimodal models.
5. **Agent Training**: Refactored the construction, training, and deployment of agent datasets, making the agent pipeline more complete.
6. **Human Alignment**: Added alignment algorithms such as KTO and CPO, and refactored the human alignment code.

New Models
1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3

New Datasets
1. Over 30 foundational multimodal datasets, including GQA, RefCOCO, and Llava-pretrain.
2. Swift-mix general mixed dataset.
3. Video-chatgpt video dataset.

中文版本
新功能

1. 多模态：支持了非常多的多模态数据集，并重构了多模态架构，部分模型开始支持grounding任务
2. Web-ui：支持了RLHF、评测和量化
3. 评测功能：进行了重构，内部使用了OpenCompass，支持50+评测集
4. 部署功能：VLLM infer_backend支持多模态模型
5. Agent训练：重构了Agent数据集构造、训练、部署，Agent链路更加完整
6. 人类对齐：增加了KTO、CPO等对齐算法，并重构了人类对齐的代码

新模型
1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3


新数据集
1. GQA、RefCOCO、Llava-pretrain等30+多模态基础数据集
2. swift-mix通用混合数据集
3. video-chatgpt视频数据集

What's Changed
* fix py38 by Jintao-Huang in https://github.com/modelscope/swift/pull/1152
* refactor rlhf by hjh0119 in https://github.com/modelscope/swift/pull/1090
* Fix/web UI 0617 by tastelikefeet in https://github.com/modelscope/swift/pull/1158
* Add more datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1065
* support openbuddy-llama3-70b by Jintao-Huang in https://github.com/modelscope/swift/pull/1162
* Add Deepseek coder v2 model by tastelikefeet in https://github.com/modelscope/swift/pull/1164
* Update web-ui labels by tastelikefeet in https://github.com/modelscope/swift/pull/1168
* fix hf download by tastelikefeet in https://github.com/modelscope/swift/pull/1169
* fix kto bugs by hjh0119 in https://github.com/modelscope/swift/pull/1173
* Fix preprocessor by tastelikefeet in https://github.com/modelscope/swift/pull/1174
* refactor ui by tastelikefeet in https://github.com/modelscope/swift/pull/1178
* fix DPO by hjh0119 in https://github.com/modelscope/swift/pull/1181
* fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1171
* 在argument.py添加hccl参数，用于NPU环境下训练使用 by jiaozhentian in https://github.com/modelscope/swift/pull/1165
* Custom agent dataset training support by hjh0119 in https://github.com/modelscope/swift/pull/1184
* RLHF UI by tastelikefeet in https://github.com/modelscope/swift/pull/1182
* fix kto by hjh0119 in https://github.com/modelscope/swift/pull/1187
* update agent doc by hjh0119 in https://github.com/modelscope/swift/pull/1189
* Fix device map in limited mem by tastelikefeet in https://github.com/modelscope/swift/pull/1190
* Fix sk will be uploaded by tastelikefeet in https://github.com/modelscope/swift/pull/1192
* Fix dataset concatenation by tastelikefeet in https://github.com/modelscope/swift/pull/1193
* Fix glm4v dataloader by CjangCjengh in https://github.com/modelscope/swift/pull/1183
* fix glm4v images by Jintao-Huang in https://github.com/modelscope/swift/pull/1194
* [TorchAcc] Cache the compiled results and remove some xla flags by anw90 in https://github.com/modelscope/swift/pull/1160
* Fix vqa dataset loading by tastelikefeet in https://github.com/modelscope/swift/pull/1195
* [TorchAcc] Add acc_steps arguments to speed up torchacc training by anw90 in https://github.com/modelscope/swift/pull/1196
* fix minicpm-v data_collator by Jintao-Huang in https://github.com/modelscope/swift/pull/1197
* Add a tuner to train a part of the model by tastelikefeet in https://github.com/modelscope/swift/pull/1200
* Fix ConversationPreprocessor loading by tastelikefeet in https://github.com/modelscope/swift/pull/1201
* Fix map cache on multi node by tastelikefeet in https://github.com/modelscope/swift/pull/1203
* refactor multi-modal by Jintao-Huang in https://github.com/modelscope/swift/pull/1212
* support llava 1.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1217
* add in_browswer by tastelikefeet in https://github.com/modelscope/swift/pull/1220
* [TorchAcc] Add USE_TORCH_XLA=0 flag for native swift scripts by anw90 in https://github.com/modelscope/swift/pull/1221
* Fix glm4v batch_size>1 by Jintao-Huang in https://github.com/modelscope/swift/pull/1223
* Add debug log support by tastelikefeet in https://github.com/modelscope/swift/pull/1226
* Fix qlora deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1224
* Add new dataset by tastelikefeet in https://github.com/modelscope/swift/pull/1227
* fix refcoco by tastelikefeet in https://github.com/modelscope/swift/pull/1232
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1241
* fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1242
* refactor inference by Jintao-Huang in https://github.com/modelscope/swift/pull/1245
* Add more datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1246
* Support gemma2 by Jintao-Huang in https://github.com/modelscope/swift/pull/1247
* Support gemma2 by Jintao-Huang in https://github.com/modelscope/swift/pull/1248
* add push to hub tracker by tastelikefeet in https://github.com/modelscope/swift/pull/1249
* support Florence series model by hjh0119 in https://github.com/modelscope/swift/pull/1244
* update eval by Jintao-Huang in https://github.com/modelscope/swift/pull/1251
* reset snapshot_download to modelscope by tastelikefeet in https://github.com/modelscope/swift/pull/1254
* Feat/open compass by tastelikefeet in https://github.com/modelscope/swift/pull/1213
* fix output empty will be filtered by tastelikefeet in https://github.com/modelscope/swift/pull/1256
* add phi3 models by tastelikefeet in https://github.com/modelscope/swift/pull/1259
* update dataset preprocess by Jintao-Huang in https://github.com/modelscope/swift/pull/1257
* fix rlhf by tastelikefeet in https://github.com/modelscope/swift/pull/1261
* fix rlhf by Jintao-Huang in https://github.com/modelscope/swift/pull/1260
* Support llava1.6 mistral by Jintao-Huang in https://github.com/modelscope/swift/pull/1268
* Add new dataset by tastelikefeet in https://github.com/modelscope/swift/pull/1270
* support vllm & multimodal by Jintao-Huang in https://github.com/modelscope/swift/pull/1229
* Fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1275
* fix llamapro by tastelikefeet in https://github.com/modelscope/swift/pull/1276
* fix xcomposer deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1280
* Fix/0703 2 by tastelikefeet in https://github.com/modelscope/swift/pull/1284
* Fix llamapro merge and quantized training by tastelikefeet in https://github.com/modelscope/swift/pull/1289
* update lr_scheduler by Jintao-Huang in https://github.com/modelscope/swift/pull/1288
* support internlm2.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1291
* fix vllm by Jintao-Huang in https://github.com/modelscope/swift/pull/1292
* fix florence & deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1293
* fix mini-internvl-phi3 template by Jintao-Huang in https://github.com/modelscope/swift/pull/1298
* fix florence by hjh0119 in https://github.com/modelscope/swift/pull/1301
* Refactor UI and fix some bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1300
* fix dependency by tastelikefeet in https://github.com/modelscope/swift/pull/1306
* support codegeex4 by Jintao-Huang in https://github.com/modelscope/swift/pull/1305
* support internvl2 by hjh0119 in https://github.com/modelscope/swift/pull/1304
* support llava video by Jintao-Huang in https://github.com/modelscope/swift/pull/1307
* fix docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1309
* support lr_scheduler_kwargs by Jintao-Huang in https://github.com/modelscope/swift/pull/1310
* Fix internvl2 template by hjh0119 in https://github.com/modelscope/swift/pull/1308
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1311
* support warmup_stable_decay by Jintao-Huang in https://github.com/modelscope/swift/pull/1312
* Support xcomposer2.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1287
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1319

New Contributors
* jiaozhentian made their first contribution in https://github.com/modelscope/swift/pull/1165
* CjangCjengh made their first contribution in https://github.com/modelscope/swift/pull/1183
* anw90 made their first contribution in https://github.com/modelscope/swift/pull/1160

**Full Changelog**: https://github.com/modelscope/swift/compare/v2.1.1...v2.2.1