English Version
New Features
1. **Multimodal**: Supported a large number of multimodal datasets and restructured the multimodal architecture. Some models now support grounding tasks.
2. **Web-ui**: Added support for RLHF, evaluation, and quantization.
3. **Evaluation Functionality**: Refactored the evaluation functionality, now using OpenCompass internally, supporting over 50 evaluation datasets.
4. **Deployment Functionality**: VLLM infer_backend now supports multimodal models.
5. **Agent Training**: Refactored the construction, training, and deployment of agent datasets, making the agent pipeline more complete.
6. **Human Alignment**: Added alignment algorithms such as KTO and CPO, and refactored the human alignment code.
New Models
1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3
New Datasets
1. Over 30 foundational multimodal datasets, including GQA, RefCOCO, and Llava-pretrain.
2. Swift-mix general mixed dataset.
3. Video-chatgpt video dataset.
中文版本
新功能
1. 多模态：支持了非常多的多模态数据集，并重构了多模态架构，部分模型开始支持grounding任务
2. Web-ui：支持了RLHF、评测和量化
3. 评测功能：进行了重构，内部使用了OpenCompass，支持50+评测集
4. 部署功能：VLLM infer_backend支持多模态模型
5. Agent训练：重构了Agent数据集构造、训练、部署，Agent链路更加完整
6. 人类对齐：增加了KTO、CPO等对齐算法，并重构了人类对齐的代码
新模型
1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3
新数据集
1. GQA、RefCOCO、Llava-pretrain等30+多模态基础数据集
2. swift-mix通用混合数据集
3. video-chatgpt视频数据集
What's Changed
* fix py38 by Jintao-Huang in https://github.com/modelscope/swift/pull/1152
* refactor rlhf by hjh0119 in https://github.com/modelscope/swift/pull/1090
* Fix/web UI 0617 by tastelikefeet in https://github.com/modelscope/swift/pull/1158
* Add more datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1065
* support openbuddy-llama3-70b by Jintao-Huang in https://github.com/modelscope/swift/pull/1162
* Add Deepseek coder v2 model by tastelikefeet in https://github.com/modelscope/swift/pull/1164
* Update web-ui labels by tastelikefeet in https://github.com/modelscope/swift/pull/1168
* fix hf download by tastelikefeet in https://github.com/modelscope/swift/pull/1169
* fix kto bugs by hjh0119 in https://github.com/modelscope/swift/pull/1173
* Fix preprocessor by tastelikefeet in https://github.com/modelscope/swift/pull/1174
* refactor ui by tastelikefeet in https://github.com/modelscope/swift/pull/1178
* fix DPO by hjh0119 in https://github.com/modelscope/swift/pull/1181
* fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1171
* 在argument.py添加hccl参数，用于NPU环境下训练使用 by jiaozhentian in https://github.com/modelscope/swift/pull/1165
* Custom agent dataset training support by hjh0119 in https://github.com/modelscope/swift/pull/1184
* RLHF UI by tastelikefeet in https://github.com/modelscope/swift/pull/1182
* fix kto by hjh0119 in https://github.com/modelscope/swift/pull/1187
* update agent doc by hjh0119 in https://github.com/modelscope/swift/pull/1189
* Fix device map in limited mem by tastelikefeet in https://github.com/modelscope/swift/pull/1190
* Fix sk will be uploaded by tastelikefeet in https://github.com/modelscope/swift/pull/1192
* Fix dataset concatenation by tastelikefeet in https://github.com/modelscope/swift/pull/1193
* Fix glm4v dataloader by CjangCjengh in https://github.com/modelscope/swift/pull/1183
* fix glm4v images by Jintao-Huang in https://github.com/modelscope/swift/pull/1194
* [TorchAcc] Cache the compiled results and remove some xla flags by anw90 in https://github.com/modelscope/swift/pull/1160
* Fix vqa dataset loading by tastelikefeet in https://github.com/modelscope/swift/pull/1195
* [TorchAcc] Add acc_steps arguments to speed up torchacc training by anw90 in https://github.com/modelscope/swift/pull/1196
* fix minicpm-v data_collator by Jintao-Huang in https://github.com/modelscope/swift/pull/1197
* Add a tuner to train a part of the model by tastelikefeet in https://github.com/modelscope/swift/pull/1200
* Fix ConversationPreprocessor loading by tastelikefeet in https://github.com/modelscope/swift/pull/1201
* Fix map cache on multi node by tastelikefeet in https://github.com/modelscope/swift/pull/1203
* refactor multi-modal by Jintao-Huang in https://github.com/modelscope/swift/pull/1212
* support llava 1.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1217
* add in_browswer by tastelikefeet in https://github.com/modelscope/swift/pull/1220
* [TorchAcc] Add USE_TORCH_XLA=0 flag for native swift scripts by anw90 in https://github.com/modelscope/swift/pull/1221
* Fix glm4v batch_size>1 by Jintao-Huang in https://github.com/modelscope/swift/pull/1223
* Add debug log support by tastelikefeet in https://github.com/modelscope/swift/pull/1226
* Fix qlora deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1224
* Add new dataset by tastelikefeet in https://github.com/modelscope/swift/pull/1227
* fix refcoco by tastelikefeet in https://github.com/modelscope/swift/pull/1232
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1241
* fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1242
* refactor inference by Jintao-Huang in https://github.com/modelscope/swift/pull/1245
* Add more datasets by tastelikefeet in https://github.com/modelscope/swift/pull/1246
* Support gemma2 by Jintao-Huang in https://github.com/modelscope/swift/pull/1247
* Support gemma2 by Jintao-Huang in https://github.com/modelscope/swift/pull/1248
* add push to hub tracker by tastelikefeet in https://github.com/modelscope/swift/pull/1249
* support Florence series model by hjh0119 in https://github.com/modelscope/swift/pull/1244
* update eval by Jintao-Huang in https://github.com/modelscope/swift/pull/1251
* reset snapshot_download to modelscope by tastelikefeet in https://github.com/modelscope/swift/pull/1254
* Feat/open compass by tastelikefeet in https://github.com/modelscope/swift/pull/1213
* fix output empty will be filtered by tastelikefeet in https://github.com/modelscope/swift/pull/1256
* add phi3 models by tastelikefeet in https://github.com/modelscope/swift/pull/1259
* update dataset preprocess by Jintao-Huang in https://github.com/modelscope/swift/pull/1257
* fix rlhf by tastelikefeet in https://github.com/modelscope/swift/pull/1261
* fix rlhf by Jintao-Huang in https://github.com/modelscope/swift/pull/1260
* Support llava1.6 mistral by Jintao-Huang in https://github.com/modelscope/swift/pull/1268
* Add new dataset by tastelikefeet in https://github.com/modelscope/swift/pull/1270
* support vllm & multimodal by Jintao-Huang in https://github.com/modelscope/swift/pull/1229
* Fix bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/1275
* fix llamapro by tastelikefeet in https://github.com/modelscope/swift/pull/1276
* fix xcomposer deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1280
* Fix/0703 2 by tastelikefeet in https://github.com/modelscope/swift/pull/1284
* Fix llamapro merge and quantized training by tastelikefeet in https://github.com/modelscope/swift/pull/1289
* update lr_scheduler by Jintao-Huang in https://github.com/modelscope/swift/pull/1288
* support internlm2.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1291
* fix vllm by Jintao-Huang in https://github.com/modelscope/swift/pull/1292
* fix florence & deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/1293
* fix mini-internvl-phi3 template by Jintao-Huang in https://github.com/modelscope/swift/pull/1298
* fix florence by hjh0119 in https://github.com/modelscope/swift/pull/1301
* Refactor UI and fix some bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1300
* fix dependency by tastelikefeet in https://github.com/modelscope/swift/pull/1306
* support codegeex4 by Jintao-Huang in https://github.com/modelscope/swift/pull/1305
* support internvl2 by hjh0119 in https://github.com/modelscope/swift/pull/1304
* support llava video by Jintao-Huang in https://github.com/modelscope/swift/pull/1307
* fix docs by Jintao-Huang in https://github.com/modelscope/swift/pull/1309
* support lr_scheduler_kwargs by Jintao-Huang in https://github.com/modelscope/swift/pull/1310
* Fix internvl2 template by hjh0119 in https://github.com/modelscope/swift/pull/1308
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1311
* support warmup_stable_decay by Jintao-Huang in https://github.com/modelscope/swift/pull/1312
* Support xcomposer2.5 by Jintao-Huang in https://github.com/modelscope/swift/pull/1287
* Fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/1319
New Contributors
* jiaozhentian made their first contribution in https://github.com/modelscope/swift/pull/1165
* CjangCjengh made their first contribution in https://github.com/modelscope/swift/pull/1183
* anw90 made their first contribution in https://github.com/modelscope/swift/pull/1160
**Full Changelog**: https://github.com/modelscope/swift/compare/v2.1.1...v2.2.1