Ms-swift

New Features:
1. Default zero3.json file
2. Enhanced support for multi-modal models

New Models:
1. Orion series
2. Codefuse series
3. Internlm2-math series
4. Internlm2 series
5. Qwen2 series
6. Yi-vl series
7. Internlm-xcomposer2

What's Changed
* Update orion 14b by Jintao-Huang in https://github.com/modelscope/swift/pull/341
* update codefuse series by Jintao-Huang in https://github.com/modelscope/swift/pull/343
* Support yi vl by Jintao-Huang in https://github.com/modelscope/swift/pull/345
* fix yi-vl finetune error by Yimi81 in https://github.com/modelscope/swift/pull/347
* Fix template encode bug by Jintao-Huang in https://github.com/modelscope/swift/pull/348
* Update internlm2 math by Jintao-Huang in https://github.com/modelscope/swift/pull/349
* Removing eos_token when doing inference. by Jintao-Huang in https://github.com/modelscope/swift/pull/351
* Support zero3 by Jintao-Huang in https://github.com/modelscope/swift/pull/353
* Support internlm xcomposer2 by Jintao-Huang in https://github.com/modelscope/swift/pull/354
* update qwen2 by Jintao-Huang in https://github.com/modelscope/swift/pull/355
* fix baichuan2 bug by Jintao-Huang in https://github.com/modelscope/swift/pull/357
* fix template_encode bug by Jintao-Huang in https://github.com/modelscope/swift/pull/358
* Fix issue 342 by Jintao-Huang in https://github.com/modelscope/swift/pull/359
* fix test_run.py by Jintao-Huang in https://github.com/modelscope/swift/pull/360

New Contributors
* Yimi81 made their first contribution in https://github.com/modelscope/swift/pull/347

**Full Changelog**: https://github.com/modelscope/swift/compare/v1.5.3...v1.5.4

English Version

1. Support show log in text box of web-ui
2. Support share=True in web-ui, only need to set WEBUI_SHARE=1 in environment variable
3. Support deactivate all adapters
4. Support more SFT arguments
5. Add longlora/qalora script
6. Support custom models in web-ui
7. ModelScope SWIFT studio released: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
8. Fix some bugs

----------
中文版本
1. 支持在web-ui中直接显示日志
2. 支持share=True 仅需要在环境变量中设置WEBUI_SHARE=1
3. 支持失活所有adapters
4. 添加了更多SFT参数
5. 添加了longlora/qalora的训练脚本
6. web-ui支持了自己注册的自定义模型
7. SWFT魔搭创空间上线了: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
8. 修复了一些bug

What's Changed
* fix chatglm3 template bug by Jintao-Huang in https://github.com/modelscope/swift/pull/298
* Support studio by tastelikefeet in https://github.com/modelscope/swift/pull/300
* fix text label by tastelikefeet in https://github.com/modelscope/swift/pull/301
* fix a bug may cause module on gpu throws error by tastelikefeet in https://github.com/modelscope/swift/pull/302
* fix_ziya_template_bug by Jintao-Huang in https://github.com/modelscope/swift/pull/303


**Full Changelog**: https://github.com/modelscope/swift/compare/v1.5.1...v1.5.2

English version

New Features

1. Support dtype settings in LoRA
2. Support deactivated tuners offloading
3. Support deployment with OpenAI format restful API
4. Make LongLoRA supports the latest llama2 code

新feature
1. 支持LoRA设置dtype类型
2. 支持将不使用的tuners offloading到cpu和meta设备上
3. 支持OpenAI restful API方式的部署
4. LongLoRA支持最新的llama2代码

What's Changed
* update docs by tastelikefeet in https://github.com/modelscope/swift/pull/269
* Update benchmark 0101 by Jintao-Huang in https://github.com/modelscope/swift/pull/271
* update benchmark by Jintao-Huang in https://github.com/modelscope/swift/pull/276
* Support dtype in lora by tastelikefeet in https://github.com/modelscope/swift/pull/278
* Support deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/275
* update deploy client by Jintao-Huang in https://github.com/modelscope/swift/pull/279
* Support offload by tastelikefeet in https://github.com/modelscope/swift/pull/281
* fix tuner bug by Jintao-Huang in https://github.com/modelscope/swift/pull/285
* Support lora_modules_to_save by Jintao-Huang in https://github.com/modelscope/swift/pull/284
* update template by Jintao-Huang in https://github.com/modelscope/swift/pull/286
* support ModuleToSave original module offloading by tastelikefeet in https://github.com/modelscope/swift/pull/282
* Fix offload by tastelikefeet in https://github.com/modelscope/swift/pull/288
* fix scedit bug by tastelikefeet in https://github.com/modelscope/swift/pull/290
* fix bnb qwen bug by Jintao-Huang in https://github.com/modelscope/swift/pull/289
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/291
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/292
* Fix/longlora by tastelikefeet in https://github.com/modelscope/swift/pull/294
* support additional_trainable_parameters by Jintao-Huang in https://github.com/modelscope/swift/pull/295
* fix webui by tastelikefeet in https://github.com/modelscope/swift/pull/296
* fix trainer push to hub by Jintao-Huang in https://github.com/modelscope/swift/pull/297


**Full Changelog**: https://github.com/modelscope/swift/compare/v1.5.0...v1.5.1

English Version

New features:

1. Support multi-line inference
2. Support multi node training
3. Add benchmarks
4. Support UI training, start by `swift web-ui`
5. Support VLLM inference
6. Support RLHF(DPO) training

New tuners:

1. SCEdit, adopted by TongYi Lab, uses lesser memory but produces better performance than LoRA, and can be used to replace ControlNet in a series of scenarios like Pose control/In-Painting/Out-Paining/Label-removing, etc.

New models:

1. SUS series models
2. Mixtral-MoE series models
3. deepseek series models
4. phi2-3b
5. cogagent-chat/cogagent-vqa
6. codegeex2-6b

New datasets:

Datasets used in RLHF:

1. hh-rlhf
2. stack-exchange-paired

中文版

SWIFT本月新版本已经发布！

新特性:

1. 支持多行输入推理
2. 支持多卡训练
3. 添加了模型训练的benchmarks
4. 支持界面训练和推理，通过`swift web-ui`开启
5. 支持VLLM推理
6. 支持RLHF(DPO)训练

新tuners:

SCEdit: 通义实验室自研的优秀U-Net微调框架，显存占用远小于LoRA，效果较LoRA更好，且可以替代实现ControlNet的效果，实现In-Painting/Out-Paining/标签去除/Pose控制等能力

新模型：

SUS系列模型
Mixtral-MoE系列模型
deepseek系列模型
phi2-3b
cogagent-chat/cogagent-vqa
codegeex2-6b

新数据集:

用于RLHF的数据集：
hh-rlhf
stack-exchange-paired

What's Changed
* update multi-line input (infer) by Jintao-Huang in https://github.com/modelscope/swift/pull/196
* Fix model saving in new format by tastelikefeet in https://github.com/modelscope/swift/pull/198
* Fix compatible error by tastelikefeet in https://github.com/modelscope/swift/pull/201
* Fix bug 1206 by Jintao-Huang in https://github.com/modelscope/swift/pull/202
* fix fp16 & full bug by Jintao-Huang in https://github.com/modelscope/swift/pull/203
* Fix qwen-audio inference bug by Jintao-Huang in https://github.com/modelscope/swift/pull/204
* Support multi node by Jintao-Huang in https://github.com/modelscope/swift/pull/205
* fix typo bug by Jintao-Huang in https://github.com/modelscope/swift/pull/206
* Support sus by Jintao-Huang in https://github.com/modelscope/swift/pull/207
* Support cpu by Jintao-Huang in https://github.com/modelscope/swift/pull/208
* Add Feat: Freeze Parameters, disable_tqdm by Jintao-Huang in https://github.com/modelscope/swift/pull/210
* update dataset by Jintao-Huang in https://github.com/modelscope/swift/pull/212
* Support lazy_tokenize, preprocess_num_proc by Jintao-Huang in https://github.com/modelscope/swift/pull/211
* Support Mixtral MoE by tastelikefeet in https://github.com/modelscope/swift/pull/217
* Add benchmark by Jintao-Huang in https://github.com/modelscope/swift/pull/213
* support ui training by tastelikefeet in https://github.com/modelscope/swift/pull/219
* Fix transformers 4.36 by Jintao-Huang in https://github.com/modelscope/swift/pull/218
* Update mixtral-7b-moe by Jintao-Huang in https://github.com/modelscope/swift/pull/221
* Compatible with peft>=0.7.0 by tastelikefeet in https://github.com/modelscope/swift/pull/220
* fix dtype='fp16' sft bug by Jintao-Huang in https://github.com/modelscope/swift/pull/227
* fix generation_config warning by Jintao-Huang in https://github.com/modelscope/swift/pull/224
* Fix merge_lora & model_cache_dir bug by Jintao-Huang in https://github.com/modelscope/swift/pull/229
* fix lazy_tokenize bug by Jintao-Huang in https://github.com/modelscope/swift/pull/228
* Add inference UI and refactor machenism by tastelikefeet in https://github.com/modelscope/swift/pull/230
* Support deepseek by Jintao-Huang in https://github.com/modelscope/swift/pull/223
* relax version restriction by tastelikefeet in https://github.com/modelscope/swift/pull/232
* fix bug 1218 by Jintao-Huang in https://github.com/modelscope/swift/pull/235
* support deployment by Jintao-Huang in https://github.com/modelscope/swift/pull/231
* update docs by Jintao-Huang in https://github.com/modelscope/swift/pull/238
* Refactor some code by tastelikefeet in https://github.com/modelscope/swift/pull/237
* fix typo bug by Jintao-Huang in https://github.com/modelscope/swift/pull/239
* update readme & phi2-3b by Jintao-Huang in https://github.com/modelscope/swift/pull/241
* Fix argument 1220 by Jintao-Huang in https://github.com/modelscope/swift/pull/242
* Support CogAgent by tastelikefeet in https://github.com/modelscope/swift/pull/243
* fix infer by tastelikefeet in https://github.com/modelscope/swift/pull/244
* Support more peft tuners by tastelikefeet in https://github.com/modelscope/swift/pull/245
* Fix copying additional files by tastelikefeet in https://github.com/modelscope/swift/pull/247
* Add sft for codegeex2 by tastelikefeet in https://github.com/modelscope/swift/pull/248
* fix issue 249 by tastelikefeet in https://github.com/modelscope/swift/pull/250
* Feat/scedit by jiangzeyinzi in https://github.com/modelscope/swift/pull/253
* Update 1228 by Jintao-Huang in https://github.com/modelscope/swift/pull/254
* fix unicode error by tastelikefeet in https://github.com/modelscope/swift/pull/259
* Update readme for SCEdit by tastelikefeet in https://github.com/modelscope/swift/pull/258
* DPO by tastelikefeet in https://github.com/modelscope/swift/pull/255
* update self-cognition by Jintao-Huang in https://github.com/modelscope/swift/pull/261
* Fix/1229 by tastelikefeet in https://github.com/modelscope/swift/pull/260
* fix trainer init by tastelikefeet in https://github.com/modelscope/swift/pull/262
* fix bugs by tastelikefeet in https://github.com/modelscope/swift/pull/263
* fix import by tastelikefeet in https://github.com/modelscope/swift/pull/265
* Fix import by tastelikefeet in https://github.com/modelscope/swift/pull/266
* update perf by Jintao-Huang in https://github.com/modelscope/swift/pull/264
* fix bug by tastelikefeet in https://github.com/modelscope/swift/pull/267
* Support win32 by tastelikefeet in https://github.com/modelscope/swift/pull/268


**Full Changelog**: https://github.com/modelscope/swift/compare/v1.4.0...v1.5.0

English Version

New features:

1. Support for self-awareness fine-tuning.
2. Support for fine-tuning and inference of the AnimateDiff model in the AIGC direction.
3. Support for flash attention in more models: qwen series, qwen-vl series, llama series, openbuddy series, mistral series, yi series, ziya series, using the use_flash_attn parameter.
4. Support for multiple loras to take effect simultaneously.

New tuners:

1. NEFTune
2. ROME supports more models: chatglm

New models:

1. AnimateDiff
2. zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat
3. qwen-1_8b, qwen-1_8b-chat, qwen-1_8b-chat-int4, qwen-1_8b-chat-int8
4. qwen-72b, qwen-72b-chat, qwen-72b-chat-int4, qwen-72b-chat-int8
5. qwen-audio, qwen-audio-chat
6. yi-34b-chat, codefuse-codellama-34b-chat
7. tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4
8. bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k

New datasets:

1. hc3-zh, hc3-en
2. codefuse-python-en, codefuse-eval-instruction-zh
3. aishell1-zh, aishell1-mini-zh

中文版

新特性:

1. 支持自我认知微调.
2. 支持AIGC方向的AnimateDiff模型的微调与推理.
3. 支持更多模型的flash attention: qwen series, qwen-vl series, llama series, openbuddy series, mistral series, yi series, ziya series. 使用use_flash_attn参数.
4. 支持多个lora同时生效

新tuners:

1. NEFTune
2. ROME支持更多模型: chatglm

新模型:

1. AnimateDiff
2. zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat
3. qwen-1_8b, qwen-1_8b-chat, qwen-1_8b-chat-int4, qwen-1_8b-chat-int8
4. qwen-72b, qwen-72b-chat, qwen-72b-chat-int4, qwen-72b-chat-int8
5. qwen-audio, qwen-audio-chat
6. yi-34b-chat, codefuse-codellama-34b-chat
7. tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4
8. bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k

新数据集:

1. hc3-zh, hc3-en
2. codefuse-python-en, codefuse-eval-instruction-zh
3. aishell1-zh, aishell1-mini-zh

What's Changed

* Support Yi-6b sft by tastelikefeet in https://github.com/modelscope/swift/pull/134
* fix CLI by tastelikefeet in https://github.com/modelscope/swift/pull/135
* update readme by tastelikefeet in https://github.com/modelscope/swift/pull/137
* Support xverse 65b sft by tastelikefeet in https://github.com/modelscope/swift/pull/138
* Support bluelm by Jintao-Huang in https://github.com/modelscope/swift/pull/140
* fix doc by tastelikefeet in https://github.com/modelscope/swift/pull/143
* Add neftune by tastelikefeet in https://github.com/modelscope/swift/pull/145
* Update sh by Jintao-Huang in https://github.com/modelscope/swift/pull/144
* Add compatibility test and fix some problems with peft>=0.6.0 by tastelikefeet in https://github.com/modelscope/swift/pull/146
* fix compatible with transformers>=4.35 by Jintao-Huang in https://github.com/modelscope/swift/pull/148
* Update sh 1115 by Jintao-Huang in https://github.com/modelscope/swift/pull/150
* Update doc by tastelikefeet in https://github.com/modelscope/swift/pull/151
* support flash_attn by Jintao-Huang in https://github.com/modelscope/swift/pull/152
* Fix bug: not work on peft<=0.5.0 by tastelikefeet in https://github.com/modelscope/swift/pull/155
* fix register model bug by Jintao-Huang in https://github.com/modelscope/swift/pull/154
* Support tongyi finance 14b by Jintao-Huang in https://github.com/modelscope/swift/pull/157
* add check_model args and fix check_dataset by Jintao-Huang in https://github.com/modelscope/swift/pull/159
* fix load_from_ckpt_dir bug by Jintao-Huang in https://github.com/modelscope/swift/pull/161
* Update arguments by Jintao-Huang in https://github.com/modelscope/swift/pull/162
* new feature: save_infer_result_to_jsonl by Jintao-Huang in https://github.com/modelscope/swift/pull/163
* Feat 1121 by Jintao-Huang in https://github.com/modelscope/swift/pull/165
* update readme and fix bug by Jintao-Huang in https://github.com/modelscope/swift/pull/167
* Add cli merge lora by Jintao-Huang in https://github.com/modelscope/swift/pull/168
* update code by Jintao-Huang in https://github.com/modelscope/swift/pull/169
* support yi-34b-chat by Jintao-Huang in https://github.com/modelscope/swift/pull/164
* Add animate diff by tastelikefeet in https://github.com/modelscope/swift/pull/174
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/175
* Refine LoRA to peft by tastelikefeet in https://github.com/modelscope/swift/pull/176
* support qwem-72b qwen-1_8b qwen-audio by Jintao-Huang in https://github.com/modelscope/swift/pull/180
* Update wechat by Jintao-Huang in https://github.com/modelscope/swift/pull/186
* Fix the slow inference speed bug in qwen AutoGPTQ by Jintao-Huang in https://github.com/modelscope/swift/pull/187
* Support self cognition by Jintao-Huang in https://github.com/modelscope/swift/pull/188
* update dataset model by Jintao-Huang in https://github.com/modelscope/swift/pull/190

**Full Changelog**: https://github.com/modelscope/swift/compare/v1.3.0...v1.4.0

English Version

New Features:

1. Serving supported: LoRA and full-parameter training models are supported in vllm/chatglm.cpp/xinference deployment, check the documentation for details by `make docs` or `docs/source/GetStarted/Deployment.md` file.
2. Support training and inference with CLI and inference with Web-UI.

New Adapters:

- QALoRA
- Long-LoRA
- ROME

New Models:

1. xverse-65b
3. yi-6b
4. ziya2-13b
5. ziya2-13b-chat
6. mistral-7b
7. openbuddy-mistral-7b-chat
8. mistral-7b-chat
9. chatglm3-6b-base
10. chatglm3-6b
11. chatglm3-6b-32k

New Quantized Models:

1. qwen-7b-chat-int4
2. qwen-14b-chat-int4
3. qwen-vl-chat-int4
4. baichuan2-7b-chat-int4
5. baichuan2-13b-chat-int4
6. qwen-7b-chat-int8
7. qwen-14b-chat-int8

------------------
中文版
新功能:

1. 支持部署：全参数训练及LoRA训练支持以vllm/chatglm.cpp/xinference方式进行部署，可以通过`make docs`生成官方文档或查看`docs/source/GetStarted/Deployment.md`文件
2. 支持CLI方式运行训练和推理，同时支持以Web-UI方式运行推理

新的Adapters：

- QALoRA
- Long-LoRA
- ROME

支持训练和推理的新模型:

1. xverse-65b
4. yi-6b
5. ziya2-13b
6. ziya2-13b-chat
7. mistral-7b
8. openbuddy-mistral-7b-chat
9. mistral-7b-chat
10. chatglm3-6b-base
11. chatglm3-6b
12. chatglm3-6b-32k

支持训练和推理的新量化模型:

1. qwen-7b-chat-int4
4. qwen-14b-chat-int4
5. qwen-vl-chat-int4
6. baichuan2-7b-chat-int4
7. baichuan2-13b-chat-int4
8. qwen-7b-chat-int8
9. qwen-14b-chat-int8

----------------
Feature Commits
* add lint script by tastelikefeet in https://github.com/modelscope/swift/pull/94
* add document by Jintao-Huang in https://github.com/modelscope/swift/pull/103
* Update framework.txt by zzclynn in https://github.com/modelscope/swift/pull/105
* Feat/deepspeed by Jintao-Huang in https://github.com/modelscope/swift/pull/39
* update sh by Jintao-Huang in https://github.com/modelscope/swift/pull/106
* Add baichuan2 13b sh by Jintao-Huang in https://github.com/modelscope/swift/pull/108
* Support mistral 7b by Jintao-Huang in https://github.com/modelscope/swift/pull/112
* Support int4 by Jintao-Huang in https://github.com/modelscope/swift/pull/116
* Support qwen int8 by Jintao-Huang in https://github.com/modelscope/swift/pull/117
* support qa_lora by tastelikefeet in https://github.com/modelscope/swift/pull/104
* Add longlora for llama by tastelikefeet in https://github.com/modelscope/swift/pull/115
* Support ROME by tastelikefeet in https://github.com/modelscope/swift/pull/121
* Feat 1018 by Jintao-Huang in https://github.com/modelscope/swift/pull/119
* Add script for ROME by tastelikefeet in https://github.com/modelscope/swift/pull/123
* Update doc by tastelikefeet in https://github.com/modelscope/swift/pull/125
* Feat 1028 by Jintao-Huang in https://github.com/modelscope/swift/pull/122
* update web ui and swift cli by Jintao-Huang in https://github.com/modelscope/swift/pull/126
* ResTuning readme by jiangzeyinzi in https://github.com/modelscope/swift/pull/129
* Update doc by tastelikefeet in https://github.com/modelscope/swift/pull/131

Bug Fix:
* Fix qwen bug by Jintao-Huang in https://github.com/modelscope/swift/pull/98
* fix ci by tastelikefeet in https://github.com/modelscope/swift/pull/109
* fix resume from checkpointing bug by Jintao-Huang in https://github.com/modelscope/swift/pull/110
* fix openbuddy template by Jintao-Huang in https://github.com/modelscope/swift/pull/113
* update ziya2 by Jintao-Huang in https://github.com/modelscope/swift/pull/114
* Fix bug in ROME by tastelikefeet in https://github.com/modelscope/swift/pull/128
* fix rome by tastelikefeet in https://github.com/modelscope/swift/pull/133


New Contributors
* zzclynn made their first contribution in https://github.com/modelscope/swift/pull/105

**Full Changelog**: https://github.com/modelscope/swift/compare/v1.2.0...v1.3.0