Ms-swift

New Features

1. Support for peft 0.10.x version, with the default value of the `tuner_backend` parameter changed to `peft`. The interface of peft has been dynamically patched to support parameters like `lora_dtype`.
2. Support for vllm+lora inference.
3. Refactored and updated the README file.
4. Added English versions of the documentation. Currently, all documents have both English and Chinese versions.
5. Support for training 70B models using FSDP+QLoRA on dual 24GB GPUs. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh)
6. Support for training agents and using the ModelScopeAgent framework. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source/LLM/Agent%E5%BE%AE%E8%B0%83%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md](https://github.com/modelscope/swift/blob/main/docs/source/LLM/Agent%E5%BE%AE%E8%B0%83%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md)
7. Support for model evaluation and benchmark. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.md](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.md)
8. Support for multi-task experiment management. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A3.md](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A3.md)
9. Support for GaLore training.
10. Support for training and inference of AQLM and AWQ quantized models.

New Models
- MAMBA series models. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mamba-1.4b/lora/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mamba-1.4b/lora/sft.sh)
- DeepSeek VL series models. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/deepseek-vl-best-practice.md](https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/deepseek-vl-best-practice.md)
- LLAVA series models. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/llava%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md](https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/llava%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md)
- TeleChat models. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/telechat_12b/lora/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/telechat_12b/lora/sft.sh)
- Grok-1 models. Documentation available at: [https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Grok-1-best-practice.md](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Grok-1-best-practice.md)
- Qwen 1.5 MoE series models for training and inference.
- dbrx models for training and inference. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/dbrx-instruct/lora_mp/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/dbrx-instruct/lora_mp/sft.sh)
- Mengzi3 models for training and inference. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mengzi3_13b_base/lora_ddp_ds/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mengzi3_13b_base/lora_ddp_ds/sft.sh)
- Xverse MoE models for training and inference. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/xverse_moe_a4_2b/lora/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/xverse_moe_a4_2b/lora/sft.sh)
- c4ai-command-r series models for training and inference.
- MiniCPM series models for training and inference. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/minicpm_moe_8x2b/lora_ddp/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/minicpm_moe_8x2b/lora_ddp/sft.sh)
- Mixtral-8x22B-v0.1 models for training and inference. Script available at: [https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mixtral_moe_8x22b_v1/lora_ddp_ds/sft.sh](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mixtral_moe_8x22b_v1/lora_ddp_ds/sft.sh)

New Datasets
- Support for the `Ruozhiba` dataset: [https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Supported-models-datasets.md](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Supported-models-datasets.md)

What's Changed
* Fix RsLoRA by tastelikefeet in https://github.com/modelscope/swift/pull/567
* Fix yi-vl merge lora by Jintao-Huang in https://github.com/modelscope/swift/pull/568
* Add doc for tuner module by tastelikefeet in https://github.com/modelscope/swift/pull/571
* update agent documentation by tastelikefeet in https://github.com/modelscope/swift/pull/572
* Update agent doc to fix some conflicts by tastelikefeet in https://github.com/modelscope/swift/pull/573
* support vllm lora by Jintao-Huang in https://github.com/modelscope/swift/pull/565
* Support llava by Jintao-Huang in https://github.com/modelscope/swift/pull/577
* fix app-ui max_length is None by Jintao-Huang in https://github.com/modelscope/swift/pull/580
* support `train_dataset_mix_ds` using custom_local_path by Jintao-Huang in https://github.com/modelscope/swift/pull/582
* Fix LRScheduler by tastelikefeet in https://github.com/modelscope/swift/pull/586
* compat with transformers==4.39 by Jintao-Huang in https://github.com/modelscope/swift/pull/584
* Fix weight saving by tastelikefeet in https://github.com/modelscope/swift/pull/589
* fix mix_dataset_sample float by Jintao-Huang in https://github.com/modelscope/swift/pull/594
* Refactor all docs by tastelikefeet in https://github.com/modelscope/swift/pull/599
* fix tiny bugs in docs by tastelikefeet in https://github.com/modelscope/swift/pull/600
* fix issue template and add a pr one by tastelikefeet in https://github.com/modelscope/swift/pull/601
* Fix/security template by tastelikefeet in https://github.com/modelscope/swift/pull/603
* update docs by Jintao-Huang in https://github.com/modelscope/swift/pull/604
* support Mistral-7b-v0.2 by hjh0119 in https://github.com/modelscope/swift/pull/605
* fix deploy safe_response by Jintao-Huang in https://github.com/modelscope/swift/pull/614
* Fix Adalora with devicemap by tastelikefeet in https://github.com/modelscope/swift/pull/619
* update ui by tastelikefeet in https://github.com/modelscope/swift/pull/621
* support TeleChat-12b by hjh0119 in https://github.com/modelscope/swift/pull/607
* fix save dir (additional_files) by Jintao-Huang in https://github.com/modelscope/swift/pull/622
* fix Telechat model by hjh0119 in https://github.com/modelscope/swift/pull/623
* Add Grok model by tastelikefeet in https://github.com/modelscope/swift/pull/629
* add missing files by tastelikefeet in https://github.com/modelscope/swift/pull/631
* support qwen1.5-moe model by hjh0119 in https://github.com/modelscope/swift/pull/627
* support Telechat-7b model by hjh0119 in https://github.com/modelscope/swift/pull/630
* support model Dbrx by hjh0119 in https://github.com/modelscope/swift/pull/643
* fix ui by tastelikefeet in https://github.com/modelscope/swift/pull/648
* fix typing hint by Jintao-Huang in https://github.com/modelscope/swift/pull/649
* support Mengzi-13b-base model by hjh0119 in https://github.com/modelscope/swift/pull/646
* support Qwen1.5-32b models by hjh0119 in https://github.com/modelscope/swift/pull/655
* fix plot error by tastelikefeet in https://github.com/modelscope/swift/pull/651
* Support FSDP + QLoRA by tastelikefeet in https://github.com/modelscope/swift/pull/659
* move fsdp config path by tastelikefeet in https://github.com/modelscope/swift/pull/662
* change the default value of ddp_backend by tastelikefeet in https://github.com/modelscope/swift/pull/667
* fix ui log by tastelikefeet in https://github.com/modelscope/swift/pull/669
* support Xverse-MoE model by hjh0119 in https://github.com/modelscope/swift/pull/668
* Support longlora for transformers 4.38 by tastelikefeet in https://github.com/modelscope/swift/pull/456
* add ruozhiba datasets by tastelikefeet in https://github.com/modelscope/swift/pull/670
* compatible with old versions of modelscope by tastelikefeet in https://github.com/modelscope/swift/pull/671
* Fix data_collator by tastelikefeet in https://github.com/modelscope/swift/pull/674
* [TorchAcc][Experimental] Integrate TorchAcc. by baoleai in https://github.com/modelscope/swift/pull/647
* update Agent best practice with Modelscope-Agent by hjh0119 in https://github.com/modelscope/swift/pull/676
* support c4ai-command-r model by hjh0119 in https://github.com/modelscope/swift/pull/684
* Support Eval by tastelikefeet in https://github.com/modelscope/swift/pull/494
* fix anchor by tastelikefeet in https://github.com/modelscope/swift/pull/687
* Fix/0412 by tastelikefeet in https://github.com/modelscope/swift/pull/690
* support minicpm and mixtral-moe model by hjh0119 in https://github.com/modelscope/swift/pull/692
* fix device_map 4 (qwen-vl) by Jintao-Huang in https://github.com/modelscope/swift/pull/695
* fix multimodal model image_mode = 'CMYK' (fix issue677) by Jintao-Huang in https://github.com/modelscope/swift/pull/697
* feat(model): support minicpm-v-2(699 ) by YuzaChongyi in https://github.com/modelscope/swift/pull/699

New Contributors
* hjh0119 made their first contribution in https://github.com/modelscope/swift/pull/605
* YuzaChongyi made their first contribution in https://github.com/modelscope/swift/pull/699

**Full Changelog**: https://github.com/modelscope/swift/compare/v1.7.3...v2.0.0

New Features:
1. Added support for swift export, enabling awq-int4 quantization and gpt-int2,3,4,8 quantization. Models can be pushed to the Modelscope Hub. You can view the documentation [here](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E9%87%8F%E5%8C%96%E6%96%87%E6%A1%A3.md).
2. Enabled fine-tuning of [awq quantized models](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_awq/lora/sft.sh).
3. Enabled fine-tuning of [aqlm quantized models](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_7b_aqlm_2bit_1x16/lora/sft.sh).
4. Added support for deploying LLM with infer_backend='pt'.
5. Added web-ui with task management and visualization of training loss, eval loss, etc. Inference is accelerated using VLLM.

New Tuners:
1. Lora+.
2. LlamaPro.

New Models:
1. qwen1.5 awq series.
2. gemma series.
3. yi-9b.
4. deepseek-math series.
5. internlm2-1_8b series.
6. openbuddy-mixtral-moe-7b-chat.
7. llama2 aqlm series.

New Datasets:
1. ms-bench-mini.
2. hh-rlhf-cn series.
3. disc-law-sft-zh, disc-med-sft-zh.
4. pileval.


What's Changed
* Fix vllm==0.3 deploy bug by Jintao-Huang in https://github.com/modelscope/swift/pull/412
* Support deepseek math by Jintao-Huang in https://github.com/modelscope/swift/pull/413
* update support_vllm by Jintao-Huang in https://github.com/modelscope/swift/pull/415
* fix zero3 & swift lora by Jintao-Huang in https://github.com/modelscope/swift/pull/416
* Support peft0.8.0 by tastelikefeet in https://github.com/modelscope/swift/pull/423
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/426
* fix pai open with 'a' by Jintao-Huang in https://github.com/modelscope/swift/pull/430
* default load_best_model_at_end=False by Jintao-Huang in https://github.com/modelscope/swift/pull/432
* support openbuddy mixtral by Jintao-Huang in https://github.com/modelscope/swift/pull/437
* support gemma by Jintao-Huang in https://github.com/modelscope/swift/pull/441
* Support ms bench mini by Jintao-Huang in https://github.com/modelscope/swift/pull/442
* Add roadmap and contributing doc by tastelikefeet in https://github.com/modelscope/swift/pull/431
* support peft format by tastelikefeet in https://github.com/modelscope/swift/pull/438
* update contributing.md by Jintao-Huang in https://github.com/modelscope/swift/pull/446
* fix link by tastelikefeet in https://github.com/modelscope/swift/pull/447
* Fix rlhf dataset by tastelikefeet in https://github.com/modelscope/swift/pull/451
* Add task management for webui by tastelikefeet in https://github.com/modelscope/swift/pull/457
* Support swift export by Jintao-Huang in https://github.com/modelscope/swift/pull/455
* Fix llm quantization docs by Jintao-Huang in https://github.com/modelscope/swift/pull/458
* fix get_vllm_engine bug by Jintao-Huang in https://github.com/modelscope/swift/pull/463
* use cpu export by Jintao-Huang in https://github.com/modelscope/swift/pull/462
* Fix llama2 generation config by Jintao-Huang in https://github.com/modelscope/swift/pull/468
* Support editing model_id_or_path by tastelikefeet in https://github.com/modelscope/swift/pull/469
* Support pt deploy by Jintao-Huang in https://github.com/modelscope/swift/pull/467
* Fix swift deploy bug by Jintao-Huang in https://github.com/modelscope/swift/pull/470
* fix deploy dep by Jintao-Huang in https://github.com/modelscope/swift/pull/471
* Support LLaMAPRO and LoRA+ by tastelikefeet in https://github.com/modelscope/swift/pull/472
* Support internlm2 1.8b by Jintao-Huang in https://github.com/modelscope/swift/pull/473
* fix deepseek moe device_map by Jintao-Huang in https://github.com/modelscope/swift/pull/476
* fix peft compatible bug by tastelikefeet in https://github.com/modelscope/swift/pull/482
* Fix deepspeed init bug by Jintao-Huang in https://github.com/modelscope/swift/pull/481
* fix scripts docs by Jintao-Huang in https://github.com/modelscope/swift/pull/483
* Update `swift export` and update docs by Jintao-Huang in https://github.com/modelscope/swift/pull/484
* support gptq export by Jintao-Huang in https://github.com/modelscope/swift/pull/485
* fix docs & readme by Jintao-Huang in https://github.com/modelscope/swift/pull/486
* fix app-ui bug by Jintao-Huang in https://github.com/modelscope/swift/pull/488
* Support peft0.9 by tastelikefeet in https://github.com/modelscope/swift/pull/490
* support torchrun_args for dpo cli and support web_ui model deployment by slin000111 in https://github.com/modelscope/swift/pull/496
* Support transformers 4.33.0 by tastelikefeet in https://github.com/modelscope/swift/pull/498
* Update deepspeed config by Jintao-Huang in https://github.com/modelscope/swift/pull/500
* move docs to classroom by tastelikefeet in https://github.com/modelscope/swift/pull/503
* Support yi 9b by Jintao-Huang in https://github.com/modelscope/swift/pull/504
* Update yi sh by Jintao-Huang in https://github.com/modelscope/swift/pull/506


**Full Changelog**: https://github.com/modelscope/swift/compare/v1.6.0...v1.7.0

New Models:
1. deepseek-math series

New Datasets:
1. sharegpt-gpt4-mini
2. disc-law-sft-zh
3. disc-med-sft-zh

Bug Fix
1. Fix vllm==0.3 & swift deploy bug.
2. Fix zero3 & swift lora bug.

**Full Changelog**: https://github.com/modelscope/swift/compare/v1.6.0...v1.6.1

New Features:
1. Agent Training
2. AIGC support: controlnet, controlnet_sdxl, dreambooth, text_to_image, text_to_image_sdxl
3. Compatibility with vllm==0.3.*

New Models:
1. qwen1.5 series
2. openbmb series


What's Changed
* update openbmb sh by Jintao-Huang in https://github.com/modelscope/swift/pull/361
* Fix openbmb model name by tastelikefeet in https://github.com/modelscope/swift/pull/362
* support dpo cli and add examples controlnet and dreambooth by slin000111 in https://github.com/modelscope/swift/pull/344
* support openbmb minicpm by Jintao-Huang in https://github.com/modelscope/swift/pull/364
* Support agent training, etc. by tastelikefeet in https://github.com/modelscope/swift/pull/352
* fix tuner by tastelikefeet in https://github.com/modelscope/swift/pull/365
* Fix agent doc by tastelikefeet in https://github.com/modelscope/swift/pull/366
* Fix data format in readme by tastelikefeet in https://github.com/modelscope/swift/pull/367
* fix lazy_tokenize bug by Jintao-Huang in https://github.com/modelscope/swift/pull/369
* Fix length penalty by Jintao-Huang in https://github.com/modelscope/swift/pull/371
* fix loss by tastelikefeet in https://github.com/modelscope/swift/pull/372
* update compute loss by Jintao-Huang in https://github.com/modelscope/swift/pull/375
* fix system='' bug by Jintao-Huang in https://github.com/modelscope/swift/pull/374
* fix system='' bug by Jintao-Huang in https://github.com/modelscope/swift/pull/378
* Support PAI compat by Jintao-Huang in https://github.com/modelscope/swift/pull/373
* fix doc by tastelikefeet in https://github.com/modelscope/swift/pull/376
* Fix the conflict between agent and CT by tastelikefeet in https://github.com/modelscope/swift/pull/379
* fix cogagent_18b_chat sh typo error by Jintao-Huang in https://github.com/modelscope/swift/pull/381
* Fix loss scale by tastelikefeet in https://github.com/modelscope/swift/pull/383
* Feat/qwen1.5 by tastelikefeet in https://github.com/modelscope/swift/pull/385
* fix template name by tastelikefeet in https://github.com/modelscope/swift/pull/389
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/386
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/390
* Support max model len by Jintao-Huang in https://github.com/modelscope/swift/pull/392
* Support vllm max model len by Jintao-Huang in https://github.com/modelscope/swift/pull/394
* fix arguments bug by Jintao-Huang in https://github.com/modelscope/swift/pull/395
* support vllm 0.3 by Jintao-Huang in https://github.com/modelscope/swift/pull/396
* fix deepspeed_config_path bug by Jintao-Huang in https://github.com/modelscope/swift/pull/398
* fix file name by slin000111 in https://github.com/modelscope/swift/pull/397
* Add qwen1.5 scripts by tastelikefeet in https://github.com/modelscope/swift/pull/393
* fix many bugs by Jintao-Huang in https://github.com/modelscope/swift/pull/399
* Fix baichuan2 int4 bug by Jintao-Huang in https://github.com/modelscope/swift/pull/400
* Fix qwen1half deploy bug by Jintao-Huang in https://github.com/modelscope/swift/pull/402
* fix readme and test_llm by tastelikefeet in https://github.com/modelscope/swift/pull/404
* update readme by Jintao-Huang in https://github.com/modelscope/swift/pull/405


**Full Changelog**: https://github.com/modelscope/swift/compare/v1.5.4...v1.6.0