Mmagic

**Highlights**

In this new version of MMagic, we have added support for the following five new algorithms.

- Support ViCo, a new SD personalization method. [Click to View](https://github.com/open-mmlab/mmagic/blob/main/configs/vico/README.md)

<table align="center">
<thead>
<tr>
<td>
<div align="center">
<img src="https://github.com/open-mmlab/mmagic/assets/71176040/58a6953c-053a-40ea-8826-eee428c992b5" width="800"/>
<br/>
</thead>
</table>

- Support AnimateDiff, a popular text2animation method. [Click to View](https://github.com/open-mmlab/mmagic/blob/main/configs/animatediff/README.md)


- Support SDXL. [Click to View](https://github.com/open-mmlab/mmagic/blob/main/configs/stable_diffusion_xl/README.md)

<div align=center>
<img src="https://github.com/okotaku/diffengine/assets/24734142/27d4ebad-5705-4500-826f-41f425a08c0d"/>
</div>

- Support DragGAN implementation with MMagic. [Click to View](https://github.com/open-mmlab/mmagic/blob/main/configs/draggan/README.md)

<div align=center>
<img src="https://github.com/open-mmlab/mmagic/assets/55343765/7c397bd0-fa07-48fe-8a7c-a4022907404b"/>
</div>

- Support for FastComposer. [Click to View](https://github.com/open-mmlab/mmagic/blob/main/configs/fastcomposer/README.md)

<div align=center>
<img src="https://user-images.githubusercontent.com/14927720/265914135-8a25789c-8d30-40cb-8ac5-e3bd3b617aac.png">
</div>

**New Features & Improvements**

- \[Feature\] Support inference with diffusers pipeline, sd_xl first. by liuwenran in https://github.com/open-mmlab/mmagic/pull/2023
- \[Enhance\] add negative prompt for sd inferencer by liuwenran in https://github.com/open-mmlab/mmagic/pull/2021
- \[Enhance\] Update flake8 checking config in setup.cfg by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/2007
- \[Enhance\] Add ‘config_name' as a supplement to the 'model_setting' by liuwenran in https://github.com/open-mmlab/mmagic/pull/2027
- \[Enhance\] faster test by okotaku in https://github.com/open-mmlab/mmagic/pull/2034
- \[Enhance\] Add OpenXLab Badge by ZhaoQiiii in https://github.com/open-mmlab/mmagic/pull/2037

**CodeCamp Contributions**

- \[CodeCamp2023-643\] Add new configs of BigGAN by limafang in https://github.com/open-mmlab/mmagic/pull/2003
- \[CodeCamp2023-648\] MMagic new config GuidedDiffusion by ooooo-create in https://github.com/open-mmlab/mmagic/pull/2005
- \[CodeCamp2023-649\] MMagic new config Instance Colorization by ooooo-create in https://github.com/open-mmlab/mmagic/pull/2010
- \[CodeCamp2023-652\] MMagic new config StyleGAN3 by hhy150 in https://github.com/open-mmlab/mmagic/pull/2018
- \[CodeCamp2023-653\] Add new configs of Real BasicVSR by RangeKing in https://github.com/open-mmlab/mmagic/pull/2030

**Bug Fixes**

- \[Fix\] Fix best practice and back to contents on mainpage, add new models to model zoo by liuwenran in https://github.com/open-mmlab/mmagic/pull/2001
- \[Fix\] Check CI error and remove main stream gpu test by liuwenran in https://github.com/open-mmlab/mmagic/pull/2013
- \[Fix\] Check circle ci memory by liuwenran in https://github.com/open-mmlab/mmagic/pull/2016
- \[Fix\] remove code and fix clip loss ut test by liuwenran in https://github.com/open-mmlab/mmagic/pull/2017
- \[Fix\] mock infer in diffusers pipeline inferencer ut. by liuwenran in https://github.com/open-mmlab/mmagic/pull/2026
- \[Fix\] Fix bug caused by merging draggan by liuwenran in https://github.com/open-mmlab/mmagic/pull/2029
- \[Fix\] Update QRcode by crazysteeaam in https://github.com/open-mmlab/mmagic/pull/2009
- \[Fix\] Replace the download links in README with OpenXLab version by FerryHuang in https://github.com/open-mmlab/mmagic/pull/2038
- \[Fix\] Increase docstring coverage by liuwenran in https://github.com/open-mmlab/mmagic/pull/2039

**New Contributors**

- limafang made their first contribution in https://github.com/open-mmlab/mmagic/pull/2003
- ooooo-create made their first contribution in https://github.com/open-mmlab/mmagic/pull/2005
- hhy150 made their first contribution in https://github.com/open-mmlab/mmagic/pull/2018
- ZhaoQiiii made their first contribution in https://github.com/open-mmlab/mmagic/pull/2037
- ElliotQi made their first contribution in https://github.com/open-mmlab/mmagic/pull/1980
- Beaconsyh08 made their first contribution in https://github.com/open-mmlab/mmagic/pull/2012

**Full Changelog**: https://github.com/open-mmlab/mmagic/compare/v1.0.2...v1.0.3

**Highlights**

**1. More detailed documentation**

Thank you to the community contributors for helping us improve the documentation. We have improved many documents, including both Chinese and English versions. Please refer to the [documentation](https://mmagic.readthedocs.io/en/latest/) for more details.

**2. New algorithms**

- Support Prompt-to-prompt, DDIM Inversion and Null-text Inversion. [Click to View.](https://github.com/open-mmlab/mmagic/blob/main/projects/prompt_to_prompt/README.md)

From right to left: origin image, DDIM inversion, Null-text inversion

<center class="half">
<img src="https://github.com/FerryHuang/mmagic/assets/71176040/34d8a467-5378-41fb-83c6-b23c9dee8f0a" width="200"/><img src="https://github.com/FerryHuang/mmagic/assets/71176040/3d3814b4-7fb5-4232-a56f-fd7fef0ba28e" width="200"/><img src="https://github.com/FerryHuang/mmagic/assets/71176040/43008ed4-a5a3-4f81-ba9f-95d9e79e6a08" width="200"/>
</center>

Prompt-to-prompt Editing

<div align="center">
<b>cat -> dog</b>
<br/>
<img src="https://github.com/FerryHuang/mmagic/assets/71176040/f5d3fc0c-aa7b-4525-9364-365b254d51ca" width="500"/>
</div>

<div align="center">
<b>spider man -> iron man(attention replace)</b>
<br/>
<img src="https://github.com/FerryHuang/mmagic/assets/71176040/074adbc6-bd48-4c82-99aa-f322cf937f5a" width="500"/>
</div>

<div align="center">
<b>Effel tower -> Effel tower at night (attention refine)</b>
<br/>
<img src="https://github.com/FerryHuang/mmagic/assets/71176040/f815dab3-b20c-4936-90e3-a060d3717e22" width="500"/>
</div>

<div align="center">
<b>blossom sakura tree -> blossom(-3) sakura tree (attention reweight)</b>
<br/>
<img src="https://github.com/FerryHuang/mmagic/assets/71176040/5ef770b9-4f28-4ae7-84b0-6c15ea7450e9" width="500"/>
</div>

- Support Textual Inversion. [Click to view.](https://github.com/open-mmlab/mmagic/blob/main/configs/textual_inversion/README.md)

<div align=center>
<img src="https://github.com/open-mmlab/mmagic/assets/28132635/b2dac6f1-5151-4199-bcc2-71b5b1523a16">
</div>

- Support Attention Injection for more stable video generation with controlnet. [Click to view.](https://github.com/open-mmlab/mmagic/blob/main/configs/controlnet_animation/README.md)
- Support Stable Diffusion Inpainting. [Click to view.](https://github.com/open-mmlab/mmagic/blob/main/configs/stable_diffusion/README.md)

**New Features & Improvements**

- \[Enhancement\] Support noise offset in stable diffusion training by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1880
- \[Community\] Support Glide Upsampler by Taited in https://github.com/open-mmlab/mmagic/pull/1663
- \[Enhance\] support controlnet inferencer by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1891
- \[Feature\] support Albumentations augmentation transformations and pipeline by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1894
- \[Feature\] Add Attention Injection for unet by liuwenran in https://github.com/open-mmlab/mmagic/pull/1895
- \[Enhance\] update benchmark scripts by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1907
- \[Enhancement\] update mmagic docs by crazysteeaam in https://github.com/open-mmlab/mmagic/pull/1920
- \[Enhancement\] Support Prompt-to-prompt, ddim inversion and null-text inversion by FerryHuang in https://github.com/open-mmlab/mmagic/pull/1908
- \[CodeCamp2023-302\] Support MMagic visualization and write a user guide by aptsunny in https://github.com/open-mmlab/mmagic/pull/1939
- \[Feature\] Support Textual Inversion by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1822
- \[Feature\] Support stable diffusion inpaint by Taited in https://github.com/open-mmlab/mmagic/pull/1976
- \[Enhancement\] Adopt `BaseModule` for some models by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1543
- \[MMSIG\]支持 DeblurGANv2 inference by xiaomile in https://github.com/open-mmlab/mmagic/pull/1955
- \[CodeCamp2023-647\] Add new configs of EG3D by RangeKing in https://github.com/open-mmlab/mmagic/pull/1985

**Bug Fixes**

- Fix dtype error in StableDiffusion and DreamBooth training by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1879
- Fix gui VideoSlider bug by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1885
- Fix init_model and glide demo by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1888
- Fix InstColorization bug when dim=3 by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1901
- Fix sd and controlnet fp16 bugs by Z-Fran in https://github.com/open-mmlab/mmagic/pull/1914
- Fix num_images_per_prompt in controlnet by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1936
- Revise metafile for sd-inpainting to fix inferencer init by LeoXing1996 in https://github.com/open-mmlab/mmagic/pull/1995

**New Contributors**

- wyyang23 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1886
- yehuixie made their first contribution in https://github.com/open-mmlab/mmagic/pull/1912
- crazysteeaam made their first contribution in https://github.com/open-mmlab/mmagic/pull/1920
- BUPT-NingXinyu made their first contribution in https://github.com/open-mmlab/mmagic/pull/1921
- zhjunqin made their first contribution in https://github.com/open-mmlab/mmagic/pull/1918
- xuesheng1031 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1923
- wslgqq277g made their first contribution in https://github.com/open-mmlab/mmagic/pull/1934
- LYMDLUT made their first contribution in https://github.com/open-mmlab/mmagic/pull/1933
- RangeKing made their first contribution in https://github.com/open-mmlab/mmagic/pull/1930
- xin-li-67 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1932
- chg0901 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1931
- aptsunny made their first contribution in https://github.com/open-mmlab/mmagic/pull/1939
- YanxingLiu made their first contribution in https://github.com/open-mmlab/mmagic/pull/1943
- tackhwa made their first contribution in https://github.com/open-mmlab/mmagic/pull/1937
- Geo-Chou made their first contribution in https://github.com/open-mmlab/mmagic/pull/1940
- qsun1 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1956
- ththth888 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1961
- sijiua made their first contribution in https://github.com/open-mmlab/mmagic/pull/1967
- MING-ZCH made their first contribution in https://github.com/open-mmlab/mmagic/pull/1982
- AllYoung made their first contribution in https://github.com/open-mmlab/mmagic/pull/1996

**New Features & Improvements**

- Support tomesd for StableDiffusion speed-up. [1801](https://github.com/open-mmlab/mmagic/pull/1801)
- Support all inpainting/matting/image restoration models inferencer. [1833](https://github.com/open-mmlab/mmagic/pull/1833), [#1873](https://github.com/open-mmlab/mmagic/pull/1873)
- Support animated drawings at projects. [1837](https://github.com/open-mmlab/mmagic/pull/1837)
- Support Style-Based Global Appearance Flow for Virtual Try-On at projects. [1786](https://github.com/open-mmlab/mmagic/pull/1786)
- Support tokenizer wrapper and support EmbeddingLayerWithFixe. [1846](https://github.com/open-mmlab/mmagic/pull/1846)

**Bug Fixes**

- Fix install requirements. [1819](https://github.com/open-mmlab/mmagic/pull/1819)
- Fix inst-colorization PackInputs. [1828](https://github.com/open-mmlab/mmagic/pull/1828), [#1827](https://github.com/open-mmlab/mmagic/pull/1827)
- Fix inferencer in pip-install. [1875](https://github.com/open-mmlab/mmagic/pull/1875)

**New Contributors**

- XDUWQ made their first contribution in https://github.com/open-mmlab/mmagic/pull/1830
- FerryHuang made their first contribution in https://github.com/open-mmlab/mmagic/pull/1786
- bobo0810 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1851
- jercylew made their first contribution in https://github.com/open-mmlab/mmagic/pull/1874

We are excited to announce the release of MMagic v1.0.0 that inherits from [MMEditing](https://github.com/open-mmlab/mmediting) and [MMGeneration](https://github.com/open-mmlab/mmgeneration).


Since its inception, MMEditing has been the preferred algorithm library for many super-resolution, editing, and generation tasks, helping research teams win more than 10 top international competitions and supporting over 100 GitHub ecosystem projects. After iterative updates with OpenMMLab 2.0 framework and merged with MMGeneration, MMEditing has become a powerful tool that supports low-level algorithms based on both GAN and CNN.

Today, MMEditing embraces Generative AI and transforms into a more advanced and comprehensive AIGC toolkit: **MMagic** (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation).

In MMagic, we have supported 53+ models in multiple tasks such as fine-tuning for stable diffusion, text-to-image, image and video restoration, super-resolution, editing and generation. With excellent training and experiment management support from [MMEngine](https://github.com/open-mmlab/mmengine), MMagic will provide more agile and flexible experimental support for researchers and AIGC enthusiasts, and help you on your AIGC exploration journey. With MMagic, experience more magic in generation! Let's open a new era beyond editing together. More than Editing, Unlock the Magic!

**Highlights**

**1. New Models**

We support 11 new models in 4 new tasks.

- Text2Image / Diffusion
- ControlNet
- DreamBooth
- Stable Diffusion
- Disco Diffusion
- GLIDE
- Guided Diffusion
- 3D-aware Generation
- EG3D
- Image Restoration
- NAFNet
- Restormer
- SwinIR
- Image Colorization
- InstColorization

https://user-images.githubusercontent.com/49083766/233564593-7d3d48ed-e843-4432-b610-35e3d257765c.mp4

**2. Magic Diffusion Model**

For the Diffusion Model, we provide the following "magic" :

- Support image generation based on Stable Diffusion and Disco Diffusion.

- Support Finetune methods such as Dreambooth and DreamBooth LoRA.

- Support controllability in text-to-image generation using ControlNet.

- Support acceleration and optimization strategies based on xFormers to improve training and inference efficiency.

- Support video generation based on MultiFrame Render.
MMagic supports the generation of long videos in various styles through ControlNet and MultiFrame Render.
prompt keywords: a handsome man, silver hair, smiling, play basketball

https://user-images.githubusercontent.com/12782558/227149757-fd054d32-554f-45d5-9f09-319184866d85.mp4

prompt keywords: a girl, black hair, white pants, smiling, play basketball

https://user-images.githubusercontent.com/49083766/233559964-bd5127bd-52f6-44b6-a089-9d7adfbc2430.mp4

prompt keywords: a handsome man

https://user-images.githubusercontent.com/12782558/227152129-d70d5f76-a6fc-4d23-97d1-a94abd08f95a.mp4

- Support calling basic models and sampling strategies through DiffuserWrapper.

- SAM + MMagic = Generate Anything！
SAM (Segment Anything Model) is a popular model these days and can also provide more support for MMagic! If you want to create your own animation, you can go to [OpenMMLab PlayGround](https://github.com/open-mmlab/playground/blob/main/mmediting_sam/README.md).

https://user-images.githubusercontent.com/49083766/233562228-f39fc675-326c-4ae8-986a-c942059effd0.mp4

**3. Upgraded Framework**

To improve your "spellcasting" efficiency, we have made the following adjustments to the "magic circuit":

- By using MMEngine and MMCV of OpenMMLab 2.0 framework, We decompose the editing framework into different modules and one can easily construct a customized editor framework by combining different modules. We can define the training process just like playing with Legos and provide rich components and strategies. In MMagic, you can complete controls on the training process with different levels of APIs.
- Support for 33+ algorithms accelerated by Pytorch 2.0.
- Refactor DataSample to support the combination and splitting of batch dimensions.
- Refactor DataPreprocessor and unify the data format for various tasks during training and inference.
- Refactor MultiValLoop and MultiTestLoop, supporting the evaluation of both generation-type metrics (e.g. FID) and reconstruction-type metrics (e.g. SSIM), and supporting the evaluation of multiple datasets at once.
- Support visualization on local files or using tensorboard and wandb.

**New Features & Improvements**

- Support 53+ algorithms, 232+ configs, 213+ checkpoints, 26+ loss functions, and 20+ metrics.
- Support controlnet animation and Gradio gui. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/configs/controlnet_animation)
- Support Inferencer and Demo using High-level Inference APIs. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/demo)
- Support Gradio gui of Inpainting inference. [Click to view.](https://github.com/open-mmlab/mmagic/blob/main/demo/gradio-demo.py)
- Support qualitative comparison tools. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/tools/gui)
- Enable projects. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/projects)
- Improve converters scripts and documents for datasets. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/tools/dataset_converters)

MMEditing 1.0.0rc1 has merged MMGeneration 1.x.

- Support 42+ algorithms, 169+ configs and 168+ checkpoints.
- Support 26+ loss functions, 20+ metrics.
- Support tensorboard, wandb.
- Support unconditional GANs, conditional GANs, image2image translation and internal learning.

MMEditing 1.0.0rc0 is the first version of MMEditing 1.x, a part of the OpenMMLab 2.0 projects.

Built upon the new [training engine](https://github.com/open-mmlab/mmengine), MMEditing 1.x unifies the interfaces of dataset, models, evaluation, and visualization.

And there are some BC-breaking changes. Please check [the migration tutorial](https://mmagic.readthedocs.io/en/latest/migration/overview.html) for more details.