Diffusers

SDXL ControlNets 🚀

The 🧨 diffusers team has trained two ControlNets on [Stable Diffusion XL (SDXL)](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl):

- Canny ([diffusers/controlnet-canny-sdxl-1.0](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0))
- Depth ([diffusers/controlnet-depth-sdxl-1.0](https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0))


You can find all the SDXL ControlNet checkpoints [here](https://huggingface.co/models?other=stable-diffusion-xl&other=controlnet), including some [smaller](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-small) [ones](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-mid) (5 to 7x smaller).

To know more about how to use these ControlNets to perform inference, check out the respective model cards and the [documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl). To train custom SDXL ControlNets, you can try out [our training script](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md).

MultiControlNet for SDXL

This release also introduces support for combining multiple ControlNets trained on SDXL and performing inference with them. Refer to the [documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl#multicontrolnet) to learn more.

GLIGEN

The GLIGEN model was developed by researchers and engineers from **[University of Wisconsin-Madison, Columbia University, and Microsoft](https://github.com/gligen/GLIGEN)**. The `StableDiffusionGLIGENPipeline` can generate photorealistic images conditioned on grounding inputs. Along with text and bounding boxes, if input images are given, this pipeline can insert objects described by text at the region defined by bounding boxes. Otherwise, it’ll generate an image described by the caption/prompt and insert objects described by text at the region defined by bounding boxes. It’s trained on COCO2014D and COCO2014CD datasets, and the model uses a frozen CLIP ViT-L/14 text encoder to condition itself on grounding inputs.


*(GIF from the [official website](https://gligen.github.io/))*

**Grounded inpainting**

python
import torch
from diffusers import StableDiffusionGLIGENPipeline
from diffusers.utils import load_image

Insert objects described by text at the region defined by bounding boxes
pipe = StableDiffusionGLIGENPipeline.from_pretrained(
"masterful/gligen-1-4-inpainting-text-box", variant="fp16", torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

input_image = load_image(
"https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/gligen/livingroom_modern.png"
)
prompt = "a birthday cake"

0.19.3 is a patch release to make sure `import diffusers` works without `transformers` being installed.

It includes a fix of [this issue](https://github.com/huggingface/diffusers/issues/4356).

All commits

[SDXL] Fix dummy imports incorrect naming by patrickvonplaten in https://github.com/huggingface/diffusers/pull/4370

We still had some bugs 🐛 in 0.19.1 some bugs, notably:

SDXL (Kohya-style) LoRA

The official SD-XL 1.0 LoRA ([Kohya](https://github.com/Linaqruf/kohya-trainer)-styled) is now supported thanks to https://github.com/huggingface/diffusers/pull/4287. You can try it as follows:

py
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
pipe.load_lora_weights("stabilityai/stable-diffusion-xl-base-1.0", weight_name="sd_xl_offset_example-lora_1.0.safetensors")
pipe.to("cuda")

prompt = "beautiful scenery nature glass bottle landscape, purple galaxy bottle"
negative_prompt = "text, watermark"

image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=25).images[0]



In addition, a couple more SDXL LoRAs are now supported:

Patch release to fix:
- 1. `torch.compile` for SD-XL for certain GPUs
- 2. `from_single_file` for all SD models
- 3. Fix broken ONNX export
- 4. Fix incorrect VAE FP16 casting
- 5. Deprecate loading variants that don't exist

**Note**:

Loading any stable diffusion safetensors or ckpt with `StableDiffusionPipeline.from_single_file` or `StableDiffusionmg2ImgIPipeline.from_single_file` or `StableDiffusionInpaintPipeline.from_single_file` or `StableDiffusionXLPipeline.from_single_file`, ...

is now almost as fast as `from_pretrained(...)` and it's much more tested now.

All commits:

* Make sure torch compile doesn't access unet config by patrickvonplaten in 4008
* [DiffusionPipeline] Deprecate not throwing error when loading non-existant variant by patrickvonplaten in 4011
* Correctly keep vae in `float16` when using PyTorch 2 or xFormers by pcuenca in 4019
* minor improvements to the SDXL doc. by sayakpaul in 3985
* Remove remaining `not` in upscale pipeline by pcuenca in 4020
* FIX `force_download` in download utility by Wauplin in 4036
* Improve single loading file by patrickvonplaten in 4041
* keep _use_default_values as a list type by oOraph in 4040