Diffusers

Stable Diffusion XL's strength default was accidentally set to 1.0 when creating the pipeline. The default should be set to 0.9999 instead. This patch release fixes that.

All commits
- [SDXL Inpaint] Correct strength default by patrickvonplaten in 4858

https://github.com/huggingface/diffusers/commit/3eb498e7b4868bca7460d41cda52d33c3ede5502#r125606630 introduced a 🐛 that broke the `torch.compile()` support for ControlNets. This patch release fixes that.

All commits

* [Docs] Fix docs controlnet missing /Tip by patrickvonplaten in 4717
* [Torch compile] Fix torch compile for controlnet by patrickvonplaten in 4795

SDXL ControlNets 🚀

The 🧨 diffusers team has trained two ControlNets on [Stable Diffusion XL (SDXL)](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl):

- Canny ([diffusers/controlnet-canny-sdxl-1.0](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0))
- Depth ([diffusers/controlnet-depth-sdxl-1.0](https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0))


You can find all the SDXL ControlNet checkpoints [here](https://huggingface.co/models?other=stable-diffusion-xl&other=controlnet), including some [smaller](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-small) [ones](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0-mid) (5 to 7x smaller).

To know more about how to use these ControlNets to perform inference, check out the respective model cards and the [documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl). To train custom SDXL ControlNets, you can try out [our training script](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md).

MultiControlNet for SDXL

This release also introduces support for combining multiple ControlNets trained on SDXL and performing inference with them. Refer to the [documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl#multicontrolnet) to learn more.

GLIGEN

The GLIGEN model was developed by researchers and engineers from **[University of Wisconsin-Madison, Columbia University, and Microsoft](https://github.com/gligen/GLIGEN)**. The `StableDiffusionGLIGENPipeline` can generate photorealistic images conditioned on grounding inputs. Along with text and bounding boxes, if input images are given, this pipeline can insert objects described by text at the region defined by bounding boxes. Otherwise, it’ll generate an image described by the caption/prompt and insert objects described by text at the region defined by bounding boxes. It’s trained on COCO2014D and COCO2014CD datasets, and the model uses a frozen CLIP ViT-L/14 text encoder to condition itself on grounding inputs.


*(GIF from the [official website](https://gligen.github.io/))*

**Grounded inpainting**

python
import torch
from diffusers import StableDiffusionGLIGENPipeline
from diffusers.utils import load_image

Insert objects described by text at the region defined by bounding boxes
pipe = StableDiffusionGLIGENPipeline.from_pretrained(
"masterful/gligen-1-4-inpainting-text-box", variant="fp16", torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

input_image = load_image(
"https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/gligen/livingroom_modern.png"
)
prompt = "a birthday cake"

0.19.3 is a patch release to make sure `import diffusers` works without `transformers` being installed.

It includes a fix of [this issue](https://github.com/huggingface/diffusers/issues/4356).

All commits

[SDXL] Fix dummy imports incorrect naming by patrickvonplaten in https://github.com/huggingface/diffusers/pull/4370

We still had some bugs 🐛 in 0.19.1 some bugs, notably:

SDXL (Kohya-style) LoRA

The official SD-XL 1.0 LoRA ([Kohya](https://github.com/Linaqruf/kohya-trainer)-styled) is now supported thanks to https://github.com/huggingface/diffusers/pull/4287. You can try it as follows:

py
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
pipe.load_lora_weights("stabilityai/stable-diffusion-xl-base-1.0", weight_name="sd_xl_offset_example-lora_1.0.safetensors")
pipe.to("cuda")

prompt = "beautiful scenery nature glass bottle landscape, purple galaxy bottle"
negative_prompt = "text, watermark"

image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=25).images[0]



In addition, a couple more SDXL LoRAs are now supported: