SDXL VAE How to use with 🧨 diffusers You can integrate this fine tuned VAE decoder to your existing diffusers workflows, by including a vae argument to the StableDiffusionPipeline Model SDXL is a latent diffusion model, where the diffusion operates in a pretrained, learned (and fixed) latent space of an autoencoder. While the bulk of the semantic composition is done by the latent diffusion model, we can improve local , high frequency details in generated images by improving the quality of the autoencoder. To this end, we train the same autoencoder architecture used for the original Stable Diffusion at a larger batch size (256 vs 9) and additionally track the weights with an exponential moving average (EMA). The resulting autoencoder outperforms the original model in all evaluated reconstruction metrics, see the table below. Evaluation SDXL VAE vs original kl f8 VAE vs f8 ft MSE COCO 2017 (256x256, val, 5000 images) Model rFID PSNR SSIM PSIM Link Comments SDXL VAE 4.42 24.7 +/ 3.9 0.73 +/ 0.13 0.88 +/ 0.27 https://huggingface.co/stabilityai/sdxl vae/blob/main/sdxl vae.safetensors as used in SDXL original 4.99 23.4 +/ 3.8 0.69 +/ 0.14 1.01 +/ 0.28 https://ommer lab.com/files/latent…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy