How AI Image Generators Turn Noise Into Pictures

Many AI image generators build a picture by starting with random noise and cleaning it up in small steps, guided by the words in your prompt. This approach is called a diffusion model. This article explains the idea in plain terms, and what the main parts of a system such as Stable Diffusion do.

Learning by adding noise

Training starts with a large set of images. According to Wikipedia, each training image is gradually corrupted with random noise over many steps until it is almost pure static. That forward process needs no learning, since adding noise is easy.

The learning happens in the reverse direction. A neural network is trained to undo one small noise step at a time, and in practice it predicts the noise that was added. Wikipedia says the network is typically a U-Net or a transformer. Hugging Face's documentation describes the same idea: at each step the model predicts a noise residual, and a separate scheduler uses it to produce a less noisy image.

Generating a new image

To make a new image, the system starts from fresh random noise the size of the target. It then runs the trained network again and again. After each pass the result is a little less noisy, until a picture is left. Hugging Face's example uses 50 steps for a basic model and 25 for its Stable Diffusion walkthrough, though the number is a setting, not a fixed rule.

Because the starting noise is random, a different seed gives a different picture. Using the same seed and settings is what lets a user reproduce a result.

Steering with text

A plain diffusion model produces images that look like its training data, but it does not know what you asked for. Text-to-image systems add a text encoder. A tokenizer first splits the prompt into tokens, and the encoder turns them into numbers called embeddings. Wikipedia says commercial generators combine diffusion with text encoders and cross-attention modules, which let the image-making network consult the text as it works.

Hugging Face also describes a setting called guidance scale, which decides how much weight the prompt gets. The pipeline runs the model both with and without the prompt and pushes the result in the direction of the difference. Its example uses a value of 7.5.

Working in a smaller space

Doing all this on full-size images is costly. Rombach, Blattmann, Lorenz, Esser and Ommer addressed this in a paper first posted in December 2021 and presented at CVPR 2022. They ran diffusion in the compressed space of a pretrained autoencoder, instead of on raw pixels. The paper reports that training powerful pixel-based models can consume hundreds of GPU days, and that working in this latent space cuts the cost while keeping detail.

Stable Diffusion is built this way. According to Hugging Face, it has three separate pretrained models:

• A text encoder that turns the prompt into embeddings
• A U-Net that removes noise from the compressed picture, guided by the text
• A VAE, whose encoder compresses images and whose decoder turns the compressed form back into pixels

In the Hugging Face example, a 512 by 512 image is handled as a latent whose height and width are one eighth of that, because the VAE has three down-sampling layers. The denoising loop works on that small grid, and only at the end does the decoder expand it to the full picture.

Where the idea came from

Wikipedia says diffusion models were introduced in 2015, drawing on ideas from non-equilibrium thermodynamics, and that a 2020 paper on the Denoising Diffusion Probabilistic Model improved the method. The latent version followed in 2021. Newer systems keep changing the parts; for example, one source notes that Stable Diffusion 3 moved away from the U-Net to a transformer design.

What this explains about the results

The process makes some behaviour easier to understand. The system does not look up and paste pictures. It builds each image from noise using patterns learned in training, so it can produce things that are new but not always correct. Fine details can go wrong because the image is created at a compressed scale first. This is a general description; individual products may differ in architecture and are not all public.

Sources

• Diffusion model, Wikipedia
• High-Resolution Image Synthesis with Latent Diffusion Models, arXiv
• Understanding pipelines, models and schedulers, Hugging Face Diffusers documentation

Image credits: Cover photo: Image noise in low light sample Canon 100D at ISO 12800_at_ISO_12800.jpg) by Rodrigo.Argenton, CC BY-SA 4.0; Picture in the text: Artificial Neural Network with Chip by mikemacmarketing / photo on flickr, CC BY 2.0. Pictures from Wikimedia Commons.