ControlNet

advancedwidespread

ControlNet is an add-on for image-generating AI that allows you to precisely define how the image is constructed using a sketch, pose, or outline.

In plain terms

If you describe what you want to an image generator only with text, the composition largely depends on luck: where the figure stands, which way it looks, where the horizon is. ControlNet significantly reduces this uncertainty by allowing you to provide a 'control image' alongside the text.

It is like a coloring book: the outlines are given, and the machine "colors them in", meaning it invents the details, style, and lighting within the given structure. A control image can be a hand-drawn sketch, a stick-figure pose skeleton, a photo's edge map, or a depth map showing what is near and what is far.

With simple text-to-image generation, you only specify what should be in the image. With ControlNet, you also specify where and in what form, making it a favorite tool among designers, architects, and illustrators.

How it works

You start with an image: it can be your own drawing, photo, or 3D sketch. A pre-processing program extracts the essential structure from it, such as edges, posture, or depth. With a ready line drawing, this step can be omitted.
This structural map will be the control image, which the image generation model receives along with your text description.
ControlNet is a separately trained auxiliary network alongside the original model: it does not rewrite the original but 'whispers' to it step by step so that the image conforms to the specified structure.
During image generation, the model gradually cleans the image from noise, while ControlNet steers it toward the control image at every step.
In most programs, you can adjust with a slider how strictly the machine should follow the sketch, and you can combine multiple controls, such as posture and depth simultaneously.

An everyday example

You photograph your living room and ask the machine to show it in Scandinavian style. ControlNet preserves the location of walls, windows, and furniture; only colors, materials, and mood change, so you truly see your own room in a new outfit.

Who is affected

marketerYou can create campaign images so that the product's position, the model's posture, and the composition remain the same in every variation. → Marketing

industrial worker, engineerAs an architect or product designer, you can get a visual rendering from a line sketch in minutes that follows the original proportions. → Industry

developerIf you integrate image generation into an application, you can give users predictable, controllable results. → Code

everyoneSome AI images seen online are built on the structure of real photographs, so a familiar setting does not prove the image is authentic.

A common misconception

MythControlNet is a standalone image generator that competes with Midjourney or Stable Diffusion.

In realityNot a standalone program but an add-on built on top of an existing image generation model. The base model still creates the image; ControlNet only defines the structure.

Limits and risks

It follows the structure but may err in details: hands, text, and small objects can break just like in any AI-generated image.
A separately trained version is required for each model family, and typically for each control type as well, although there are so-called "union" versions that handle multiple types together. Therefore, not every combination is available or works equally well.
If you set it too strict, the image becomes rigid and lifeless; if too loose, the machine ignores the sketch.
It can also copy the structure of someone else's photo or drawing, which raises copyright and personality rights issues.

Similar, but not the same

Image-to-image generation (img2img) – Img2img takes the entire original image (colors, patches) as a starting point, while ControlNet uses only an extracted structural layer, such as outlines or pose.
LoRA – LoRA teaches the model a style, character, or object, while ControlNet directs the layout of a specific image.
ControlNet industrial network – The same name is also used for a communication standard in factory automation, which has nothing to do with image generation.

Related concepts

Broader

Image generation, Diffusion model, Generative AI

Narrower

Pose control, Depth map, Edge detection

On the edge

Stable Diffusion, LoRA, image-to-image generation, Inpainting, Prompt

A brief history

ControlNet was introduced in February 2023 by Stanford University researchers Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. The study won the Marr Prize for the best paper at the ICCV conference later that year. The open-source solution quickly became one of the basic tools of the Stable Diffusion community.

Where things stand

As of late 2026, open image-generation tools include this as standard, and the idea has outgrown the original solution. Newer image and video models often offer similar structural control built in.

Status: 2026 November

Also known as

ControlNet · Control Net · ControlNet-modell · ControlNet control · control network · conditional control · conditional control · ControlNet model

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.