Image editing, inpainting and outpainting

Generating an image from nothing is the easy problem. Changing one you already have is harder, because success now has two requirements that pull against each other: the edit must happen, and everything else must stay exactly as it was. This page covers the five mechanisms the field actually uses, treats inpainting and outpainting properly, and is blunt about the fact that the central obstacle to progress here is evaluation, not generation.

Symbols used on this page are defined in the Nomenclature → learning.

The latent-diffusion machinery underneath — the forward and reverse processes, classifier-free guidance, the VAE and the 9-channel inpainting UNet — is derived on the Stable Diffusion page and is not repeated here. This page is about control.

The whole problem, in one sentence. An edit is a constrained generation: maximise agreement with the instruction subject to minimal change elsewhere. Every mechanism below is a different way of expressing that constraint, and the constraint is what is hard — not the generation.

1 · Five mechanisms

Mechanism	What the user supplies	How the constraint is imposed	Locality	Representative
1 · Mask-based	Image + mask + prompt	Hard — pixels outside the mask are copied	Guaranteed	SD inpainting, FLUX Fill
2 · Attention control	Image + two prompts	Reuse or inject cross-attention maps from the source generation	Soft	Prompt-to-Prompt, MasaCtrl
3 · Inversion-based	Real image + prompts	Recover the noise that would generate the image, then re-denoise with a new prompt	Soft	DDIM inversion, Null-text, pix2pix-zero
4 · Instruction-tuned	Image + plain instruction	Learned from synthetic (before, after, instruction) triples	Learned, not enforced	InstructPix2Pix, MagicBrush, Emu Edit
5 · Structural / in-context	Image + reference or control signal	Extra conditioning branch, or the reference as context tokens	Structural	ControlNet, IP-Adapter, FLUX Kontext

2 · Inpainting — the mask-based mechanism

The naive approach is to blend in latent space at every denoising step: run the normal reverse process, but after each step overwrite the unmasked region with a correctly-noised copy of the original.

$$ \mathbf z_{t-1} \;\leftarrow\; \underbrace{\mathbf m \odot \mathbf z^{\text{gen}}_{t-1}}_{\text{generated, inside}} \;+\; \underbrace{(1-\mathbf m) \odot \mathbf z^{\text{orig}}_{t-1}}_{\text{known, outside}}, \qquad \mathbf z^{\text{orig}}_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\mathbf z_0 + \sqrt{1-\bar\alpha_{t-1}}\,\boldsymbol\epsilon $$

This works, and it needs no training — but it has a characteristic failure: the model inside the mask only learns about the outside through the blend, not through the network, so it tends to produce something plausible in isolation that does not match its surroundings. Seams and lighting mismatches are the signature.

The inpainting UNet takes nine input channels: four for the noisy latent, four for the VAE-encoded masked image and one for the downsampled mask; the decoded result is composited back onto the original pixels. — The nine-channel input, and the pixel-space composite that has to follow it. The five added channels are zero-initialised, so the model begins as an exact copy of the text-to-image checkpoint.

2.1 The trained alternative: a 9-channel UNet

A dedicated inpainting model gives the network the context explicitly, by widening its first convolution:

Channels	Content
4	The noisy latent being denoised, as usual
4	The masked image, VAE-encoded — what is actually around the hole
1	The mask, downsampled to latent resolution

The checkpoint is initialised from the text-to-image weights with the five new channels zeroed, so it starts out behaving exactly like the base model and learns to use the new inputs. Training uses (image, mask, prompt) triples with randomly generated masks — and the mask distribution matters: train only on small rectangles and the model will not handle a large irregular region.

Always composite the result back onto the original pixels. Even with a perfect mask, the pipeline VAE-decodes the whole latent, so every pixel carries the autoencoder's reconstruction error — including the region you never touched. The fix is to decode and then $\mathbf x_{\text{final}} = \mathbf m_{\text{px}} \odot \mathbf x_{\text{dec}} + (1-\mathbf m_{\text{px}}) \odot \mathbf x_{\text{orig}}$, with a feathered mask to avoid a hard edge. A surprising number of pipelines skip this and silently degrade the entire image.

3 · Outpainting — the same mechanism, a harder problem

Outpainting is inpainting with the mask inverted: the known region is the original image and the unknown region is new canvas around it. Mechanically identical, substantially harder, for three reasons.

Difficulty	Why it is specific to outpainting	Mitigation
One-sided context	An inpainted hole is surrounded by evidence. An outpainted margin has context on one side only, so the problem is far less constrained	Expand in small increments rather than one large jump
Error accumulation	Iterative expansion conditions each step on the previous generation, not on real pixels. Drift compounds — colour, style and perspective wander	Re-anchor on the original at every step; keep the original visible in context
Global geometry	Extending a building requires a consistent vanishing point the model has no explicit representation of	Depth or line conditioning via ControlNet

iterative outpainting, and where it goes wrong [ original ] step 0: real pixels [ orig | new1 ] step 1: conditioned on real [ orig | new1 | new2 ] step 2: conditioned on new1 <- drift starts [ orig | new1 | new2 | new3 ] step 3: conditioned on new2 <- drift compounds the fix: at every step, keep the ORIGINAL in the conditioning window, not merely the most recent generation.

4 · Instruction-based editing

Masks are precise and tedious. The instruction-based line removes them: the user says "make it winter" and the model works out what to change. The problem is training data — there is no natural corpus of (before, after, instruction) triples.

InstructPix2Pix's solution was to synthesise the dataset using two existing models, which is worth understanding because the whole line inherits its properties:

1. GPT-3 generates (caption, edit instruction, edited caption) text triples "a photo of a cat" / "make it a tiger" / "a photo of a tiger" 2. Prompt-to-Prompt + Stable Diffusion turns each text pair into an IMAGE pair that differs only where the instruction says 3. Train a conditional diffusion model on the resulting triples, with TWO guidance scales at inference

Because the model is conditioned on both an image and a text instruction, it needs two guidance scales — one controlling instruction adherence, one controlling faithfulness to the input:

$$ \tilde{\boldsymbol\epsilon} = \boldsymbol\epsilon_\varnothing + s_I\big(\boldsymbol\epsilon(\mathbf c_I) - \boldsymbol\epsilon_\varnothing\big) + s_T\big(\boldsymbol\epsilon(\mathbf c_I, \mathbf c_T) - \boldsymbol\epsilon(\mathbf c_I)\big) $$

with $s_I$ the image scale and $s_T$ the text scale. Tuning these is the locality/fidelity trade, exposed as two knobs: raise $s_T$ and the edit is applied aggressively but the image drifts; raise $s_I$ and the image is preserved but the instruction may be ignored.

The inherited weakness of this whole family. The training pairs were generated by a diffusion model using attention control. So the student can only learn edits the teacher pipeline could already produce, and it inherits the teacher's biases — including its tendency to change more than asked. This is why instruction-based models are good at global style changes ("make it winter") and comparatively weak at precise local ones ("remove the third book from the left"). The limitation is in the data generator, not the architecture, which makes it a tractable target: a better triple-generation process is a publishable contribution on its own.

5 · Inversion — editing a real photograph

Attention-control methods need the attention maps from the image's generation, which you do not have for a real photograph. Inversion recovers them by running the sampler backwards to find the noise that would have produced this image. DDIM is deterministic, so it can be reversed:

$$ \mathbf z_{t+1} = \sqrt{\frac{\bar\alpha_{t+1}}{\bar\alpha_t}}\,\mathbf z_t + \left(\sqrt{1-\bar\alpha_{t+1}} - \sqrt{\frac{\bar\alpha_{t+1}(1-\bar\alpha_t)}{\bar\alpha_t}}\right) \boldsymbol\epsilon_\theta(\mathbf z_t, t, \mathbf c) $$

Inversion is only exact in a limit that is never used in practice. The derivation assumes the ODE step is small enough that $\boldsymbol\epsilon_\theta(\mathbf z_t)\approx\boldsymbol\epsilon_\theta (\mathbf z_{t+1})$. With 50 steps that is already approximate, and — critically — it breaks badly when classifier-free guidance is on, because the guided score is not the score of any single distribution. Since useful editing needs guidance, naive DDIM inversion reconstructs poorly, and the literature is largely a sequence of repairs for this one problem.

Method	Fix	Cost
Null-text inversion	Optimise the unconditional embedding per timestep so the guided trajectory reproduces the image	Per-image optimisation — minutes
Negative-prompt inversion	Set the null embedding to the source prompt analytically instead of optimising	Nearly free; slightly worse
pix2pix-zero	Compute a direction in text-embedding space and regularise cross-attention toward the source	No per-image training
Rectified-flow inversion	Straight ODE paths invert far more stably — a real advantage of SD3/FLUX	Cheap, and the current best option

6 · Structural and reference conditioning

Method	Mechanism	Preserves
ControlNet	A trainable copy of the encoder half takes a control image (Canny, depth, pose, segmentation) and injects features through zero-initialised convolutions	Geometry and structure, while style changes freely
T2I-Adapter	Same goal, a much smaller side network	As above, cheaper, slightly weaker
IP-Adapter	Decoupled cross-attention: a separate attention path for image-prompt tokens	Identity or style from a reference image
LoRA / DreamBooth	Fine-tune on a few images of a subject	A specific subject's identity across any prompt
In-context (FLUX Kontext)	Feed the reference image as context tokens alongside the instruction	Character and scene consistency without per-subject training

Zero-initialisation is the recurring trick, and it is worth recognising. ControlNet's injection convolutions, the inpainting UNet's five new channels, and Flamingo's gated cross-attention all start at exactly zero. The new pathway therefore contributes nothing at step one, the model is initially identical to the pretrained one, and training can only improve on it. It is the standard answer to "how do I add a module to a working model without breaking it", and the same idea appears in adaLN-Zero in DiT.

7 · The locality–fidelity trade

Every mechanism above lands somewhere on one axis: how strongly the unedited region is protected.

strong constraint weak constraint (cannot change (can change the wrong thing) anything) | | |-- hard mask composite | |-- 9-channel inpaint UNet | |-- ControlNet structural | |---------- attention injection | |------------------ inversion + reprompt | |-------------------------- instruction-tuned | |-------------------------------- plain img2img, high noise | moving right: less user effort, more unintended change moving left: more precision, more work per edit

Symptom	Likely cause	Try
The edit did not happen	Image guidance too high, or denoising strength too low	Raise $s_T$, raise strength, make the instruction more explicit
The whole image changed	Text guidance too high; no locality constraint	Lower $s_T$, add a mask, switch to a mask-based method
Visible seam at the mask edge	Hard mask boundary, latent/pixel resolution mismatch	Feather the mask; composite in pixel space; blur at latent resolution
Identity lost ("same person, different face")	No identity conditioning	IP-Adapter or a subject LoRA
Lighting inconsistent with the scene	The inpainted region was generated without enough surrounding context	Use the trained inpainting model, not latent blending; widen the context crop
Outpainted region drifts in style	Conditioning on previous generations	Re-anchor on the original every step; smaller increments

8 · Evaluation — the actual bottleneck

There is no ground truth for "make it winter". That single fact is why editing research is harder to assess than VQA, and why progress is hard to verify.

Metric	Measures	How it misleads
CLIP score	Output–prompt similarity	Satisfied by ignoring the input image entirely and generating the target from scratch
Directional CLIP	Whether the image changed in the same direction as the caption change	Better, since it uses both images — but still CLIP, so blind to the word-order and compositional failures of §CLIP
LPIPS / L1 to input	How little changed	Minimised by doing nothing. Meaningless unless paired with an edit-success term
FID	Distributional realism	Needs thousands of images; says nothing about whether this edit was right
Human preference	What we actually care about	Expensive, noisy, unreproducible across studies
VLM-as-judge	Ask a strong VLM whether the edit was performed	Promising, and inherits every hallucination problem from VQA §5

The structural flaw: the two headline metrics are trivially gamed in opposite directions. Report CLIP score alone and the winning strategy is to ignore the input image. Report LPIPS-to-input alone and the winning strategy is to return the input unchanged. Any single-number editing metric has a degenerate optimum, which is why results must be reported as a curve — sweep the guidance scale and plot edit success against preservation, exactly as one plots a precision–recall curve. A paper quoting one operating point is choosing its own favourable threshold, and a reviewer should ask for the curve.

A concrete, cheap contribution available here. Benchmarks that supply paired data — MagicBrush has human-annotated before/after pairs, EditVal and EmuEdit provide task-typed suites — make it possible to report an actual trade-off curve with a scalar summary, the way average precision summarises a PR curve. Nobody has standardised that summary for editing, and the community visibly needs one.

9 · Practical summary

If you can draw a mask, draw one. A hard constraint beats a learned one, and the trained 9-channel model beats latent blending.
Always composite back in pixel space with a feathered mask. Otherwise the VAE degrades the untouched region.
Outpaint in small steps, re-anchored on the original, or style will drift.
For real photographs, prefer a rectified-flow model — inversion is markedly more stable than with DDIM plus guidance.
Two guidance scales, two failure modes. Nothing changed → raise $s_T$. Everything changed → raise $s_I$ or add a mask.
Report a curve, not a point. Any single editing metric can be gamed by a degenerate strategy.

10 · References

The figures on this page are original redrawings in the site's palette, not reproductions — the architectures are credited to the papers below. Where a blog post is the clearest explanation of something, it is cited as such.

10.1 Editing mechanisms

Meng et al., SDEdit (ICLR 2022) — the noise-and-denoise basis of img2img.
Lugmayr et al., RePaint (CVPR 2022) — the latent-blending inpainting scheme in §2.
Hertz et al., Prompt-to-Prompt Image Editing with Cross Attention Control (ICLR 2023).
Brooks et al., InstructPix2Pix (CVPR 2023) — including the synthetic-triple generation discussed in §4.
Zhang et al., MagicBrush (NeurIPS 2023); Sheynin et al., Emu Edit (CVPR 2024).
Mokady et al., Null-text Inversion for Editing Real Images (CVPR 2023); Parmar et al., pix2pix-zero (SIGGRAPH 2023).
Zhang et al., Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet, ICCV 2023).
Ye et al., IP-Adapter (2023); Ruiz et al., DreamBooth (CVPR 2023); Hu et al., LoRA (ICLR 2022).

10.2 Evaluation

Gal et al., StyleGAN-NADA (2021) — the origin of the directional CLIP loss.
Basu et al., EditVal (2023); Ku et al., ImagenHub (ICLR 2024) — standardised editing evaluation, and honest about how unsettled it is.
Hessel et al., CLIPScore (EMNLP 2021) — and read §8 here for how it is gamed.