Vision-Language Models

A working research survey. The field has converged on a small number of structural choices — how pixels become tokens, what connects the vision tower to the language model, whether generation is autoregressive or diffusive — and almost every model is a point in that design space. This section maps the space, then goes deep on the two tasks where the open problems are most tractable: visual question answering and image editing.

Symbols used in this section are defined in the Nomenclature → learning.

Written to support original work, not just to summarise. The emphasis throughout is on where the claims are weak — which benchmarks are saturated, which metrics measure the wrong thing, which comparisons are confounded by data rather than architecture. Those are the gaps worth writing a paper about, and they are collected in open problems & publishing directions.

Reference

Start here. The first page is the comparison table the rest of the section refers back to; the second is a dictionary of the modules the table's "architecture" column names.

📊

Model comparison table

Every major family in one scrollable table: parameter counts, vision encoder, connector, language model, training data, reported benchmarks, special modules, and whether it can edit or only describe.

CLIP → FLUXsizesdatasetsbenchmarks

🧩

Architecture modules

What each named block actually does and why it was introduced: CLIP and SigLIP objectives, ViT and Swin, FPN, Q-Former and perceiver resamplers, flash attention, VAEs, latent diffusion, MMDiT and rectified flow.

CLIPSwinQ-FormerMMDiT

Model-family deep dives

Two pages that derive the maths of the components everything else is built from, and compare the state of the art within each family.

🔗

CLIP, SigLIP & BLIP

The InfoNCE and pairwise-sigmoid objectives derived, what the learned temperature does, why the loss makes these models blind to word order, BLIP's captioner–filter bootstrap and the Q-Former's rise and fall.

InfoNCEsigmoid lossQ-FormerSOTA

🌀

Stable Diffusion deep dive

Forward and reverse processes, the score view, v-prediction and the zero-terminal-SNR bug, samplers as ODE integration, why guidance is a hack, and rectified flow with MMDiT.

DDPMCFGrectified flowMMDiT

Task deep dives

Two tasks, chosen because they are where the field's self-assessment is weakest: VQA because the metric is older than the models and rewards the wrong behaviour, and editing because there is no ground truth at all.

❓

Visual question answering

The task taxonomy, the benchmark landscape from VQAv2 to MMMU, how the accuracy metric actually works, why language priors let a blind model score well, and the hallucination literature.

VQAv2GQAMMMUPOPE

🖌️

Image editing, inpaint & outpaint

Five mechanisms for changing an image you already have — mask-based, attention control, inversion, instruction-tuned, and in-context — plus the locality/fidelity trade and why evaluation is the bottleneck.

InstructPix2PixControlNetinversionoutpaint

Research

🔬

Open problems & publishing directions

Where the unsolved problems are, which ones are tractable with modest compute, what the standard baselines and evaluation protocols are, and the failure modes a reviewer will ask about.

evaluationhallucinationefficiencycompositionality

Already on this site

Three existing pages are prerequisites rather than duplicates, and this section links to them instead of repeating them:

Stable Diffusion — the latent-diffusion maths, CFG and the inpainting UNet.
GroundingDINO — Swin backbone and contrastive text–region alignment.
OpenEMMA — a VLM used as a driving policy, with measured failure analysis.

One page of orientation

If you read nothing else in this section, this is the shape of it. Four questions separate essentially every model in the table:

Question	The options actually taken	Consequence
How do pixels become tokens?	Frozen ViT patch grid · tiled "any-resolution" crops · native dynamic resolution · raw patches straight into the decoder, no encoder at all	Sets the ceiling on fine detail. A 336×336 encoder cannot read small text no matter how good the language model is.
What joins vision to language?	Linear or MLP projection · cross-attention · a learned query bottleneck (Q-Former, perceiver resampler) · nothing, shared vocabulary	Decides the token budget and therefore the cost. A bottleneck fixes the token count; a projection makes it grow with resolution.
Contrastive or generative?	Dual-encoder trained with InfoNCE or a sigmoid loss · decoder producing text · both	Dual encoders retrieve and classify cheaply but behave like bags of words; decoders reason but cannot be used as an embedding index.
If it generates images, how?	Latent diffusion with a UNet · diffusion transformer (DiT / MMDiT) with rectified flow · autoregressive over discrete visual tokens	Determines how editing works at all — whether you can invert, mask, or must instruction-tune.

The honest caveat, stated once and meant throughout. Published benchmark numbers for these models are self-reported, evaluated under inconsistent protocols, and move every few months. Prompt format, decoding parameters, whether chain-of-thought was allowed and which split was used all shift results by several points — often more than the gap between two models being compared. Treat every number in this section as indicating a tier, never a ranking, and re-measure before you cite one in a paper.

References

The figures on this page are original redrawings in the site's palette, not reproductions — the architectures are credited to the papers below. Where a blog post is the clearest explanation of something, it is cited as such.

Surveys and orientation

Bordes et al., An Introduction to Vision-Language Modeling (2024) — the most complete survey of the design space.
Li et al., Multimodal Foundation Models: From Specialists to General-Purpose Assistants (2023).
Hugging Face, Vision Language Models Explained blog series — a good practical orientation, and the source of several framing choices here.