Open problems & publishing directions

The problems in this field are not evenly distributed in difficulty. A few require frontier compute and are effectively closed to outside contribution; many more are measurement and analysis problems that need a careful experiment and a modest GPU budget. This page sorts the open questions by tractability and states, for each, what the experiment would actually be.

Symbols used on this page are defined in the Nomenclature → learning.

The structural observation worth building a research plan around. Almost everything in the model table was established by scaling — more data, more parameters, higher resolution. Comparatively little was established by controlled experiment. That asymmetry is an opportunity: the field has a large backlog of unverified causal claims, and verifying one cleanly is a publishable contribution that does not require training a frontier model.

1 · Sorted by what it costs to attempt

Tier	Compute	Kind of contribution	Examples below
A · Analysis	Inference only, 1 GPU	Measure something nobody measured; build a diagnostic; show a claim is confounded	§2, §3, §8
B · Light training	LoRA / connector only, 1–8 GPUs	Ablate one design choice properly; new conditioning module; new preference data	§4, §5, §6
C · Full training	Pretraining a VLM, 64+ GPUs	New architecture or data recipe at scale	§7, unified models
D · Frontier	Thousands of GPUs	Effectively closed; read the reports and build on the released weights	—

2 · Evaluation is the biggest open problem Tier A

This is the field's weakest link and simultaneously its cheapest entry point. Four concrete gaps, each of which is a paper:

Gap	The experiment	Why it would be cited
Prompt-format variance is unquantified	Take 10 open models × 6 benchmarks × 8 prompt templates × fixed decoding. Report the variance attributable to template alone, and whether model rankings change	Everyone suspects rankings are unstable; nobody has published the number. If rankings flip, that invalidates a large body of comparisons
No degenerate-optimum-free editing metric	Per editing §8: sweep guidance, plot edit-success against preservation, and define an area-under-curve summary analogous to average precision	Editing papers currently quote single operating points of their own choosing. A standard curve-based summary would be adopted quickly
Contamination is asserted, not measured	For models with open data (OpenCLIP, Molmo, Idefics), directly search the corpus for benchmark items and measure the accuracy gap between contaminated and clean subsets	Converts an unfalsifiable worry into a calibrated correction factor
LLM-judge reliability for vision	Collect human labels on a few thousand free-form VQA answers and measure judge agreement, broken down by answer length, by whether the judge shares a family with the model under test, and by question type	VLM-as-judge is spreading fast with almost no validation

Why evaluation papers are undervalued and why that is an opportunity. They do not produce a number that goes up, so they feel less exciting to write. But they are disproportionately cited, because every subsequent paper in the area has to adopt or rebut them — POPE and Winoground are both small papers that reshaped how their subfields report results.

3 · What does the encoder actually destroy? Tier A

Several design decisions in the model table were made on aggregate scores without anyone characterising the underlying information loss. These are clean, cheap experiments.

The Q-Former bottleneck. Per architecture §3.1, the field abandoned learned-query connectors on benchmark scores alone. Train probes to recover attributes — object count, relative position, colour, text content — from $k$ learned queries as $k$ varies, and plot what survives against $k$. This gives a quantitative answer to "what did the bottleneck cost", which the literature currently lacks.
The MMVP failure, localised. MMVP shows CLIP embeds visually distinct images nearly identically. Measure the embedding distance distribution per failure category, and test whether a different tower (SigLIP, DINOv2) separates the pairs the CLIP tower collapses. If DINOv2 — trained without language — separates them better, that is a strong, surprising, well-evidenced claim about what language supervision costs.
Resolution versus everything else. Hold the LM and data fixed and sweep only the encoder resolution. Report per-category deltas. The hypothesis to test is that OCR and small-object gains are entirely resolution-driven while reasoning gains are not — widely assumed, not cleanly shown.

A note on what makes these publishable. Each is framed as a falsifiable hypothesis with a measurement, not as "we tried X and scores improved". The second form is crowded and hard to distinguish; the first is rare and is what a reviewer is actually looking for.

4 · Compositionality Tier B

The clearest known defect. As derived in architecture §2.1, the contrastive objective never requires word order to matter, so CLIP-family models behave like bags of words — and on Winoground many strong models score below chance at the group level.

Direction	Status	Open question
Hard negatives with swapped word order	Tried; helps on the probe benchmarks	Does it transfer, or does the model learn the probe format? Rarely tested off-benchmark
Scene-graph or structured supervision	Partially explored	How much structured data is needed before relational accuracy moves at all?
Generative-only training (no contrastive tower)	Fuyu-style, under-studied	Does skipping contrastive pretraining avoid the bag-of-words failure, at the cost of data efficiency? This is a clean, interesting trade nobody has mapped
Text-encoder choice	Imagen's T5 result suggests it matters	Per §5.3: is CLIP-text's invariance the cause? Swap encoders, hold everything else, measure relational accuracy

5 · Hallucination and calibration Tier B

Mechanisms are catalogued in VQA §5. What remains genuinely open is less about reducing hallucination than about knowing when to abstain.

Abstention is untrained. Instruction data almost never contains "I cannot tell from this image", so the behaviour does not exist to be elicited. Building a dataset where the correct answer is abstention — occluded objects, cropped-out evidence, genuinely ambiguous scenes — and measuring whether abstention generalises, is a well-defined Tier B project.
Calibration is barely studied for VLMs. Does token probability correlate with correctness on visual questions the way it does on text? If not, every confidence-thresholded deployment is unsound. Measuring this is inference-only.
Separating "cannot see" from "will not say". A model failing on a small object may have genuinely unresolved pixels, or may have resolved them and been overridden by a language prior. Those need different fixes, and no standard diagnostic distinguishes them — resolution sweeps plus attention attribution could.
Grounded generation as a structural fix. Requiring the model to emit a box with each object claim makes the claim checkable by a detector. Molmo's pointing output is a step; a full claim-level grounding protocol is not yet standard.

6 · Editing control Tier B

Problem	Why it is still open	Approach
Instruction data inherits its generator's limits	Per editing §4, InstructPix2Pix-style triples can only contain edits the teacher pipeline could already make	Build triples from real paired data — video frames, before/after photographs, rendered scenes with programmatic changes — and compare precise-local-edit ability
Counting and quantity edits fail	"Remove one of the three cups" needs enumeration, which diffusion conditioning has no mechanism for	Compose with a detector: ground the referent, derive the mask, then inpaint. A pipeline contribution, but a measurable one
Multi-turn editing drifts	Each edit re-encodes, so VAE error and style drift accumulate — the same mechanism as outpainting drift	Quantify drift per turn; test whether pixel-space compositing bounds it
Identity preservation	IP-Adapter and LoRA help but are not reliable across pose and lighting change	Needs an identity metric that is not itself a CLIP similarity

7 · Architecture questions that need real training Tier C

Is the encoder necessary at all? Fuyu projects raw patches with no vision tower. Nobody has published a controlled comparison — same data, same LM, encoder versus no encoder — so the data- efficiency cost of discarding contrastive pretraining is unknown.
Tiling versus native dynamic resolution. Both are in production; they have never been compared with everything else held fixed. Tiling splits objects at boundaries, which is a specific, testable prediction about which questions it fails.
Do unified understand-and-generate models actually help each other? Chameleon and Transfusion assume shared representations benefit both directions. The ablation — train the same model on understanding only, generation only, and both — would settle whether the unification is synergy or just convenience.
Video is still mostly images in a row. Temporal modelling remains shallow; long-horizon causal reasoning over video is largely unsolved and is where the next capability jump probably is.

8 · If you are writing one of these papers Tier A

A checklist assembled from the failure modes catalogued across this section. Most rejected VLM papers fail on these rather than on the idea.

CLAIM HYGIENE [ ] One sentence stating what is new, and what would falsify it. [ ] The claim is localised: which component, which capability. CONTROLS (the single most common omission) [ ] Question-only / no-image baseline for any VQA claim. [ ] Input-unchanged and ignore-input baselines for any editing claim. [ ] Only ONE thing differs between your model and the baseline. If encoder AND data AND LM differ, the comparison attributes nothing. PROTOCOL [ ] Prompt templates published verbatim. [ ] Decoding fixed and stated (greedy, temperature 0). [ ] Input resolution stated. It is often the dominant factor. [ ] CircularEval or an explicit note that position bias is unmeasured. EVALUATION [ ] At least one benchmark postdating the baselines' data cutoff. [ ] Per-category results, not only an aggregate. [ ] A hallucination probe with the yes-ratio, not just accuracy. [ ] Editing results as a CURVE over guidance, not one point. [ ] Seeds: multiple runs, variance reported. A 1-point gap on one seed is not a result. HONESTY [ ] A failure-mode section with real examples, not cherry-picks. [ ] Compute and data stated, so others can judge comparability. [ ] Negative results from the same study included.

The two items that do the most work. First, change one thing at a time — the confounded-comparison problem from models §caveats is endemic, and a genuinely controlled ablation stands out immediately. Second, report the control baselines — they are nearly free, they are the first thing a sceptical reader reaches for, and if your result survives them you should say so loudly.

9 · Orientation reading

Grouped by what each is useful for, rather than chronologically. Search by title; venue and year are given to disambiguate.

Area	Papers that define the area
Contrastive foundations	Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021) · Sigmoid Loss for Language Image Pre-Training (SigLIP, 2023) · Scaling Up Visual and Vision-Language Representation Learning (ALIGN, 2021)
Connector designs	Flamingo: a Visual Language Model for Few-Shot Learning (2022) · BLIP-2 (2023) · Visual Instruction Tuning (LLaVA, 2023) · Improved Baselines with Visual Instruction Tuning (LLaVA-1.5, 2023)
The failure literature read these first if you want a problem	Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022) · When and why vision-language models behave like bags-of-words (ARO, 2023) · Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (MMVP, 2024) · Evaluating Object Hallucination in Large Vision-Language Models (POPE, 2023)
Benchmarks worth using	MMMU (2023) · MMBench (2023) · MathVista (2023) · BLINK (2024) · Making the V in VQA Matter (VQAv2, 2017) · GQA (2019)
Generation & editing	High-Resolution Image Synthesis with Latent Diffusion Models (2022) · Classifier-Free Diffusion Guidance (2022) · Prompt-to-Prompt Image Editing with Cross Attention Control (2022) · InstructPix2Pix (2023) · Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet, 2023) · Null-text Inversion (2023) · Scaling Rectified Flow Transformers (SD3, 2024)
Efficiency	FlashAttention (2022) and FlashAttention-2 (2023) · LoRA (2021) · QLoRA (2023)
Open recipes to replicate	Molmo and PixMo (2024) · What matters when building vision-language models? (Idefics2, 2024) · InternVL series · Qwen2-VL (2024)

If you read four papers before starting, read the failure row. Winoground, ARO, MMVP and POPE between them define what current models cannot do and — more usefully — demonstrate the style of contribution that lands: a small, sharply-designed diagnostic that localises a defect, rather than an incremental score.

10 · References

The figures on this page are original redrawings in the site's palette, not reproductions — the architectures are credited to the papers below. Where a blog post is the clearest explanation of something, it is cited as such.

10.1 Starting points for each direction

Thrush et al., Winoground (CVPR 2022); Yuksekgonul et al., ARO (ICLR 2023); Tong et al., MMVP (CVPR 2024); Li et al., POPE (EMNLP 2023) — the four papers that best demonstrate the style of contribution recommended in §9.
Bordes et al., An Introduction to Vision-Language Modeling (2024) — for locating a gap.
Laurencon et al., What matters when building vision-language models? (2024) — the closest thing to a controlled ablation study in this area, and a model for how to run one.
Deitke et al., Molmo and PixMo (2024) — open data, which is what makes contamination measurable at all.