Vision-Language-Action models & task graphs

A Vision-Language-Action (VLA) model closes the autonomy loop: it takes pixels and a language instruction and outputs robot actions directly. It is the point where everything on this site converges — perception features, language grounding, and control — into a single learned policy.

Symbols used on this page are defined in the Nomenclature → deep learning & training.

OpenVLA Notebooks in deeplearning/notebooks/ (VLA task-graph, real-data pipeline, LoRA fine-tuning, ViT, 3D patch embedding)

The core idea

A pretrained vision encoder tokenizes the image; those visual tokens are concatenated with the tokenized instruction and fed to an LLM backbone. Crucially, the action is emitted as a sequence of discrete tokens (binned end-effector deltas), so the whole thing is trained like next-token prediction — reusing all the LLM machinery.

Actions as tokens

Continuous actions (Δx, Δy, Δz, Δroll, …, gripper) are discretized into bins and mapped to spare vocabulary tokens. The policy then autoregressively predicts the action tokens, decoded back to continuous commands:

$$ a_t = \text{detokenize}\big(\arg\max \pi_\theta(\cdot\mid \text{image}, \text{instruction}, a_{<t})\big) $$

Because the vision tokens go through the same transformer as the words, the same attention primitive lets an action query attend to the relevant image regions and instruction words:

Task graphs

For long-horizon tasks, a single reactive policy is brittle. A task graph decomposes an instruction into a directed graph of sub-skills (pick, move, place) with pre/post-conditions; a high-level planner walks the graph while a VLA executes each node. This mirrors classical GNC guidance-vs-control layering: a planner sets waypoints, a controller tracks them.

Adapting with LoRA

Just as with LLMs, VLAs are adapted to new robots/tasks with parameter-efficient LoRA fine-tuning on demonstration data — freezing the big backbone and training small adapters, which keeps the pretrained visuo-linguistic knowledge intact.

Toward 3D-aware policies

The 3D patch-embedding notebook extends the ViT patch idea into volumes — the same lift from 2D tokens to 3D structure that powers the occupancy and BEV representations in the Perception section. 3D-aware tokens give a policy metric spatial understanding, not just image appearance.