DETR — detection as set prediction

Every detector before DETR produced a list of overlapping candidate boxes and then deleted most of them with a hand-written rule. DETR instead predicts a set of fixed size, and trains it with an assignment problem solved exactly at every step. That single change removes anchors, removes non-maximum suppression, and makes the loss end-to-end — at the cost of a notoriously slow convergence that the follow-up papers exist to fix.

Symbols used here are collected in the Nomenclature.

Prerequisites: attention and transformer blocks, ViT backbones. The companion page RF-DETR from scratch implements everything below and reports the measured tensor shape at every stage.

1 · Why set prediction at all

Consider what an anchor-based detector must do. It tiles the image with thousands of reference boxes, scores each one, and then faces an awkward fact: a single object fires many anchors. The output is therefore not the answer — it is a cloud of near-duplicates that must be thinned. Thinning is non-maximum suppression, a greedy loop with an IoU threshold, and it sits outside the network where no gradient reaches it.

Set prediction attacks the cause rather than the symptom. If the model emits exactly $Q$ predictions and the training objective forces a one-to-one correspondence between predictions and ground-truth objects, then duplicates are not suppressed — they are never generated, because producing two boxes for one object is penalised directly by the loss.

2 · The objective

The model outputs a fixed-size set of $Q$ predictions (DETR uses $Q = 100$; the implementation on the companion page uses the same). Let

$$ \hat y = \{\hat y_j\}_{j=1}^{Q}, \qquad \hat y_j = (\hat p_j, \hat b_j) \tag{1} $$

where $\hat p_j \in \Delta^{K}$ is a distribution over $K$ classes plus a special ∅ ("no object") class, and $\hat b_j \in [0,1]^4$ is a box as $(c_x, c_y, w, h)$ normalised by image size. The ground truth is a set of $N$ objects $y = \{y_i\}_{i=1}^{N}$ with $N \le Q$, padded with ∅ up to length $Q$ so that both sides are the same size and a permutation between them is well defined.

3 · Bipartite matching

The predictions arrive in no meaningful order. Query 57 is not "the query for the dog". So before any loss can be computed, we must decide which prediction is graded against which object. DETR picks the assignment of minimum total cost:

$$ \hat\sigma = \operatorname*{arg\,min}_{\sigma \in \mathfrak{S}_Q} \sum_{i=1}^{Q} \mathcal{L}_{\text{match}}\big(y_i, \hat y_{\sigma(i)}\big) \tag{2} $$

This is the linear assignment problem, solved exactly by the Hungarian algorithm in $O(Q^3)$. With $Q = 100$ that is negligible next to a single forward pass through the backbone.

The matching cost for a non-∅ target combines class and geometry:

$$ \mathcal{L}_{\text{match}}(y_i, \hat y_j) = -\underbrace{\hat p_j(c_i)}_{\text{class}} + \underbrace{\lambda_{\text{L1}}\,\|b_i - \hat b_j\|_1}_{\text{position}} + \underbrace{\lambda_{\text{giou}}\big(1 - \operatorname{GIoU}(b_i, \hat b_j)\big)}_{\text{overlap}} \tag{3} $$

Rows matched to a padded ∅ target contribute a constant and are excluded from the cost matrix in practice — it is cheaper to build a $Q \times N$ matrix and let the solver return $\min(Q, N)$ pairs, which is what the reference implementations and the companion code do.

4 · The Hungarian loss

With $\hat\sigma$ fixed, the loss is a sum of three terms:

$$ \mathcal{L}_{\text{Hung}}(y, \hat y) = \sum_{i=1}^{Q} \Big[ -\log \hat p_{\hat\sigma(i)}(c_i) + \mathbb{1}_{\{c_i \ne \varnothing\}}\, \mathcal{L}_{\text{box}}\big(b_i, \hat b_{\hat\sigma(i)}\big) \Big] \tag{4} $$ $$ \mathcal{L}_{\text{box}}(b, \hat b) = \lambda_{\text{L1}}\,\|b - \hat b\|_1 + \lambda_{\text{giou}}\big(1 - \operatorname{GIoU}(b, \hat b)\big) \tag{5} $$

Note the asymmetry. The classification term runs over all $Q$ predictions, including those matched to ∅ — that is what teaches the model to stay quiet. The box terms are gated by $\mathbb{1}_{\{c_i \ne \varnothing\}}$, because a prediction matched to "no object" has no box to be graded against.

4.1 The ∅ class dominates, and must be down-weighted

COCO averages about 7 objects per image. With $Q = 100$, roughly 93% of targets are ∅. A model that predicts "no object" for everything therefore already scores well on plain cross-entropy. DETR down-weights the ∅ class by a factor of 10:

$$ w_c = \begin{cases} 1 & c \ne \varnothing \\ 0.1 & c = \varnothing \end{cases} \tag{6} $$

4.2 Why both L1 and GIoU

Each box loss alone has a specific defect, and they are complementary:

GIoU repairs IoU's dead zone by subtracting the share of the smallest enclosing box $C$ that neither box covers:

$$ \operatorname{GIoU}(A, B) = \underbrace{\frac{|A \cap B|}{|A \cup B|}}_{\text{IoU}} - \frac{|C \setminus (A \cup B)|}{|C|}, \qquad \operatorname{GIoU} \in [-1, 1] \tag{7} $$

5 · Architecture and tensor flow

The original DETR is a CNN backbone, a transformer encoder over the flattened feature map, and a transformer decoder over $Q$ learned object queries. The queries are the mechanism that makes the output a set: there are exactly $Q$ of them, they are permutation-symmetric, and each one ends up responsible for at most one object.

For a $640 \times 640$ input at stride 32 the token grid is $20 \times 20 = 400$, so encoder self-attention costs $400^2 = 160{,}000$ pairwise scores per head per layer. That is affordable. The problem appears when you want higher resolution: at stride 8 the grid is $80 \times 80 = 6400$ and self-attention costs $6400^2 \approx 41$M scores — a 256× increase for a 4× finer grid, because the cost is quadratic in token count.

6 · Deformable attention

Standard attention computes a weight for every (query, key) pair and is therefore quadratic in the number of keys. Deformable attention replaces that with a learned sparse sample: each query predicts a handful of offsets from its own reference point and attends only to those locations, interpolated bilinearly from the feature map.

$$ \operatorname{DeformAttn}(z_q, p_q, x) = \sum_{m=1}^{M} W_m \Bigg[ \sum_{l=1}^{L} \sum_{k=1}^{P} A_{mlqk} \cdot W'_m\, x^l\big(\phi_l(p_q) + \Delta p_{mlqk}\big) \Bigg] \tag{8} $$

The offsets $\Delta p$ are real-valued, so $x^l(\cdot)$ is a bilinear interpolation of the four surrounding pixels. That keeps the whole operation differentiable with respect to the offsets, which is what allows the sampling pattern itself to be learned.

6.1 The cost difference, concretely

Cross-attention work per decoder layer, for $Q$ queries against $S$ tokens per level over $L$ levels:

Those are the measured numbers from the companion implementation's shape report, for a $644 \times 644$ image with a patch-14 ViT tapped at four depths. The figure is reported there because that implementation deliberately uses dense attention for readability, and the honest thing is to state what that choice costs.

7 · The convergence problem

DETR's headline weakness is training time: the original paper needed 500 epochs on COCO where Faster R-CNN reaches comparable accuracy in around 12. Three causes are usually identified, and the follow-up papers map almost one-to-one onto them.

8 · The lineage, and where RF-DETR sits

9 · References

• Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko, End-to-End Object Detection with Transformers, ECCV 2020 — the DETR paper; §2–§4 above follow its formulation.
• Zhu, Su, Lu, Li, Wang, Dai, Deformable DETR: Deformable Transformers for End-to-End Object Detection, ICLR 2021 — source of equation (8).
• Rezatofighi et al., Generalized Intersection over Union, CVPR 2019 — GIoU and equation (7).
• Li et al., DN-DETR: Accelerate DETR Training by Introducing Query DeNoising, CVPR 2022.
• Liu et al., DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR, ICLR 2022.
• Zhang et al., DINO: DETR with Improved DeNoising Anchor Boxes, ICLR 2023.
• Zhao et al., DETRs Beat YOLOs on Real-time Object Detection (RT-DETR), CVPR 2024.
• Kuhn, The Hungarian method for the assignment problem, 1955 — the solver behind equation (2).

Related: RF-DETR from scratch · RF-DETR notebook · GroundingDINO · DINOv2 · Architecture modules · Nomenclature