RF-DETR from scratch

A deliberately flat PyTorch implementation of an RF-DETR-style detector — frozen DINOv2 backbone, projection, transformer decoder, two heads — written so you can read the data path top to bottom and see what shape the tensor is at every step. Every number on this page was measured by running the code, not written from memory.

Symbols used here are collected in the Nomenclature.

Read DETR — detection as set prediction first for the derivation of bipartite matching and the loss. This page is the implementation of that math, plus the things that only show up when you actually run it.

Where the code lives. /home/nadurthi/repos/deeplearning/rfdetr_tutorial/ — nine files, no config system, no registry, no build_from_cfg. The whole architecture is one file (model.py) with a shape comment on essentially every line.

1 · The files

File	What it is
`model.py`	The entire architecture, one file, shape-annotated
`matcher.py`	Hungarian matching, cost matrix built explicitly
`criterion.py`	The three loss terms
`boxes.py`	Box format conversions, IoU, GIoU
`data.py`	COCO loading, letterbox, and a data-leak detector
`shapes.py`	Runs the model and prints the real measured shapes
`train.py`	Training loop with MLflow logging
`get_coco.sh`	Fetch COCO, verify the splits are disjoint
`setup.sh`	Virtual environment
`test_*.py`	Four test files — see §7

2 · Every tensor shape, measured

Shapes written in prose rot the moment a config value changes. shapes.py instead observes a real forward pass and prints what flowed through. The table below is its output for a batch of 2 at $644 \times 644$:

% python shapes.py backbone : dinov2_vits14 image : 644 x 644, patch 14 token grid : 46 x 46 = 2116 tokens per level levels tapped : [3, 5, 7, 11] -> memory is 4 x 2116 = 8464 tokens d_model 256, heads 8, decoder layers 6, queries 100, classes 91+1

Stage	Shape	Why that shape
input image	`(2, 3, 644, 644)`	$B \times 3 \times H \times W$. 644 is $46 \times 14$ — a whole number of patches.
backbone block 3	`(2, 2116, 384)`	$B \times S \times D$ patch tokens, CLS stripped. $S = (644/14)^2 = 2116$.
backbone block 5	`(2, 2116, 384)`	Same grid, deeper features.
backbone block 7	`(2, 2116, 384)`	Same grid, deeper features.
backbone block 11	`(2, 2116, 384)`	Final block.
position embedding	`(2116, 256)`	Fixed 2-D sine/cosine, $S \times d$, added to every level.
proj level 0–3	`(2, 2116, 256)`	$D = 384 \to d = 256$ by a $1\times1$ (i.e. a `Linear`) + LayerNorm.
memory	`(2, 8464, 256)`	The four levels concatenated: $4 \times 2116 = 8464$.
object queries	`(2, 100, 256)`	$Q$ learned vectors, broadcast over the batch.
decoder output	`(2, 100, 256)`	$Q$ is unchanged by attention — one $d$-vector per query.
class logits	`(2, 100, 92)`	$K + 1 = 92$: 91 COCO ids plus ∅.
box predictions	`(2, 100, 4)`	$(c_x, c_y, w, h)$ through a sigmoid, so $\in [0,1]$.

A ViT has no pyramid, so "multi-scale" means multi-depth. A CNN backbone gives you genuinely different resolutions at different stages. DINOv2 keeps $46 \times 46$ throughout, so tapping blocks 3/5/7/11 gives four views at the same resolution but different levels of abstraction. A learned per-level embedding is added so the decoder can tell them apart after concatenation.

2.1 Where the parameters are

Module	Trainable	Frozen
backbone (DINOv2 ViT-S/14)	0	14.43M
input_proj (4 × Linear+LN)	396.3K	0
query_embed	25.6K	0
decoder (6 layers)	6.32M	0
class_head	23.6K	0
box_head (3-layer MLP)	132.6K	0
Total	6.90M	14.43M

Only 32.3% of the network is trained. The decoder is 92% of the trainable parameters — everything else is plumbing.

3 · The one simplification, and what it costs

This implementation uses dense cross-attention, not true deformable attention. Real deformable attention samples a few learned offsets per query with bilinear interpolation, and needs a custom CUDA kernel to be fast. The pure-PyTorch version is correct but slow and obscures the data flow, so model.py substitutes standard dense attention over the flattened token grid and marks the exact line where it does so.

The honest accounting, printed by shapes.py:

Scheme	Work per decoder layer	Measured
This implementation (dense)	$Q \times L \cdot S = 100 \times 8464$	846,400 scores
True deformable, $P = 4$	$Q \times L \times P = 100 \times 4 \times 4$	1,600 samples
Ratio		529×

So this code does not reproduce the paper's speed claims and should not be cited for them. It does reproduce the architecture, the matching and the loss. A tutorial that quietly diverges from the paper it claims to teach is worse than one that says where it diverges.

4 · Matching, and one non-obvious choice

The cost matrix is built explicitly, $Q \times N$, from the three terms of equation (3). One detail is worth calling out because it looks like a typo if you expect cross-entropy:

# Cost 1: classification. NOTE the probability, not the log-probability. cost_class = -prob[:, tgt_labels] # (Q, n) # Cost 2: L1 between box parameter vectors cost_l1 = torch.cdist(outputs['boxes'][b], tgt_boxes, p=1) # Cost 3: -GIoU cost_giou = -generalized_box_iou(cxcywh_to_xyxy(outputs['boxes'][b]), cxcywh_to_xyxy(tgt_boxes)) C = w_class * cost_class + w_l1 * cost_l1 + w_giou * cost_giou

$-\log p$ is unbounded below, so a single confidently-wrong pair can dominate the sum and distort the whole assignment — and early in training every prediction is confidently wrong. $-p$ is bounded in $[-1, 0]$, so matching stays stable. The loss still uses log-probabilities, because there the job is a strong gradient rather than a stable ranking.

The cost matrix is also checked for non-finite entries before being handed to SciPy. A NaN there does not raise — it silently produces an arbitrary assignment, which is a miserable thing to debug from a loss curve.

5 · The loss in code

The classification term covers all $Q$ queries; the box terms cover only matched ones. The construction that makes that work is to fill the target vector with ∅ and then write the matched entries in:

# Everything starts as "no object" ... target_classes = torch.full((B, Q), self.no_object, dtype=torch.int64) # ... then the matched queries get their real target written in. for b, (pred_i, tgt_i) in enumerate(indices): target_classes[b, pred_i] = targets[b]['labels'][tgt_i] loss_class = F.cross_entropy(logits.reshape(B * Q, -1), target_classes.reshape(B * Q), weight=self.class_weight) # eos_coef on the last class

Box losses are normalised by the number of matched boxes, not by batch size, so the loss scale does not swing with how crowded the images happen to be.

The empty-image case needs care. An image with no annotations yields zero matched boxes. Returning a bare 0.0 there breaks backward() because it carries no grad_fn. The code returns pred_boxes.sum() * 0.0 instead, which is numerically zero but still attached to the graph. The test suite checks exactly this: logits must still receive a gradient (that is how the model learns to stay quiet) while box parameters receive none.

6 · Tests, and what they caught

Four test files, 60-odd assertions. Three findings worth reporting:

6.1 The overfit test is the one that matters

Shape tests prove the code runs. Only an optimisation test proves the gradient points downhill. test_loss.py builds a trivially-learnable model (free parameters producing the output directly) and demands the loss be driven near zero on two fixed images:

step	loss	class	L1	GIoU
0	7.0739	1.7918	0.7333	0.8077
200	0.0430	0.0050	0.0030	0.0115
400	0.0486	0.0023	0.0034	0.0146
600	0.0122	0.0018	0.0010	0.0028
799	0.0017	0.0017	0.000001	0.000005

Final box error is exactly 0.00000 against the targets. If a model cannot overfit two images, nothing learned on COCO afterwards can be trusted.

A plateau that looked like a loss bug was a learning-rate artefact. At a fixed $\text{lr} = 0.1$ this test stalled around 0.10 and was non-monotonic — 0.035 at step 200, back up to 0.15 at step 300. The tempting conclusion is that the loss has a floor. Rather than guess, I held the loss identical and changed only the schedule: cosine decay reached 0.0046, and 1500 steps reached 0.00065 with box error 0. So it was Adam oscillating around the optimum at too large a step, not a defect in the objective. The decay is now part of the test rather than a workaround for it.

6.2 "Frozen" has two failure modes, and both are silent

Setting requires_grad = False is necessary but not sufficient. The classic bug is that model.train() puts the backbone back into training mode, so its normalisation layers keep updating their running statistics and "frozen" quietly stops being true. FrozenDinoV2 overrides train() to hold the ViT in eval(), and the tests assert both halves:

backbone params are all requires_grad=False ok backbone stays in eval() after model.train() ok decoder DOES go to train() after model.train() ok no gradients reach the backbone ok (0 tensors got grads) backbone weights identical after backward ok

6.3 Box transforms are verified by round trip

Letterbox scale, pad offset, normalisation and horizontal flip are four chances to corrupt coordinates in a way that still trains, just to a worse optimum. So rather than trusting the arithmetic, the test inverts the whole pipeline and compares against the raw annotation file: worst error over the sample is 0.000 px. The flip is checked by the property $c_x \mapsto 1 - c_x$ with widths unchanged.

7 · The data bug that would have invalidated everything

The COCO copy already on disk had train2017 as a byte-for-byte duplicate of val2017. Same 5000 files (md5 identical), and instances_train2017.json was a copy of the val annotations — 5000 images and 36781 annotations in both. Training on that and "validating" on it measures nothing at all: the validation curve looks excellent and means precisely nothing.

This is invisible unless something checks, so something checks. data.py exposes assert_splits_disjoint(), and train.py calls it before building any model:

% python train.py --smoke *** SPLIT OVERLAP *** 5000 image ids appear in BOTH train2017 and val2017 (e.g. [139, 285, 632]). The download is broken -- train and val are the same data, so any validation metric is meaningless. Run ./get_coco.sh to fetch the real train2017 split. Refusing to train.

Training can be forced with --allow-split-overlap for plumbing tests, and when it is, the run logs splits_disjoint = False to MLflow — so a contaminated run can never be mistaken for a clean one when the numbers are read back months later.

8 · Training and MLflow

bash setup.sh # venv source .venv/bin/activate bash get_coco.sh # real train2017, with verification mlflow ui --backend-store-uri sqlite:////mnt/mydata/mlflow/mlflow.db --port 5000 & python train.py --epochs 50 --batch 8

Logged per epoch: train and val loss broken out by term (class / L1 / GIoU), matched boxes per batch, learning rate, seconds per epoch, peak GPU memory, and COCO mAP at the end.

Per-term logging is not decoration. A single scalar loss cannot tell you whether the model is failing to classify or failing to localise, and those have opposite fixes. When the total stalls, the breakdown says which term stalled.

8.1 Evidence that it learns

A short deliberately-overfitting run (16 images, 30 epochs) to confirm the loop descends — not an accuracy claim:

epoch	train	class	L1	GIoU	val
13	6.1966	1.3661	0.4551	1.2776	6.0948
19	5.7147	1.1442	0.4320	1.2053	5.5745
24	5.5528	1.1273	0.4036	1.2038	5.3627
30	5.3051	1.1224	0.3583	1.1956	5.3256

Every term falls and the validation loss is monotonic over the run; mAP@50 moved from $4\times10^{-5}$ to 0.026. At 0.7 s/epoch on an RTX 6000 Ada that is a fast feedback loop for experiments.

These numbers are from a run with --allow-split-overlap, before the real train split was fetched. They demonstrate that the optimisation works. They are not accuracy results, and the val figures are not generalisation estimates. Stated plainly because a table of losses looks like a result whether or not it is one.

9 · One environment trap

MLflow 3.x refuses a plain file store ("maintenance mode") and wants a database backend. That backend needs alembic, and the system alembic was too old for its own sqlalchemy 2.x:

ImportError: cannot import name '_BindParamClause' from 'sqlalchemy.sql.expression'

Upgrading system-wide would have broken codechecker, which pins sqlalchemy==1.3.23. So setup.sh creates a venv with --system-site-packages: torch 2.3.0+cu121 is inherited rather than reinstalled (no point pulling several GB to get the same wheel, and a CUDA mismatch is a real risk), while alembic, sqlalchemy and mlflow are installed locally where they shadow the broken copies without touching them.

10 · What this is not

Not deformable attention. Dense substitution, 529× more attention work — §3.
No denoising queries (DN-DETR) and no two-stage query selection, so it inherits DETR's slow convergence. See the convergence problem.
No auxiliary decoding losses. DETR applies the loss at every decoder layer and the ablations show that matters; adding it is the single highest-value next change here.
Light augmentation — horizontal flip only. No scale jitter, no crops.
Not tuned. No accuracy claim is made anywhere on this page.

11 · References

Carion et al., End-to-End Object Detection with Transformers, ECCV 2020.
Zhu et al., Deformable DETR, ICLR 2021.
Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, 2023 — the frozen backbone.
Rezatofighi et al., Generalized Intersection over Union, CVPR 2019.
Roboflow, RF-DETR model card and repository, 2025 — the architecture this follows.
Lin et al., Microsoft COCO: Common Objects in Context, ECCV 2014.