Tracking vs ground truth — the full nuScenes val split

Every keyframe of all 150 nuScenes validation scenes — 6,019 frames — rendered with the BEVFusion → SimpleTrack output and the ground truth drawn in the same picture, so each box is labelled as a hit, a false positive, or a miss. The aggregate numbers say the tracker is 94% precise and recalls 45% of annotated objects. The videos below show why both of those numbers are true at once.

Symbols used on this page are defined in the Nomenclature → metrics.

The single most useful thing on this page. A detector evaluated by a scalar mAP tells you nothing about which objects it missed. Drawing GT and predictions together, frame by frame, makes the answer immediate — and then lets you go and count it. The answer turns out to be almost entirely about range and lidar return count, not about object class: recall is 0.91 inside 10 m and 0.001 beyond 50 m. See §4, which also documents a bug the exercise found in this page's own harness.

1 · What you are looking at

Each frame is a composite. On the left, the CAM_FRONT photo with 3-D boxes projected into it; on the right, a bird's-eye view in the ego frame with range rings every 10 m and the ego vehicle as a white triangle. Both panels draw the same four categories of box, in the same colours:

Colour	Style	Meaning
green	dashed	Ground truth that was matched by a prediction
cyan	solid	Prediction matched to GT — a true positive. Labelled with its track id
red	solid	Prediction with no GT match — a false positive
orange	dashed	GT with no prediction — a miss

The header of every frame carries the per-frame counts and the running precision and recall, so you can watch the numbers move as the scene gets harder.

1.1 How a box is called a hit

Matching is greedy nearest-neighbour on bird's-eye-view centre distance, restricted to boxes of the same class, with a 2 m gate. Predictions are considered in descending score order, each claiming the nearest unclaimed GT box of its class:

$$ \text{match}(i) = \arg\min_{j \,\in\, \text{unclaimed},\ c_j = c_i} \left\| \mathbf p_i^{xy} - \mathbf p_j^{xy} \right\|_2 \quad\text{subject to}\quad \left\|\cdot\right\|_2 < 2\,\text{m}. $$

Centre distance rather than 3-D IoU is deliberate — it is what the official nuScenes detection metric uses, because IoU is unstably sensitive to orientation error on small objects like pedestrians.

These precision and recall figures are not mAP, NDS or AMOTA. They are a single-threshold greedy proxy computed at score > 0.3 purely so each box can be coloured. The benchmark numbers come from the official evaluator. Treat the colours as a diagnostic aid, not a leaderboard entry.

2 · Four scenes, four different failure modes

These were not picked for looking good. They are the extremes of the 118 scenes that carry per-scene statistics, chosen by ranking on recall and false-positive count.

Scene	Frames	GT	Pred	TP	FP	Miss	P	R	Why it is here
scene-1073	40	597	522	499	23	98	0.96	0.84	Best recall of any long scene — the system working
scene-0920	41	329	170	164	6	165	0.97	0.50	The median, typical case
scene-0275	40	1878	1148	799	349	1079	0.70	0.43	Most false positives in the split
scene-0913	40	2702	449	407	42	2295	0.91	0.15	Busiest scene — recall collapses

2.1 scene-1073 — what success looks like

Recall 0.84, precision 0.96. A relatively open road with a modest number of nearby vehicles. Almost every box is cyan with a stable id, and the few orange ones are at the far edge of the BEV panel. This is the regime the stack is actually good at.

▶ scene-1073 — tracking with ground truth overlaid

40 frames · 10 fps (5× real time) · GT dashed green · TP cyan · FP red · miss orange

2.2 scene-0913 — recall collapses under crowding

Recall 0.15 against 2,702 ground-truth boxes in 40 frames — nearly 68 annotated objects per frame. Precision stays at 0.91, so the detections it does make are still right. The screen fills with orange.

This is the clearest illustration of something the aggregate number hides: the tracker did not get worse here, it got overwhelmed by annotation density at range. nuScenes labels every object that has at least one lidar return, which on a busy street means dozens of distant objects a detector has almost no signal for — §5.3 shows two-thirds of all misses have five or fewer returns. Counting those as recall failures is correct but not very informative.

▶ scene-0913 — 2,702 GT boxes, recall 0.15

40 frames · the orange mass is annotated-but-undetected objects, mostly beyond 30 m

2.3 scene-0275 — the false-positive case

349 false positives, precision down to 0.70 — the worst in the split, and the only scene where red boxes are a significant presence rather than an occasional flicker. Worth watching for where they appear: clustered around real objects rather than scattered randomly, which is the signature of duplicate detections surviving NMS on large or oddly-shaped vehicles, plus a handful of genuine confusions in clutter.

▶ scene-0275 — 349 false positives, precision 0.70

2.4 scene-0920 — the typical case

The median scene: recall 0.50, precision 0.97, 329 GT boxes. Half the annotated objects found, and virtually everything claimed is correct. If you want one clip that represents the whole 6,019-frame run rather than its extremes, this is it.

▶ scene-0920 — the median scene

3 · The aggregate result

Quantity	Value
Scenes rendered	150 (the entire val split)
Frames rendered	6,019
Scenes with per-scene statistics	118
True positives	51,109
False positives	3,157
Misses	63,532
Precision	0.942
Recall, as rendered (10-class GT)	0.446
Recall, corrected (7-class tracking GT)	~0.59
Render throughput	4.5 frame/s single-process
Output size	573 MB of JPEGs

The TP/FP/miss counts above are from the render pass, which used the 10-class detection vocabulary. Per §5 that scores unmatchable classes as misses, so the 0.446 is an understatement; on a 20-scene re-measurement with the corrected 7-class set the figure is 0.592. The precision figure is unaffected — it depends only on the predictions, which never contained those classes. The frames themselves also still draw cones and barriers as missed, since re-rendering 6,019 images to recolour three classes was not worth the compute; treat orange street furniture in the clips as a known artefact.

32 of the 150 scenes show zero counters in the summary. That is a bookkeeping artefact, not a result: the first render pass was interrupted, and the resumed pass ran with --skip-existing, so those already-complete scenes were skipped and contributed no statistics. Their frames are present and correct — they are simply excluded from the per-scene ranking, because treating them as "recall 0.00" would have put four perfectly good scenes at the top of the worst-performers list. It did, on the first attempt.

3.1 Why precision and recall are so lopsided

0.94 precision with 0.45 recall is the signature of a conservative operating point, and it is a direct consequence of the score threshold. The tracker's raw output is dominated by very low-confidence boxes — the median tracking score over a 400-frame sample is 0.044:

Threshold	Track boxes kept	Detection boxes kept
≥ 0.0	100%	100%
≥ 0.1	40.7%	22.4%
≥ 0.3 (used here)	28.8%	15.6%
≥ 0.5	20.3%	10.9%
≥ 0.7	9.2%	4.6%

Drawing at score > 0.3 keeps roughly the most confident quarter of tracks. Those are nearly all correct, which is where 0.94 precision comes from; everything discarded that happened to be a real object becomes a miss, which is where recall goes. Lower the threshold and the two numbers trade against each other in the usual way — the single operating point shown here is a visualisation choice, not a property of the tracker.

4 · Reproducing it

Two scripts, both in perception/tools/. The renderer walks each scene's sample chain, loads GT from sample_annotation, merges SimpleTrack's seven per-class result files, matches, and writes one composite JPEG per keyframe:

# every val scene, resumable python tools/render_frames.py --source track \ --out /mnt/mydata/nuscenes/results/frames_gt_track \ --workers 1 --skip-existing # one scene, quick look python tools/render_frames.py --source track --scene scene-1073 --limit 5 --out /tmp/viz # encode: small clips for this page, plus a full-resolution archive ./tools/make_tracking_videos.sh web ./tools/make_tracking_videos.sh youtube # rank the scenes (this is what chose the four clips above) python tools/rank_scenes.py # break the misses down by class / range / lidar points / visibility (§5) python tools/analyze_misses.py --scenes 20

4.1 Two encodes, on purpose

	Web clips (above)	Archive copy
Scope	4 illustrative scenes	All 150 scenes, all 6,019 frames
Resolution	1280 × 434	1816 × 616 (native, padded)
Codecs	H.264 crf 30 + VP9 crf 38	H.264 crf 20
Size	~0.4 MB per clip, 7.8 MB total	~300 MB
Purpose	Loads instantly, lives in git	Upload target; keeps the fine detail

The split exists because downscaling destroys exactly the detail worth keeping — the track ids and the per-frame counter are small text — while a 300 MB file has no business in a static site. Both run at 10 fps, which is 5× real time: nuScenes keyframes are 2 Hz, and 2 fps is unwatchable.

One ffmpeg trap worth recording. The frames are 1815 px wide, and H.264 with yuv420p requires both dimensions to be even because chroma is subsampled 2×2. A naive native-resolution encode fails outright with width not divisible by 2. The fix is to pad to 1816 rather than crop or scale — padding adds one black column and leaves every rendered pixel intact, where cropping would shave a column off the BEV panel and scaling would soften the small text. The web encode never hit this, because scale=1280:-2 forces an even height by construction.

4.2 Verifying the renderer before trusting it

A visualisation that is subtly wrong is worse than none, because it looks authoritative. Three checks ran before the full pass:

Box geometry asserted numerically — a known box at a known pose must produce a known footprint, the shoelace area must equal width × length, and a 90° yaw must swap the extents. This caught a real bug: the BEV footprint was being taken from corner indices 0,1,2,3, which all share the same $x$ and are therefore the front face, so every vehicle rendered as a zero-width sliver. The correct indices are 0,1,5,4.
Score distribution inspected — the first smoke test reported zero false positives, which is not plausible for a real detector. It turned out to be the threshold doing its job (median score 0.044), not a matching bug.
Match quality measured — over 251 matched pairs the median centre distance is 0.17 m and the width, length and height ratios all sit between 1.00 and 1.09. Had the size ratios come out inverted it would have meant the (w, l, h) ordering was wrong somewhere, which is an easy mistake since nuScenes stores width before length.

5 · What is annotated but not detected

The videos make the misses visible; this section counts them. Over a 20-scene sample — 18,371 ground-truth boxes — every missed box was binned by class, by range from the ego vehicle, by how many lidar and radar returns actually landed on it, and by nuScenes' own visibility (occlusion) label.

This exercise found a bug in the harness used for the rest of this page, and it mattered. SimpleTrack produces the 7-class nuScenes tracking vocabulary. I was scoring it against the 10-class detection vocabulary, which adds traffic_cone, barrier and construction_vehicle. Those three can never be matched — the tracker does not emit them — so every one was counted as a missed detection. That was 4,833 boxes, 39% of all reported misses, and it understated recall by roughly ten points. The giveaway was in the per-class table: traffic_cone 2742/2742 missed, barrier 1683/1683, construction_vehicle 408/408 — an exact 0.000 recall, which is the signature of a class that is absent rather than hard. Every number below is post-fix.

5.1 By class — not the story

Class	GT	Missed	Recall	Share of misses
car	10,293	3,406	0.669	45.4%
pedestrian	4,584	2,035	0.556	27.1%
truck	1,884	1,132	0.399	15.1%
trailer	436	366	0.161	4.9%
bus	450	227	0.496	3.0%
bicycle	329	173	0.474	2.3%
motorcycle	395	163	0.587	2.2%

Cars are 45% of the misses, not pedestrians — simply because cars are 56% of the annotations. An earlier version of this page claimed the misses were "dominated by pedestrians", which the data does not support; that claim has been removed. The one genuinely alarming per-class number is trailer at 0.161: articulated vehicles are long, their annotated box centre can sit far from the visible mass, and a 2 m centre-distance gate is unforgiving about that.

5.2 By range — this is the story

Range	GT	Missed	Recall	Share of misses
0–10 m	1,772	163	0.908	2.2%
10–20 m	3,850	525	0.864	7.0%
20–30 m	4,165	897	0.785	12.0%
30–40 m	3,401	1,408	0.586	18.8%
40–50 m	2,090	1,419	0.321	18.9%
50 m +	3,093	3,090	0.001	41.2%

Beyond 50 m the tracker detects essentially nothing — 3 of 3,093 — and that single bin is 41% of all misses. This is not a model deficiency so much as a configuration fact: the BEVFusion detection range is bounded (roughly ±54 m in the BEV grid), while nuScenes annotates anything with a lidar return, including objects past the grid edge. Those boxes are outside the model's declared field of regard. Add the 40–50 m bin and 60% of all misses sit at or beyond the design range.

5.3 By lidar return count — the mechanism

Lidar+radar points on the object	GT	Missed	Recall	Share of misses
1–5	6,847	5,088	0.257	67.8%
5–20	5,518	1,657	0.700	22.1%
20–100	3,736	563	0.849	7.5%
100 +	2,270	194	0.915	2.6%

This is the cleanest monotonic relationship in the whole dataset, and it is the same finding as §5.2 viewed from the sensor's side: distance reduces return count, and return count is what the detector has to work with. Two-thirds of all misses are objects with five or fewer returns. With 100+ returns recall is 0.92.

nuScenes includes any box with at least one return. A box with a single lidar point is, to a detector, a point — there is no extent, no orientation and no shape to classify. Counting those as recall failures is defensible as a benchmark convention but tells you very little about the model.

5.4 By occlusion — weaker than you would guess

nuScenes visibility	GT	Missed	Recall	Share of misses
0–40% visible	5,204	3,308	0.364	44.1%
40–60%	2,047	862	0.579	11.5%
60–80%	2,788	1,135	0.593	15.1%
80–100% visible	8,332	2,197	0.736	29.3%

Occlusion matters — 0.36 against 0.74 — but far less than range or point count, and note that 29% of misses are objects labelled 80–100% visible. Those are not occluded at all; they are simply far away. Visibility is a camera-centric label, and this is a lidar-led detector, so the correlation is weaker than intuition suggests.

5.5 Recall by class × range

Class	0–10 m	10–20	20–30	30–40	40–50
car	1.00	0.95	0.87	0.66	0.42
bus	0.90	0.88	0.74	0.54	0.30
pedestrian	0.85	0.78	0.67	0.53	0.01
motorcycle	0.81	0.80	0.66	0.42	0.00
truck	0.64	0.78	0.68	0.46	0.22
trailer	—	0.50	0.33	0.29	0.45

Cars inside 10 m: perfect recall. Every class degrades with range in the same way, and the 0.45 aggregate is an average over a distribution that happens to contain a lot of distant objects — it is not the performance you would experience at any particular range. Two anomalies worth noting: pedestrians fall off a cliff at 40 m (0.53 → 0.01), consistent with a small object dropping below the return count needed to be detected at all; and trucks are worse at 0–10 m (0.64) than at 10–20 m (0.78), which is what happens when a large vehicle is close enough to be clipped by the detection grid or to have its centre poorly constrained.

6 · What the videos show that the metrics do not

Recall is a function of range, not a constant. Watch the orange boxes: they are almost all at the outer rings of the BEV panel. Inside 10 m recall is 0.91; past 50 m it is 0.001. Quantified in §5.2.
Most misses are objects the sensor barely saw. Two-thirds have five or fewer lidar returns (§5.3). The orange mass in crowded scenes is annotation density at range, not a detector that stopped working.
Identity is mostly stable. The track id printed on each cyan box persists across frames for nearby vehicles, which is the association doing its job. Id switches are visible as a number changing on a box that plainly did not change object.
False positives cluster on real objects. They are duplicates, not hallucinations — which implies the fix is in NMS and track confirmation logic, not the detector.
A visualisation is a debugging tool for the harness too. The class-vocabulary mismatch in the callout above was invisible in the aggregate numbers and obvious the moment the misses were broken out per class.

Related: Perception pipeline for the architecture these outputs come from, online evaluation for computing this continuously rather than in a batch, and Nomenclature for the metric definitions.