What do the metrics actually measure?

1 · First define which objects count

At LiDAR keyframe time $t$, pair ground truth and estimated boxes by sensor stamp (not arrival order), transform into the same frame and apply the same validity and distance policy throughout. Drop GT with zero LiDAR returns when reproducing the cited nuScenes-style protocol, and restrict to the configured 54 m model point-cloud range. A valid detection beyond the dataset's annotation range has no GT: publish it as an excluded observation, not a scored false positive. Track the count of unpaired GT frames: silently skipping difficult frames inflates every reported score. Official nuScenes results additionally require the devkit's class-specific filtering, interpolation, attributes and dataset splits; this online evaluator is an approximation unless it reproduces every official setting.

2 · Class-aware, score-ordered detection matching

For a GT center $g=(x_g,y_g)$ and candidate detection $d=(x_d,y_d)$ in the same reference frame, define $\delta(d,g)=\sqrt{(x_d-x_g)^2+(y_d-y_g)^2}$. For each class $c$ and center-distance gate $\tau\in\{0.5,1,2,4\}\,\mathrm{m}$, sort its predictions by decreasing detector score. For each prediction, match the nearest unassigned valid GT of that class within $\tau$; count one TP and mark that GT used. If none exists, count an FP (unless the prediction is excluded by annotation policy). A duplicate for an already claimed GT is an FP; any valid unmatched GT is an FN. Detection uses greedy score ordering so a low-confidence candidate cannot steal a match from an earlier high-confidence one. Tracking instead uses Hungarian assignment with persistence preference, because identity must be globally consistent.

For score threshold $s$ let $TP(s),FP(s),FN(s)$ be accumulated over keyframes. Then

Report $P,R$ as zero when their denominator is zero and show the sample count. Recall is the fraction of valid GT objects matched, not the fraction of predictions correct. Example: 20 valid pedestrians, 12 matched, 4 scored false alarms ⇒ $R=12/20=0.60$, $P=12/16=0.75$, $FN=8$, $F_1=2(0.60)(0.75)/(1.35)=0.667$. Stratify by class and range: 0–20 m recall and 40–54 m recall answer very different driving questions.

3 · AP from ranked detections, step by step

AP integrates precision as recall rises while we lower the score threshold; a single operating-point precision or recall is not AP. For sorted detections $i=1,\ldots,n$, with $y_i=1$ for TP and $0$ for FP, define $T_i=\sum_{j=1}^{i} y_j$, $F_i=i-T_i$, $p_i=T_i/i$, $r_i=T_i/N_{GT}$. Sweep the ranked list and take the monotone upper precision envelope $\bar p_i=\max_{j\ge i}p_j$. To sample at recall $r$, use $\tilde p(r)=\max_{i:r_i\ge r}p_i$ (zero if no detection reaches $r$); this equals the first envelope value at or above $r$.

Reverse-maximum smoothing gives $(1,.75,.75,.75)$: at rank 2, $\max(.5,2/3,.75)=.75$. At sampled recall $.50$ or $.75$, interpolated precision is $.75$; above $.75$ it is zero. Duplicate recall ranks must not let a false positive mask the later true positive.

The evaluator implements a 101-point clipped approximation (default $r_{min}=p_{min}=0.1$):

In this example, 16 grid points from $.10$ through $.25$ have precision 1, 50 points from $.26$ through $.75$ have precision $.75$, and 25 remaining points have zero. Thus $AP=[16(.9)+50(.65)]/[91(.9)]=46.9/81.9\approx0.573$. The online code samples the first score rank whose recall is at least the grid point after applying the reverse maximum; duplicate recall ranks are handled by that envelope. This is not COCO IoU AP: COCO varies IoU thresholds and has its own sampling rules.

Across evaluated classes $\mathcal C$ and the four distance gates $\mathcal T$:

Example: car APs $(.50,.65,.75,.85)$ average $.6875$; pedestrian APs $(.20,.30,.45,.55)$ average $.375$; the two-class mAP is $(.6875+.375)/2=.53125$. The running evaluator averages over classes with valid GT; log class support, because changing which classes are present changes that mean. A 2 m match can be a TP while the 0.5 m gate counts a localization miss.

4 · Errors, NDS and the operating point

For matched true positives measure mean translation error $ATE=\mathbb E[\|d_{xy}-g_{xy}\|]$, scale error $ASE=\mathbb E[1-\operatorname{IoU}(d^{aligned},g^{aligned})]$ (align centers and yaw first), yaw error $AOE=\mathbb E[|\operatorname{wrap}(\theta_d-\theta_g)|]$, and velocity error $AVE=\mathbb E[\|v_d-v_g\|]$. In addition publish signed ego-frame longitudinal/lateral, yaw and speed errors: a 30 cm systematic near-range bias suggests calibration/timing, while symmetric random errors suggest noise. The canonical nuScenes detection score is

where $AAE$ is attribute error. If a local implementation lacks attribute predictions, omits the official matching point or uses a different normalization, label its result NDS-like, not an official nuScenes NDS. Example if $mAP=.60$ and all five normalized TP scores are $.8$: $NDS=(3+4)/10=.70$.

The deployed detector confidence cutoff chooses one $(P,R)$ point and cannot be recovered from mAP alone. Move it only after comparing $\Delta FN$ to $\Delta FP$ at relevant ranges and classes; inspect BELOW_THRESHOLD, FEW_POINTS, WRONG_CLASS, POOR_LOCALIZATION, OCCLUDED and NO_DETECTION causes separately.

5 · MOTA: accuracy of a sequence of identities

For each frame $t$, let $GT_t$ be the number of valid annotated objects, $FN_t$ unmatched GT, $FP_t$ unmatched estimated tracks, and $IDSW_t$ GT identities matched to a different estimated ID than their previous valid association. Tracking assignment uses a 2 m gate (configurable) with a small continuity bonus to prevent noise-induced switching. CLEAR-MOT accuracy is

Example: 50 object-frames, 8 misses, 4 ghosts and 3 identity switches ⇒ $MOTA=1-15/50=.70$. If there are 50 object-frames and 60 ghosts alone, $MOTA=1-60/50=-.20$: never clamp MOTA. Detector recall can be .88 while MOTA is .64, because MOTA also charges FPs and ID switches. Reset identity mapping on a bag loop, scene break or tracker reset while retaining aggregate frame counts; otherwise the first frame of the new scene creates spurious switches.

Identity switches: a GT car matched to track 7, then track 9 in the next associated frame adds 1. Fragmentation (as counted by mot_accumulator.cpp): a GT object that was matched in the previous frame is unmatched in this frame while still annotated, i.e. each tracked → missed transition adds 1, whether or not it is re-acquired later, and regardless of which ID picks it up. Some benchmark implementations count only interruptions that are later resumed, so compare FRAG across tools with care. ID switches and fragmentations are independent: a swap with no gap is 1 IDSW and 0 FRAG; a gap bridged by the same track is 0 IDSW and 1 FRAG (see the worked table). Mostly-tracked (MT) means tracked in more than 80% of the frames in which it is annotated, mostly-lost (ML) under 20%; for four GT trajectories with covered fractions $(.95,.83,.40,.10)$, $MT=2/4=.5$ and $ML=1/4=.25$. Sequence-boundary and occlusion policies must be fixed before interpreting these.

6 · MOTP, AMOTA and identity F1

In this implementation, with $M$ matches and position distances $d_k$, mean matched-position error is $\bar d=\sum_k d_k/M$ meters; displayed normalized $MOTP=\max(0,1-\bar d/\tau)$, with gate $\tau$ in meters. Four matches at distances $.2,.4,.6,.8$ m have $\bar d=.5$ m and $MOTP=.75$ for $\tau=2$ m. State the convention: original CLEAR-MOT publications may report raw mean overlap/position distance, and nuScenes AMOTP has a different aggregation; this normalized value is a project-specific display, not directly comparable to another benchmark.

For configured track-existence cutoffs $s_k$ (this stack uses $0,.2,.4,.6,.8$), recompute tracking totals after filtering tracks, then display $AMOTA_{local}=\frac1K\sum_k\max(0,MOTA(s_k))$ and $AMOTP_{local}=\frac1K\sum_k MOTP(s_k)$. Example $MOTA(s_k)=(.50,.60,.55,.40,-.10)$ ⇒ local AMOTA $(.50+.60+.55+.40+0)/5=.41$. These cutoffs are confidence thresholds, not guaranteed evenly spaced achieved-recall targets. Official nuScenes AMOTA is interpolated across recall operating points and normalized differently; do not claim leaderboard equivalence without rerunning its evaluator.

Proper sequence-level identity F1 solves a global trajectory-to-trajectory matching problem: $IDP=IDTP/(IDTP+IDFP)$, $IDR=IDTP/(IDTP+IDFN)$, $IDF1=2IDTP/(2IDTP+IDFP+IDFN)$. Example $IDTP=70,IDFP=10,IDFN=20$ ⇒ $IDF1=140/170\approx.824$. A frame-level surrogate $2M/(2M+FP+FN)$, where $M$ is the count of frame matches, does not penalize identity switches properly. The current evaluator's id_f1() uses that surrogate: treat its published field as frame association F1 until a global ID assignment is implemented. This distinction prevents a polished dashboard from overstating tracker identity quality.

7 · Make evaluation observable without contaminating control

Publish cause-labeled misses and false positives as EvalFrame alongside class/range summaries; clear rviz markers every frame so stale visualizations cannot masquerade as live obstacles. Keep the evaluator single-threaded for reproducibility and off the critical GPU and GNC executors. Record stamps, parameter versions, error frames, metrics and sequence IDs for retrospective triage without rerunning inference. Synthetic GT exercises an occluded pedestrian, zero-point distant car, approaching vehicle and static barrier; exact expected totals become CI assertions. On a real car without GT, dashboard detection error counts are not available; report health and surrogate coverage separately, never present them as measured recall.