Perception › ROS 2 implementation › Acronym glossary
Critique: what could make this design unsafe?
The graph and metrics are hypotheses. A senior review asks what happens when the road, sensors, labels, compute or operational design domain (ODD) violate the assumptions that produced those numbers. For each question below, demand a measurable test, an owner and a stop/degrade rule, not an assurance that training data “probably covers it.”
Symbols used on this page are defined in the Nomenclature → metrics.
1 · Requirements before model selection
Suppose ego travels at 25 m/s and an oncoming object approaches at 15 m/s. At 150 ms age, the relative displacement is $(25+15)(.15)=6$ m before planner, actuator and braking delay. If reliable detection starts at only 40 m in rain, what is the remaining stopping margin? Write the envelope explicitly: $D_{required}=v_{rel}(t_{capture-to-action}+t_{reaction})+v_{ego}^2/(2a_{min})+D_{uncertainty}$. Which team owns the margin if exposure or thermal throttling adds 100 ms? Do not use an average latency to answer a worst-case question.
| Challenge | Evidence required | Action when failed |
|---|---|---|
Can a planner treat COASTED as measured? | Contract test of age, covariance growth, provenance and predicted time. | Reduce speed; use conservative occupancy, not an invisible stale track. |
| Can control still brake if the GPU process dies? | Fault-injection trace: deadline, watchdog, actuator response to complete crash. | Independent GNC/AEB safety path and safe stop. |
| What detects common-mode loss? | Both modalities impaired simultaneously in darkness + rain; coverage simulation. | Exit ODD rather than promise a camera or LiDAR fallback that also failed. |
2 · Sensor choice is an observability decision
Six cameras
How do HDR exposure, headlight glare, low sun, tunnels, rolling shutter, lens water, dirt and vibration interact? Is a 5/6-view frame truly covered, or is the missing camera the only view of a crossing pedestrian? Can we detect a blinded but still publishing camera, not just an absent topic?
LiDAR + radar
How many returns does a dark child-sized object or tire have at the stopping horizon, in fog or spray? Can rain backscatter look like an obstacle? Is Doppler radar required for cut-ins/AEB when LiDAR range collapses? Does a front radar still cover a side approach?
Work the sensor coverage map by azimuth, range, weather and occlusion; label where only one modality contributes. Ask whether 10 Hz full LiDAR sweeps leave blind gaps for a fast lateral bicycle; would partial sweeps reduce detection age at the cost of coverage and model retraining? How do PTP loss, camera trigger drift, motion-distorted scans and calibration creep turn two individually working sensors into a wrong fused box? For each sensor failure record both a transport health signal (deadline, dropped packet) and an observability signal (saturation, blur, low-return region, temperature, calibration residual).
3 · Model metrics that hide deployment failures
- mAP high but pedestrian recall low: was the mAP averaged equally across classes, over which recall range and object distance? Show near-VRU miss count, 40–54 m recall, PR curves and confidence intervals per ODD slice.
- Position “correct” within 2 m: at 5 m, does a 1.8 m lateral error place a pedestrian in the wrong lane? Does the planner need collision-polygon overlap rather than center-distance mAP? Test IoU and clearance distributions too.
- Tracking MOTA high: how often do IDs swap during occlusion or a merge, and does this break velocity prediction? Compare identity-consistent sequence metrics and velocity errors. The current dashboard's IDF1 field is only a frame-match surrogate (see metric definition).
- Recall improves when frames drop: are the unpaired GT frames or high-motion frames excluded? Score every expected keyframe, count missing output as a miss for a safety-oriented metric, report latency/quality jointly.
- Confidence looks high: is it calibrated under fog, sensor loss and low light? Show reliability diagrams and expected calibration error by class, range and weather; threshold only after validating the FP cost on the operating point.
For each regression, compare signed longitudinal/lateral bias, few-points misses, below-threshold misses, duplicates, wrong class, ghosts and occlusion; this distinguishes calibration, threshold, NMS, sensor limit and training-gap root causes. Be explicit about two incompatible evaluation questions: “does this match the published nuScenes protocol?” and “would we stop for an unlabeled obstacle?” Neither answers the other.
4 · Training process: what leaked, what changed?
| Question grounded in production | Validation / countermeasure |
|---|---|
| Do train and test share the same route, fleet car, camera, weather episode or near-identical frames? | Split by scene/time/route/vehicle; audit duplicates and geospatial leakage. |
| Are yaw/size/velocity labels accurate on occluded or truncated objects? | Double-label a safety-critical slice, adjudicate disagreements, track label uncertainty. |
| Did training match live augmentation, lens distortion, lidar time offsets, class order, box origin and image normalization? | Stage-by-stage Python↔ROS numerical parity on frozen examples before engine optimization. |
| Does BEVFusion learn to trust cameras or LiDAR exclusively because one dominates the dataset? | Modality-dropout training and controlled ablations; quantify joint/common-mode failures. |
| Did rare classes get sufficient independent examples, or only many correlated frames from one drive? | Count independent encounters by ODD slice and hard-case source, not only labeled boxes. |
| Does confidence stay calibrated after INT8, TensorRT, a new sensor SKU or a calibration update? | Parity + per-slice PR/calibration and shadow replay, then gated on-road release. |
What if nine-sweep accumulation includes a future packet in offline preprocessing but online only has past packets? What if a camera-lidar extrinsic is measured while the vehicle is parked but moves thermally at highway speed? A model can reproduce a leaderboard and still be unfit for deployment. Freeze label schema, calibration version, model weights, preprocessing, training seeds and inference build artifacts alongside every evaluation.
5 · Data collection and the missing denominator
How many independent miles and encounters do we have for a pedestrian in rain at night, a wheelchair in fog, or a rolling tire at 40 m? A million easy daytime boxes cannot establish a rare-event miss rate. If zero failures occur in $n$ independent events, a rough 95% one-sided upper bound on failure probability is $\approx3/n$ (“rule of three”); correlation between frames makes the effective $n$ far smaller. Thus 3,000 truly independent encounters with no miss only bounds the event failure rate around $10^{-3}$, not $10^{-6}$.
Collect with explicit ODD-cell targets (range × weather × lighting × object × occlusion × sensor-health), intervention triggers and versioned metadata. Use fleet logs of uncertainty/near misses to find examples, but sample a random baseline too or active-learning selection biases prevalence. Audit sensor dropouts in recorded bags: a bag that lost camera frames can falsely represent a model miss; a bag with only successful outputs hides hard frames. Make privacy, consent, retention and ground-truth annotation reach separate review items. Human-reviewed corner-case scenarios must accompany synthetic weather and domain-randomization tests: synthetic success alone is not real-road coverage.
6 · What-if tabletop exercises
| What if… | System-level test and expected safe outcome |
|---|---|
The front-left camera freezes yet continues publishing old frames with fresh header.stamp? | Detect repeated image content / trigger sequence mismatch, mask view, constrain valid azimuth, slow down or exit ODD. |
| LiDAR sees a clear “hole” beneath a reflective truck, camera sees a solid obstacle? | Disagreement monitor preserves conservative occupancy, not an overconfident fused free space. |
| A toddler darts from behind a parked van at 8 m and zero LiDAR points were available while occluded? | Evaluate visibility onset-to-brake delay, risk-based unobserved-region speed policy, not benchmark GT exclusion. |
| Two adjacent bicycles cross and the tracker swaps IDs? | Observe IDSW, velocity discontinuity and planner trajectory; use conservative association or uncertainty expansion. |
| A lost PTP sync shifts cameras 70 ms behind LiDAR with plausible timestamps? | Hardware counter and motion consistency monitor; reject cross-modal fusion, fall back or stop. |
| Thermal throttling makes the camera branch 200 ms, and the depth-one queue discards five hard frames? | Joint latency/quality alert; count drops, lower load safely, enforce age watchdog independent of DDS. |
| Roadworks introduce unfamiliar cones, temporary barriers, smoke and a stopped emergency vehicle? | Evaluate open-set occupancy/objectness and unknown-object policy, not only seven training labels. |
A stale transient-local TrackArray is replayed after planner restart? | Consumer checks source stamp, sequence epoch and validity; no motion authorization from replayed state. |
| rviz/rosbag joins a private CUDA handle topic or disks stall during logging? | Guard foreign PID; separate debug converter and low-priority recorder; perception deadline unaffected. |
| Both camera and LiDAR fail simultaneously due to spray, or GPU resets mid-turn? | Independent GNC watchdog, measured safe stop and calibrated residual danger; never claim single-sensor fallback. |
7 · Review artifacts to demand
- ODD + sensor-range/latency/stopping-envelope spreadsheet signed by planning and safety owners.
- Per-class, per-range, per-weather, per-visibility recall, FP, AP and velocity-error distributions with confidence intervals and denominators.
- A missing-data report: unpaired frames, zero-point GT, annotation-range exclusions, dropped packets/frames and clocks/calibration drift.
- Trace evidence for p99.9 age at heat, packet loss, DDS discovery and heavy logging; an independent watchdog-to-brake trace.
- Reproducible Python↔ROS tensor/box parity, scenario bags with exact expected metrics, and a shadow deployment plan.
- A residual-risk list with explicit “stop, slow, degrade or continue” per failure; unresolved items block ODD expansion.