ROS 2 Perception System Design

1 · Requirements — what the system must do

Every design decision later in this page traces back to one of these rows. Writing them down first is what distinguishes a system design from a collection of nodes.

1.1 Functional

1.2 Non-functional — the ones that shape the architecture

2 · The latency budget — where 100 ms goes

A 10 Hz sensor cadence gives a 100 ms period. The p99 target is 150 ms from the instant light hits the sensor to the instant the planner can act on an object. Here is the accounting, with the concurrency made explicit:

Total with overlap: ~85–95 ms, leaving ~55 ms of margin. That margin is not slack:

• Thermal throttling. On an Orin the GPU clock drops progressively under sustained load. The first symptom is a rising p99, not a hard failure — so the system degrades before it starts dropping frames.
• Tail events. A page fault, a log flush, or a DDS discovery burst adds milliseconds. Margin is what keeps a tail event from becoming a missed deadline.
• Future work. An occupancy head costs ~25 ms. Knowing the margin tells you whether it fits without re-architecting.

2.1 Latency is a safety parameter, not a performance one

At 15 m/s (54 km/h) the vehicle covers 1.5 m per 100 ms. So perception latency translates directly into a position uncertainty for every object:

For an oncoming vehicle at 15 m/s closing on our 15 m/s, a 150 ms latency is \(30 \times 0.15 = 4.5\) m of positional staleness. This is why the tracker publishes velocity and covariance: the planner forward-predicts each object to the start of its planning horizon and inflates by the propagated uncertainty, rather than treating a 150 ms-old box as current truth.

3 · Architecture at a glance

                    ┌──────────── SENSORS (separate processes) ────────────┐
                    │ 6× GMSL camera   LiDAR   radar   GNSS/IMU            │
                    └───┬──────────────┬─────────┬────────┬────────────────┘
      /cam_*/image_raw  │ /lidar/points│ /radar  │ /imu   │ /gnss
       (BE, KEEP_LAST 1)│ (BE, KL 1)   │         │        │
                        ▼              ▼         ▼        ▼
╔═══════════════ perception_container ── ONE PROCESS, 3 EXECUTORS ═══════════════╗
║                                                                                ║
║  image_ingest ──┐                          lidar_decoder                       ║
║   (6 streams,   │   GpuTensor handles        (de-skew, crop, accumulate)        ║
║    1× H2D)      └──────────► frame_sync ◄───────────┘                           ║
║                          (sweep-anchored; calib + ego-pose snapshot)            ║
║                                 │ SensorFrame  (handles + all matrices)         ║
║                    ┌────────────┴────────────┐                                  ║
║           camera_encoder               lidar_encoder      ← CONCURRENT           ║
║         (backbone + LSS + BEVPool)   (voxelize + sparse conv)                    ║
║                    └────────────┬────────────┘                                  ║
║                            bev_fuser   (seq-keyed join; LiDAR-only fallback)     ║
║                    ┌────────────┼────────────┐                                  ║
║               det_head      seg_head     (occ_head)                              ║
║                    │            │                                               ║
║                tracker       grids                                              ║
║                    │                                                            ║
║          health_node (degradation state machine)   viz_node (off by default)     ║
╚════════════════════════════════════════════════════════════════════════════════╝
      │ detections3d  tracks  objects  status  drivable_grid  stage_latency
      ├─────────────────────────────┬─────────────────────────────┐
      ▼                             ▼                             ▼
┌─ GNC container ────────┐ ┌─ eval_container ───────────┐ ┌─ rviz2 / rosbag2 ──┐
│ ekf → predictor →      │ │ det_eval    track_eval     │ │ markers, recording │
│ planner → controller   │ │ error_viz   dashboard      │ │                    │
│ 100 Hz  SCHED_FIFO     │ │ single-threaded, nice +10  │ │                    │
└────────────────────────┘ └────────────────────────────┘ └────────────────────┘
                                        ▲
                              /ground_truth/objects
                        (separate process: perception CANNOT consume it)

3.1 The three structural decisions, in order of impact

4 · Process topology and node composition

4.1 What composition actually buys — measured

4.2 The one flag that makes it work

// Every composable node must forward NodeOptions, and the container must set this:
auto opts = rclcpp::NodeOptions().use_intra_process_comms(true);

class ImageIngestNode : public rclcpp::Node {
 public:
  explicit ImageIngestNode(const rclcpp::NodeOptions & options)
  : rclcpp::Node("image_ingest", options) {}   // <-- forwarding is MANDATORY
};
RCLCPP_COMPONENTS_REGISTER_NODE(perception::ImageIngestNode)

4.3 Both composition paths are supported

# custom container: explicit graph, 3 executors, CPU affinity, RT-friendly (vehicle)
ros2 launch perception_bringup full_system.launch.py container_type:=custom

# standard component_container_mt: dynamic load/unload (demos, A/B testing a head)
ros2 launch perception_bringup full_system.launch.py container_type:=mt
ros2 component list
ros2 component unload /perception_container 8      # drop the seg head, live
ros2 component load /perception_container perception_nodes perception::SegHeadNode

The dynamic path is genuinely useful: you can A/B two detection heads on the same recorded stream without restarting the pipeline or reloading the LiDAR backbone. Details and the custom container source →

5 · The nodes — what each one owns

5.1 Perception (11 nodes, one container)

5.2 Evaluation and display (4 nodes + ground truth)

5.3 The three nodes worth a closer look

frame_sync — the correctness heart

Six cameras and a LiDAR are not triggered together, and DDS gives no cross-topic ordering. message_filters::ApproximateTime optimizes for "match all topics", can hold a frame for a full period, and offers no control over the deadline or over partial results. For a surround-view detector that must keep working on 5/6 cameras, that is disqualifying. So the policy is explicit and unit-tested:

anchor      = LiDAR sweep END time            (the BEVFusion / nuScenes convention)
per camera  = nearest stamp within slop (50 ms), residual skew reported in the message
emit when   = anchor + ≥ min_cameras (4) available
              OR max_wait (60 ms) expires → PARTIAL frame + valid_camera_mask
new anchor  = SUPERSEDES any pending one (latest-only); the drop is COUNTED

It also snapshots every calibration matrix and the ego pose into the outgoing SensorFrame. Consequence: no model node ever calls TF2 on the hot path (no mutex, no extrapolation exception, no timeout), and a recorded bundle replays deterministically in a unit test.

bev_fuser — the interesting part is the join, not the convolution

Two independent asynchronous producers must be paired for the same frame. The join is keyed on seq (never arrival order), uses a 2-slot table, never blocks on a straggler, and has an explicit degraded path:

tracker — stateful, therefore strictly serialized

The tracker holds Kalman state across frames. It lives in a MutuallyExclusive callback group and must never be reentrant: two concurrent steps would corrupt covariances and scramble IDs. It also tracks in the global (map) frame — in the ego frame, the vehicle's own motion appears as object motion and a constant-velocity filter fights the vehicle's dynamics, so a parked car looks like it is moving at −15 m/s.

And a subtle one: a sequence break (bag loop, scene change, localization jump) must reset the bank. Without it the filters try to explain a teleport as velocity and emit 40 m/s pedestrians straight into the planner.

6 · Every message, and why it exists

23 message types (plus 3 services) in four families, each with different rules. The families are the design: a message's family determines its QoS, whether it can cross a process boundary, and whether it may contain a pointer.

6.1 Family A — GPU handle messages (intra-process ONLY)

These carry a uint64 pointer, not data. A [1,256,180,180] fp16 BEV feature map is 16.6 MB; serializing it would cost ~10 ms and a full core.

6.2 Family B — result messages (to GNC)

6.3 Family C — fixed-size POD messages (loanable / true SHM zero copy)

Used only on the driver→container hop, where a copy would otherwise cost ~3 ms and a core at 20 Hz. Why sensor_msgs/Image can never be loaned →

6.4 Family D — ground truth and evaluation

6.5 Services — and when not to use one

6.6 Parameters — and the read-only discipline

Everything that affects geometry, memory or timing is a parameter, so one binary runs on nuScenes replay and on the vehicle. But two categories are declared read-only:

rcl_interfaces::msg::ParameterDescriptor ro;
ro.read_only = true;                        // changing it must be a RECONFIGURE, not a set
engine_path_  = declare_parameter<std::string>("engine_path", default_engine, ro);
backend_      = declare_parameter<std::string>("backend", "stub", ro);

Swapping a TensorRT engine at runtime would invalidate every preallocated buffer, so it is a lifecycle transition (deactivate → cleanup → configure), not a parameter write. Everything else (thresholds, NMS type, tracker gates, deadlines) is live-tunable via add_on_set_parameters_callback with range validation — and rejected atomically if invalid.

6.7 The box convention — the single most common silent bug

A forgotten h/2 shift or a swapped (w,l) produces boxes that look plausible and quietly destroy mAP and GIoU association at range. So every conversion lives in one header with a round-trip unit test, and no node is allowed to inline one.

7 · Time, frames, and synchronization

7.1 Frames (REP-103 / REP-105)

map ──(localization: JUMPS on loop closure)──▶ odom ──(smooth)──▶ base_link
                                                                    ├─ lidar_top
                                                                    ├─ cam_front, cam_front_left, …
                                                                    ├─ radar_front
                                                                    └─ imu_link

7.2 Time rules

1. Stamp from the sensor, never now(). A replay node that stamps with now() makes every latency number a lie and breaks the synchronizer, because message_filters-style matching keys on header.stamp only.
2. RCL_STEADY_TIME for durations and timeouts; RCL_ROS_TIME for anything stamp-related. An NTP step will otherwise corrupt a measured interval.
3. Bag replay requires use_sim_time:=true and --clock, for all nodes. Mixed sim/wall time in one graph silently matches nothing.
4. PTP/gPTP across ECUs. 5 ms of clock skew at 15 m/s is 7.5 cm of position bias — which looks exactly like a calibration error and can waste weeks.

7.3 Sweep-anchored synchronization, visualized

A dead camera therefore costs one mask bit, not the frame. That single property is why the stack survives a GMSL deserializer glitch that would otherwise stall the pipeline until the watchdog fired.

8 · How this becomes real-time

"Real-time" here means bounded, measured latency with a defined behaviour when the bound is missed — not "fast". Five mechanisms, in the order they matter:

auto buf = pool->acquire();
if (!buf) {                       // pool exhausted
  stats_.record_drop();           // COUNT it
  return;                         // DROP the frame — never block
}

mlockall(MCL_CURRENT | MCL_FUTURE);  // no page faults in callbacks
cudaSetDevice(0); cudaFree(nullptr); // force CUDA context NOW (~300 ms),
                                     // not inside the first callback

so.event_callbacks.deadline_callback = [this](auto & info) {
    health_.mark_stale("cam_front", info.total_count);   // → DEGRADED_CAM
};

8.1 Core and stream allocation (Jetson AGX Orin, 12 cores)

Inside an RT callback, never: allocate, take a lock shared with a non-RT thread, log to disk, call into CUDA, call lookupTransform with a timeout, or use std::chrono::system_clock. State crosses the RT boundary through a double-buffer with an atomic sequence counter (a seqlock), not a mutex.

8.2 Measured targets

9 · The GNC contract and degraded modes

GNC consumes exactly four things. Everything else on the graph is debug. Keeping the interface this narrow is what makes the perception stack replaceable.

9.1 The state machine — owned by exactly one node

NOMINAL ──cam branch silent ×3 (~300 ms)──▶ DEGRADED_CAM    LiDAR-only head,
   ▲                                                         cap 8 m/s, trust ≤ 40 m
   │◀────────── cameras good ×5  (recover SLOW) ──────────────┘
   ├──lidar silent ×2──────────▶ DEGRADED_LIDAR   camera-only BEV, shorter horizon
   ├──p99 > latency budget─────▶ DEGRADED_CAM      (thermal: slow down BEFORE dropping)
   ├──objects older than 300 ms▶ STALE             coast ≤ coast_ms, then brake
   └──GPU / model fault────────▶ FAULT             safe stop; lifecycle retry

9.2 The three rules a planner must implement

// 1. AGE GATE. TRANSIENT_LOCAL means the first sample can be arbitrarily old.
//    A planner that trusts it will swerve around a car that was there two minutes ago.
if ((now() - msg->header.stamp).seconds() > max_age_) { coast_then_brake(); return; }

// 2. STATUS GATE. Perception tells you what it can currently see.
v_cmd = std::min(v_cmd, status_.max_speed_advice_mps);
// and ignore objects beyond status_.max_range_valid_m in the current mode

// 3. INDEPENDENT WALL-CLOCK WATCHDOG. A QoS deadline event is delivered by the same
//    executor that might itself be stuck, so the actuator keeps its own steady-clock timer.
if (steady_now() - last_cmd_ > watchdog_) actuate_safe_state();

9.3 Failure taxonomy

Escalation timing is derived, not guessed: at 10 Hz with a 150 ms budget, a deadline of 150 ms makes a dead sensor visible within ~1.5 frames; 3 frames to degrade is 300 ms ≈ 4.5 m of travel at 15 m/s; 300 ms to STALE is the most blind travel we accept before coasting to a stop.

10 · Running it

The whole system is one command, and it runs on a laptop with no GPU and no dataset — which is the point of requirement N8.

# build (host reference kernels + deterministic stub models)
cd Ros2/ws && colcon build --symlink-install
source install/setup.bash

# ── everything: sensors + ground truth + perception + evaluation + display ──
ros2 launch perception_bringup full_system.launch.py rviz:=true

# perception only, no evaluation — what actually runs on a vehicle
ros2 launch perception_bringup full_system.launch.py evaluation:=false ground_truth:=false

# real nuScenes data, real TensorRT engines, with evaluation
ros2 launch perception_bringup full_system.launch.py \
    source:=nuscenes dataroot:=$NUSCENES_ROOT backend:=tensorrt rviz:=true

# ── verify the claims ──
ros2 node list | wc -l                                   # 15
pgrep -a perception_container                             # ONE pid
ps -L -p $(pgrep -f perception_container) | wc -l         # 3+4+1+rmw threads, not 12
ros2 run perception_py latency_report                     # per-stage p50/p95/p99
ros2 topic echo /perception/status                        # the degradation contract

10.1 What the workspace contains

10.2 Test pyramid

 few  │ fault injection: SIGSTOP a camera, kill the container, step the clock, OOM the pool
      │ scenario: nuScenes scene → mAP / NDS / AMOTA vs a baseline          (nightly)
      │ integration: full_system.launch.py source:=synthetic                (CI, no GPU)
      │    asserts 1 process, dropped==0, p99 < budget, recall == expected
 many │ unit: 7 gtest suites                                               (CI, ms)

11 · Continue

11.1 Source

Everything on these pages is implemented in Ros2/: SYSTEM_DESIGN.md (the full design document), docs/01–docs/11 (topic deep dives), docs/99_interview_drills.md, ws/src/ (6 packages, 15 nodes, 7 test suites), and exercises/ (10 drills with falsifiable acceptance criteria). The algorithms it runs are documented in the BEVFusion / Occupancy / Tracking deep dive.