Middleware and real-time engineering

1 · Follow a message through ROS 2

A node is a collection of entities, not a thread or process. A context initializes the middleware; a container holds multiple nodes in one process. Performance diagnosis starts at the correct layer: CDR fragmentation/transport for a multi-megabyte cloud, executor dispatch for callback jitter, CUDA event dependencies for inference stalls.

Constrain discovery on the vehicle network; pin the rmw version and test large-message transport under packet loss. One participant per node is not a portable assumption: participant ownership is implementation/version-specific.

Time contract: stamp at hardware capture (PTP/trigger), use ROS time for message stamps and bag replay, a monotonic steady clock for latency and watchdog durations, and never assume cross-topic DDS ordering. For six cameras plus a LiDAR sweep, exact sync is brittle without common triggers; approximate sync may wait a whole period. Sweep-anchored sync picks the nearest image within a bounded slop and carries a valid-view mask. Cache static TF from map → odom → base_link → sensors at configuration; never block a model callback on TF. Use continuous odom for derivatives: map → odom can jump on loop closure.

2 · Design QoS per link, not per project

Ask five questions: (1) Is a late sample worse than a missing one? (2) How large is it, and can it cross a process boundary without fragments? (3) Must a restarted subscriber receive the previous value? (4) What depth absorbs producer jitter without retaining stale work? (5) Which deadline, lifespan and liveliness events expose failure? KEEP_LAST(1) bounds a queue but silently drops data: publish sequence-gap counters.

ros2 topic info /lidar/points --verbose
ros2 topic echo /lidar/points --qos-reliability best_effort --qos-depth 1 --once

Wire requested/offered deadline, liveliness, and incompatible-QoS event callbacks into health_node. A deadline is a signal, not a hard scheduling guarantee. A manual liveliness lease must be asserted after completed inference, not before a GPU hang; separately poll the CUDA completion event. Size DDS resource limits, socket buffers and fragment limits, and prefer async publishing if large inter-process serialization would block a callback.

3 · Executors: dispatch is a scheduling decision

An executor waits on subscriptions, timers, services, clients and guard conditions then runs ready callbacks. The usual single/multithreaded executors do not promise strict priority, fairness or a worst-case dispatch time. Dispatch details (including timer ordering) change across distributions; measure and test on your pinned ROS version. A one-thread executor serializes everything; a multithreaded executor permits parallel groups, but its default mutually-exclusive group serializes all callbacks within each node.

Put a stateful tracker and each single-context TensorRT stage into MutuallyExclusive groups; use Reentrant only where concurrent callbacks have independent storage. Explicitly routed groups must be created with automatically_add_to_executor_with_node=false, then attached by add_callback_group(group, node_base) before spinning. Put only remaining default groups into the housekeeping executor; do not add the same node indiscriminately to multiple executors.

auto gpu_group = node->create_callback_group(
    rclcpp::CallbackGroupType::MutuallyExclusive, false);
gpu_exec.add_callback_group(gpu_group, node->get_node_base_interface());
std::thread gpu_thread([&] {
    set_affinity({6,7,8,9});  // implemented in perception_common/thread_utils.hpp
    gpu_exec.spin();
});

Failure drills: same-group synchronous service call deadlocks; default groups unexpectedly serialize; an overactive timer starves sensor work; depth-one queues hide skips; reentrant tracker corrupts identities; engine deserialization or TF wait inside a callback freezes the pipeline; implicit hardware-concurrency thread counts thrash an embedded SoC. Use traces and counters to demonstrate each fix.

4 · One process, multiple nodes and threads

Register each component with the component macro and forward the incoming NodeOptions to its base constructor: remaps, parameter overrides and intra-process configuration otherwise disappear. For bring-up, use component_container_mt plus use_intra_process_comms=true; for bounded executor ownership use the custom perception_container_main with three executors. Dynamic components allow loading heads at runtime; the custom main makes allocation, affinities and explicit group routing inspectable.

Lifecycle bring-up: configure calibration, deserialize engines and allocate fixed pools before motion; activate publishers only after checks pass; on GPU fault deactivate, emit FAULT, then attempt cleanup/reconfigure while the GNC process owns safe stop. Hot-swapping a detector without checking input/output contracts and vehicle state is not safe merely because dynamic loading is available.

5 · Four different meanings of “zero copy”

A/B: a moved image has the same object address at publisher and owning subscriber. With multiple read-only consumers prefer a shared ownership callback; never retain a reference beyond callback lifetime. Volatile durability is necessary for the intra-process hot path; depth-one can discard an unprocessed frame, so count it. An external subscriber may cause a second serialized publication even while internal delivery is copy-free; never promise no serialization just because the nodes are composed.

auto image = std::make_unique<sensor_msgs::msg::Image>();
auto* before = image.get();
pub->publish(std::move(image)); // compare 'before' with owning callback's m.get()

D: the GPU adapter carries a pool-owned CUDA allocation, dimensions, encoding, producer PID and readiness event. The consumer calls cudaStreamWaitEvent(consumer_stream, ready_event), not device-wide synchronization. An address alone is not inter-process transferable: use CUDA IPC/NvSci with ownership protocol if needed, or stage into a safe wire type. On integrated-GPU SoCs evaluate mapped pinned/NvSci buffers; on discrete GPUs budget H2D and host pinning. “GPU-direct camera” requires a capture path that imports DMA buffers into the device; wrapping a CPU image in a pointer does not eliminate H2D.

6 · Loaned messages and the driver boundary

For a separate privileged LiDAR driver, small packets can cross via standard messages. If a decoded full sweep must cross processes without a CPU copy, use an rmw-supported, fixed-capacity point array with num_points; for a 1920×1080 RGB image the fixed RGB8 array needs 6,220,800 bytes. Standard Image.data and PointCloud2.data are dynamically sized, so borrow_loaned_message() cannot be assumed to provide true middleware-backed zero copy for them. can_loan_messages() and transport instrumentation distinguish a real loan from heap fallback.

Prefer the decoder inside the perception container: de-skew each point with its time offset and ego odometry, then upload/decode into GPU tensors once. A fixed-capacity packet or sweep sample is useful only when measured cross-process cost justifies custom types. Fast DDS SHM + async publish and Cyclone PSMX/iceoryx both require release-specific configuration, sufficient chunk pools and verified fallback behavior; SHM transport is not synonymous with a middleware loan.

7 · Services, actions, parameters and lifecycle

Declare parameters with descriptors, ranges and read-only restrictions; validate through an on-set callback, snapshot a consistent configuration at the frame boundary, and version changes in recordings. Use configure → activate → deactivate → cleanup lifecycle transitions to prevent engine compilation inside an active deadline. Do not call spin_until_future_complete from a callback whose response needs the same single-threaded/mutually-exclusive executor; use asynchronous client callbacks or a separately routed group.

8 · Make a real-time claim falsifiable

Preallocate CUDA, host and message pools; prebuild TRT engines; lock critical memory and prefault stacks; avoid filesystem/allocations/blocking TF in callbacks. Pin critical threads and prioritize GNC separately from GPU work; SCHED_FIFO requires explicit privileges and bounded callback execution or it can starve the entire system. Use stream events for camera/LiDAR overlap and serialize the fuser only when its inputs are ready. Keep only the newest pending sweep when inference exceeds its 100 ms period; drop and count the rest rather than accumulating latency.

Compare ROS tracepoints (publish → callback), CUDA stream events, PTP/hardware timestamps and steady-clock age. An executor hop microbenchmark is not a total system deadline proof. Also measure detection quality conditioned on dropped frames; a stack that meets 95 ms by discarding hard frames has not solved the vehicle problem.