Edge compute — TOPS is not FPS

Running a perception stack on a vehicle means fitting a transformer-scale model into a fixed power and thermal budget. Vendor TOPS numbers are a peak-arithmetic promise under ideal conditions; what you actually get is set by memory bandwidth, precision, and how well your layers map to the accelerators.

Symbols used on this page are defined in the Nomenclature → metrics.

The NVIDIA edge line-up

Board	INT8 TOPS	Memory / bandwidth	Power	Fits
Jetson Nano (legacy)	~0.5	4 GB / 25 GB/s	5–10 W	classical CV, tiny CNNs
Jetson Orin Nano 8 GB	~40	8 GB / 68 GB/s	7–15 W	YOLO-class 2-D detection
Jetson Orin NX 16 GB	~100	16 GB / 102 GB/s	10–25 W	multi-camera 2-D, light BEV
Jetson AGX Orin 32 GB	~200	32 GB / 204 GB/s	15–40 W	BEV fusion, several models at once
Jetson AGX Orin 64 GB	~275	64 GB / 204 GB/s	15–60 W	full multi-sensor stack + tracking
DRIVE Thor / AGX Thor	~1000+	higher	higher	production AV stacks, transformer-heavy

Numbers are sparse-INT8 peak figures for orientation; dense throughput is roughly half, and achieved throughput is usually well below that.

Why bandwidth, not TOPS, is the wall

A layer is compute-bound only if it does enough arithmetic per byte fetched. The deciding ratio is arithmetic intensity:

$$ I = \frac{\text{FLOPs}}{\text{bytes moved}},\qquad \text{achievable FLOP/s} = \min\!\big(\text{peak FLOP/s},\; I \times \text{bandwidth}\big) $$

Big dense convolutions and GEMMs have high $I$ and can approach peak. But the operations that dominate modern perception often do not: attention at long sequence length, voxelization and scatter ops, bev_pool-style gather/reduce, NMS, and any elementwise chain. These are memory-bound, so they scale with the GB/s column, not the TOPS column — which is why the AGX Orin 32 GB and 64 GB have the same bandwidth and very similar real-world throughput despite different memory sizes.

Practical consequence. Between two boards with the same bandwidth, more TOPS buys you little on a perception pipeline. Between two boards with the same TOPS, more bandwidth buys you a lot. Profile with trtexec and a layer timing dump before choosing.

Getting real throughput

Export to TensorRT and quantize. FP16 is nearly free; INT8 needs calibration but roughly doubles throughput again. Verify accuracy after quantizing — detection heads and regression outputs are the sensitive parts.
Custom CUDA ops are the long tail. A BEVFusion-class model needs its bev_pool and voxelization kernels compiled for the target architecture; these often become the bottleneck and are the hardest part of porting.
Measure latency percentiles, not the mean. A control loop cares about p99: one 200 ms stall matters more than a good average.
Pin the power mode and the clocks. Benchmarks taken at 60 W do not hold at 15 W; use nvpmodel and jetson_clocks deliberately and report which mode you used.
Watch thermals. Sustained throughput on a passively-cooled enclosure can be far below the bench-with-a-fan number.
Don't forget the non-NN work. Deskewing point clouds, undistorting images, and the tracker itself compete for the same memory bandwidth.

Non-NVIDIA options

Platform	Character
Qualcomm Snapdragon Ride	strong power efficiency, automotive-qualified, different toolchain
Texas Instruments TDA4	very low power, safety-oriented, modest NN throughput
Hailo-8 / Hailo-10	efficient dataflow accelerator as a PCIe/M.2 add-on to a host CPU
Intel/Movidius, Google Coral	tiny power envelopes; suited to single small models
Xilinx/AMD Versal, FPGAs	deterministic latency, custom pipelines, high engineering cost

For spaceflight the calculus changes entirely — radiation tolerance and power dominate, and options shrink to rad-hard FPGAs, rad-tolerant SoCs, or COTS parts flown with mitigation and watchdogs.

Planned benchmarks

The same detector exported to TensorRT (FP16 and INT8) across boards: achieved FPS, p50/p99 latency, watts.
Layer-level timing to identify the memory-bound operations in a BEV pipeline.
Sustained vs burst throughput in a sealed enclosure (thermal throttling).
End-to-end: sensor timestamp → published track, measured on hardware.