BEVFusion from scratch

BEVFusion written as algorithm2e-style pseudocode that you can implement with PyTorch, OpenMMLab (mmdet3d, mmdet, mmcv) and third-party libraries. It fuses cameras, LiDAR and, optionally, radar into one bird's-eye-view map for 3D detection on nuScenes.

Each algorithm opens with a short note on why it is needed, then gives numbered steps: the operation, the model or library call, and the tensor shape with named dimensions. Below each algorithm, a table links every call to its exact line in the source (mmdetection3d v1.4.0, mmdetection v3.3.0, mmcv v2.1.0, spconv v2.3.6, nuscenes-devkit). The camera-only occupancy counterpart is on the SurroundOcc page.

Notation

Shapes are written as ℝ[d1, d2, …] or simply (d1, d2, …).

Libraries: torch, mmengine, mmcv ≥ 2 (Voxelization, points_in_boxes), mmdet ≥ 3 (SwinTransformer, focal losses), mmdet3d ≥ 1.4 (nuScenes dataset and transforms, SECOND, SECONDFPN, HardSimpleVFE, projects/BEVFusion), spconv 2.x, nuscenes-devkit, pyquaternion.

BEVFusion — top-level forward pass

Algorithm 0 — BEVFusion forward pass

Why: the map of the whole section. One nuScenes batch goes in and 3D boxes come out. It shows which algorithm runs when and what each one hands to the next. Algorithm 9 adds the two-stage training schedule and the decoding details.

Algorithm 0: BEVFusion forward pass on one nuScenes batch
─────────────────────────────────────────────────────────────────────────────────────────────
Input : nuScenes samples {s_b}, b = 1..B ; mode ∈ {train, infer} ; flag use_radar
Output: boxes ℝ[B,200,9], scores ℝ[B,200], labels [B,200] ; loss L_det (train only)
Models: SwinT + GeneralizedLSSFPN + DepthLSSTransform (camera),
        Voxelization + HardSimpleVFE + BEVFusionSparseEncoder (LiDAR),
        [PillarFeatureNet] (radar), ConvFuser, SECOND + SECONDFPN, TransFusionHead
─────────────────────────────────────────────────────────────────────────────────────────────
 1  foreach sample b do
 2      K, T_cam←lidar, T_img←lidar, T_glob←lidar ← Algorithm 1(s_b)   ▷ each (N_c,4,4)
 3      I, p, r, G, g ← Algorithm 2(s_b)     ▷ (N_c,3,900,1600), (P,5), (P_r,7), (M,9)
 4  end
 5  I', A_img, p', r', G', A_lidar ← Algorithm 3
                                         ▷ random aug in train; fixed resize/crop in infer
 6  F_cam, δ ← Algorithm 4(I', p', K, T_cam←lidar, A_img, A_lidar)   ▷ (B,80,180,180)
 7  F_lidar ← Algorithm 5(p')                                         ▷ (B,256,180,180)
 8  F_radar ← Algorithm 6(r') if use_radar else ∅                     ▷ (B,64,180,180)
 9  preds, H ← Algorithm 7(F_cam, F_lidar, F_radar)
                                         ▷ fuse → (B,512,180,180) → 200 queries
10  if mode = train then return L_det ← Algorithm 8(preds, H, G', g)
11  return boxes, scores, labels ← decode(preds)         ▷ Algorithm 9, lines 9–11

API calls and source code (Algorithm 0):

Data flow at a glance:

BEVFusion — nuScenes data and calibration

Algorithm 1 — Sensor calibration relative to the key-frame LiDAR

Why: nuScenes stores each sensor's pose relative to the ego vehicle, and the ego pose relative to the world, separately for every timestamp. Cameras, LiDAR and radar fire at different moments while the car moves (about 10–15 cm apart at city speed), so a camera ray only lines up with a LiDAR point if both go through the global frame. Every later step (projecting LiDAR into images, lifting pixels into 3D, projecting occupancy voxels into cameras) uses these matrices, and most "it trains but nothing lines up" bugs come from getting this chain wrong.

Algorithm 1: Sensor calibration relative to the key-frame LiDAR
─────────────────────────────────────────────────────────────────────────────────────────────
Input : sample s with LIDAR_TOP and 6 camera records (calibrated_sensor, ego_pose)
Output: K ℝ[N_c,4,4], T_cam←lidar ℝ[N_c,4,4], T_img←lidar ℝ[N_c,4,4], T_glob←lidar ℝ[4,4]
Libs  : pyquaternion.Quaternion ; precomputed by mmdet3d tools/create_data.py nuscenes
─────────────────────────────────────────────────────────────────────────────────────────────
 1  function MakeT(q, t):                    ▷ q = wxyz quaternion, t = translation
 2      T ← I_4 ;  T[0:3,0:3] ← R(q) ;  T[0:3,3] ← t
 3      return T
 4  T_glob←lidar ← MakeT(ego_pose_lidar) · MakeT(calib_lidar)   ▷ lidar → ego → global
 5  foreach camera i = 1..N_c do
 6      T_glob←cam_i ← MakeT(ego_pose_cam_i) · MakeT(calib_cam_i)
 7      T_cam←lidar[i] ← inv(T_glob←cam_i) · T_glob←lidar    ▷ through global: time offsets
 8      K[i] ← pad_4x4(camera_intrinsic_i)
 9      T_img←lidar[i] ← K[i] · T_cam←lidar[i]               ▷ "lidar2img"
10  end
11  return K, T_cam←lidar, T_img←lidar, T_glob←lidar

API calls and source code (Algorithm 1):

Algorithm 2 — Load multi-modal inputs

Why: a single 32-beam LiDAR sweep is very sparse at range, so about 10 sweeps (0.5 s) are stacked to densify the cloud; the time-lag channel lets the network tell old points on moving objects from current ones. Radar adds direct Doppler velocity at long range, which is where cameras and LiDAR are weakest. This algorithm produces the raw tensors every model consumes.

Algorithm 2: Load multi-modal inputs
─────────────────────────────────────────────────────────────────────────────────────────────
Input : sample s ; LiDAR sweeps S = 10 (key + 9) ; radar sweeps S_r = 6
Output: images I ℝ[N_c,3,900,1600], points p ℝ[P,5], radar r ℝ[P_r,7],
        boxes G ℝ[M,9], labels g [M] in 0..K-1
Libs  : mmdet3d (BEV)LoadMultiViewImageFromFiles, LoadPointsFromFile, LoadPointsFromMultiSweeps,
        NuScenesDataset ; nuscenes RadarPointCloud
─────────────────────────────────────────────────────────────────────────────────────────────
 1  I ← LoadMultiViewImageFromFiles(s)       ▷ (N_c,3,900,1600); camera order as in Alg. 1
 2  p ← LoadPointsFromFile(LIDAR_TOP, load_dim=5)    ▷ (P_0,5): x, y, z, intensity, ring
 3  for k ← 1 to S-1 do                      ▷ LoadPointsFromMultiSweeps(sweeps_num=9) (cfg)
 4      p_k ← sweep k ; drop points within 1 m of the sensor
 5      p_k[:,0:3] ← inv(T_glob←lidar,key) · T_glob←lidar,k · p_k[:,0:3]   ▷ into key frame
 6      p_k[:,4] ← t_key − t_k                ▷ time lag Δt replaces ring
 7      p ← concat(p, p_k)
 8  end                                       ▷ p: (P,5) = [x,y,z,int,Δt], P ≈ 250k–350k
 9  if use_radar then                         ▷ optional; official BEVFusion is cam + LiDAR
10      RadarPointCloud.disable_filters()
11      foreach radar j (5 radars), sweep k = 1..S_r do
12          load (x, y, z, rcs, vx_comp, vy_comp) ; move xyz to key lidar frame ;
            rotate (vx, vy) only ; append Δt
13      end                                   ▷ r: (P_r,7), P_r ≈ 1k–3k
14  end
15  G, g ← boxes [x,y,z,w,l,h,θ,vx,vy], class ids   ▷ LiDARInstance3DBoxes, with_velocity
16  return I, p, r, G, g

API calls and source code (Algorithm 2):

BEVFusion — model, losses and training

Shared BEV grid: x, y ∈ [−54, 54] m. LiDAR voxels 0.075 × 0.075 × 0.2 m → 1440 × 1440 × 41. Camera BEV cells 0.3 m → 360 × 360, downsampled to 180 × 180 (0.6 m), which is the grid every branch meets on.

Algorithm 3 — BEVFusion training augmentation

Why: BEVFusion needs strong augmentation on both modalities, but camera and LiDAR must stay geometrically consistent: if the point cloud is rotated by 10°, the camera BEV features must rotate by the same 10°. Recording each augmentation as a matrix (A_img, A_lidar) lets Algorithm 4 undo the image augmentation and apply the LiDAR augmentation when it places pixels in 3D, so both branches land on the same BEV grid.

Algorithm 3: BEVFusion training augmentation (every transform kept as a matrix)
─────────────────────────────────────────────────────────────────────────────────────────────
Input : I, p, r, G
Output: I' ℝ[N_c,3,256,704], A_img ℝ[N_c,4,4], p', r', G', A_lidar ℝ[4,4]
Libs  : projects/BEVFusion ImageAug3D, BEVFusionGlobalRotScaleTrans, BEVFusionRandomFlip3D ;
        mmdet3d PointsRangeFilter, ObjectRangeFilter, ObjectNameFilter, PointShuffle,
        ObjectSample
─────────────────────────────────────────────────────────────────────────────────────────────
 1  foreach camera i do    ▷ ImageAug3D(final_dim=[256,704], resize_lim=[0.38,0.55],
                           ▷            rot_lim=[−5.4°,5.4°], rand_flip=True) (cfg)
 2      sample scale σ, crop box, flip f, rotation φ ; warp image to 256×704
 3      A_img[i] ← 4×4 matrix of (resize, crop, flip, rotate)    ▷ undone in Algorithm 4
 4  end
 5  normalise I' (ImageNet mean/std, RGB)                        ▷ (N_c,3,256,704)
 6  sample yaw ψ ∈ [−45°, 45°] (±0.785 rad), scale ∈ [0.9, 1.1], translation σ = 0.5 m,
    random BEV flips                          ▷ BEVFusionGlobalRotScaleTrans, RandomFlip3D
 7  A_lidar ← 4×4 matrix of (rotate, scale, translate, flip)
 8  p', r', G' ← A_lidar · (p, r, G)          ▷ boxes incl. yaw and velocity
 9  filter to range [−54, −54, −5, 54, 54, 3]
10  if LiDAR-only stage then ObjectSample (GT-paste) else no GT-paste
                                              ▷ pasted objects have no pixels
11  return I', A_img, p', r', G', A_lidar
                       ▷ collate: I' → (B,N_c,3,256,704), matrices → (B,N_c,4,4)

API calls and source code (Algorithm 3):

Algorithm 4 — Camera branch: Swin, FPN, depth-aware LSS, BEV pooling

Why: cameras give dense appearance and semantics but no depth. This branch implements Lift-Splat-Shoot: each pixel predicts a probability over 118 depth bins and spreads its feature along its viewing ray (lift), and all frustum features are summed into a top-down grid (splat). The result is a camera feature map in the same bird's-eye-view space as LiDAR, so the two can be fused by concatenation. BEVFusion also feeds projected LiDAR depth into the depth network, which sharpens the depth distribution considerably.

Algorithm 4: Camera branch — Swin, FPN, depth-aware LSS, BEV pooling
─────────────────────────────────────────────────────────────────────────────────────────────
Input : I' ℝ[B,N_c,3,256,704], points p', K, T_cam←lidar, A_img, A_lidar
Output: camera BEV F_cam ℝ[B,80,180,180], depth probabilities δ
Libs  : mmdet SwinTransformer ; projects/BEVFusion GeneralizedLSSFPN, DepthLSSTransform,
        ops/bev_pool (CUDA)
─────────────────────────────────────────────────────────────────────────────────────────────
 1  x ← reshape(I')                              ▷ (B·N_c, 3, 256, 704)
 2  c8, c16, c32 ← SwinT(x)     ▷ embed 96, depths [2,2,6,2], heads [3,6,12,24],
                                ▷ out_indices [1,2,3], ImageNet Swin-T init (cfg)
                                ▷ c8 (B·N_c,192,32,88), c16 (…,384,16,44), c32 (…,768,8,22)
 3  f ← GeneralizedLSSFPN(c8, c16, c32)[0]      ▷ in [192,384,768] → 256 (cfg)
                                                ▷ (B·N_c,256,32,88): fH = 32, fW = 88
 4  ▷ Depth prior (the "Depth" in DepthLSS)
 5  u ← A_img · K · T_cam←lidar · inv(A_lidar) · p'_xyz ; keep in-image hits ;
    scatter depths → d_img                      ▷ (B·N_c,1,256,704), sparse
 6  e ← dtransform(d_img)                       ▷ conv 1→8→32→64, stride 8: (B·N_c,64,32,88)
 7  ▷ Lift
 8  y ← depthnet(concat(f, e))                  ▷ (B·N_c, D+80, 32, 88)
                                                ▷ D = (60−1)/0.5 = 118, dbound [1,60,0.5]
 9  δ ← softmax over D of y[:D]                 ▷ (B·N_c,118,32,88): p(depth | pixel)
10  c ← y[D:]                                   ▷ (B·N_c,80,32,88): context
11  F_fr ← δ ⊗ c                                ▷ outer product → (B,N_c,118,32,88,80)
12  ▷ Geometry: 3D position of every frustum cell
13  frustum ← all (u, v, d) on the augmented image      ▷ (D, fH, fW, 3)
14  (u, v, 1) ← inv(A_img) · (u, v, 1)          ▷ undo resize/crop/flip/rotate
15  g_cam ← inv(K) · [u·d, v·d, d]              ▷ pixel → point on the camera ray
16  g ← A_lidar · inv(T_cam←lidar) · g_cam      ▷ camera → lidar → augmented lidar
                                                ▷ (B,N_c,118,32,88,3), metres
17  ▷ Splat (BEV pooling)
18  idx ← floor((g − [−54, −54, −10]) / [0.3, 0.3, 20]) ; drop out-of-range
                                ▷ xbound/ybound [−54,54,0.3], zbound [−10,10,20] (cfg)
19  F_bev ← bev_pool(F_fr, idx, grid 360×360×1) ▷ sum per cell: (B,80,1,360,360)
                                                ▷ fallback: torch_scatter.scatter_add
20  F_bev ← flatten channel and Z               ▷ (B,80,360,360)
21  F_cam ← downsample(F_bev)                   ▷ conv3×3 s1 → s2 → s1 + BN + ReLU
                                                ▷ (B,80,180,180)
22  return F_cam, δ

API calls and source code (Algorithm 4):

Algorithm 5 — LiDAR branch: voxelization and sparse 3D convolution

Why: LiDAR gives accurate geometry and, in practice, most of the detection accuracy. Raw points are irregular, so they are grouped into small voxels and only the non-empty ones (about 5 % of the grid) are processed with sparse 3D convolutions, which is fast and memory-efficient. The height axis is then folded into channels to give a BEV map aligned with the camera BEV.

Algorithm 5: LiDAR branch — voxelization and sparse 3D convolution
─────────────────────────────────────────────────────────────────────────────────────────────
Input : points {p'_b ℝ[P_b,5]}, b = 1..B
Output: LiDAR BEV F_lidar ℝ[B,256,180,180]
Libs  : mmcv.ops.Voxelization ; mmdet3d HardSimpleVFE ;
        projects/BEVFusion BEVFusionSparseEncoder (on spconv)
─────────────────────────────────────────────────────────────────────────────────────────────
 1  foreach sample b do
 2      v_b, i_b, n_b ← Voxelization(p'_b)
                  ▷ voxel [0.075,0.075,0.2], max_num_points 10, max_voxels 120k/160k (cfg)
                  ▷ v_b (V_b,10,5), i_b (V_b,3) as (z,y,x), n_b (V_b)
 3      i_b ← [b | i_b]                       ▷ prepend batch index
 4  end
 5  v, i, n ← concat over b                   ▷ (V,10,5), (V,4), (V)
 6  h ← HardSimpleVFE(v, n)                   ▷ mean of the points in each voxel: (V,5)
 7  S ← SparseConvTensor(h, i, spatial_shape=[41,1440,1440], batch_size=B)
 8  S ← BEVFusionSparseEncoder(S)  ▷ SubMConv3d + 3 stride-2 SparseConv3d stages,
                                   ▷ channels 16→32→64→128 (cfg)
 9  F ← dense(S)                              ▷ (B,128,Z'=2,180,180)
10  F_lidar ← fold Z' into channels           ▷ (B,256,180,180)
11  return F_lidar

API calls and source code (Algorithm 5):

Algorithm 6 — Radar branch (optional extension)

Why: radar is cheap, works in rain, fog and darkness, sees beyond 100 m and measures radial velocity directly, but it is very sparse (a few hundred points) and has almost no height information. A pillar encoder (PointPillars) turns it into a BEV map on the same grid so it can be a third input to the fuser. The official BEVFusion does not include this branch; it follows published camera–radar BEV models such as CRN and RCBEVDet.

Algorithm 6: Radar branch (optional)
─────────────────────────────────────────────────────────────────────────────────────────────
Input : radar {r'_b ℝ[P_r,b,7]}
Output: radar BEV F_radar ℝ[B,64,180,180]
Libs  : mmcv.ops.Voxelization ; mmdet3d PillarFeatureNet, PointPillarsScatter
─────────────────────────────────────────────────────────────────────────────────────────────
 1  v, i, n ← Voxelization(r', voxel=[0.6,0.6,8], max_num_points=10)
                                       ▷ pillars on the same 180×180 grid: (V_r,10,7)
 2  h ← PillarFeatureNet(v, n, i)      ▷ PointNet per pillar, 7 → 64: (V_r,64)
 3  F ← PointPillarsScatter(h, i, [180,180])       ▷ (B,64,180,180)
 4  F_radar ← Conv3×3 ∘ Conv3×3 (F)    ▷ spread the sparse returns: (B,64,180,180)
 5  return F_radar

API calls and source code (Algorithm 6):

Algorithm 7 — Fusion, BEV decoder and TransFusion head

Why: once every sensor lives on the same 180 × 180 grid, fusion is a concatenation followed by a convolution; the convolution also corrects small misalignments between the camera and LiDAR BEV maps. The SECOND backbone and FPN add multi-scale context (whole vehicles, lane structure). The TransFusion head then finds objects: a class heatmap proposes 200 candidate centres, and a transformer decoder refines each candidate by attending to the whole BEV map, predicting box, velocity and class directly, without anchors.

Algorithm 7: Fusion, BEV decoder and TransFusion head
─────────────────────────────────────────────────────────────────────────────────────────────
Input : F_cam, F_lidar, optional F_radar
Output: predictions for Q = 200 queries, dense heatmap H ℝ[B,K,180,180]
Libs  : projects/BEVFusion ConvFuser, TransFusionHead ; mmdet3d SECOND, SECONDFPN
─────────────────────────────────────────────────────────────────────────────────────────────
 1  F ← ConvFuser(concat(F_cam, F_lidar [, F_radar]))
                     ▷ in [80,256(,64)] → 256, conv3×3 + BN + ReLU: (B,256,180,180)
 2  s1, s2 ← SECOND(F)     ▷ out [128,256], layer_nums [5,5], strides [1,2] (cfg)
                           ▷ s1 (B,128,180,180), s2 (B,256,90,90)
 3  F ← SECONDFPN(s1, s2)  ▷ out [256,256], upsample [1,2], concat: (B,512,180,180)
 4  E ← shared_conv(F)                    ▷ (B,128,180,180)
 5  H ← heatmap_head(E)                   ▷ class-wise centre heatmap logits: (B,10,180,180)
 6  Ĥ ← sigmoid(H) masked to its 3×3 local maxima
                              ▷ peak NMS; kernel 1 for pedestrian and traffic cone
 7  (k_q, c_q), q = 1..Q ← top-200 of Ĥ   ▷ class and cell of the strongest peaks
 8  q ← E[:, c_q] + class_encoding(onehot(k_q))         ▷ (B,128,200)
 9  q_pos ← xy(c_q) ; k ← flatten(E) ; k_pos ← BEV grid ▷ (B,200,2) ; (B,128,32400)
10  for layer ← 1 to num_decoder_layers (= 1) do
11      q ← SelfAttention_8heads(q + MLP(q_pos))
12      q ← CrossAttention_8heads(q + MLP(q_pos), k + MLP(k_pos)) ; q ← FFN_256(q)
                                                        ▷ (B,128,200)
13  end
14  predict with 1D conv heads on q:                    ▷ TransFusionHead common_heads (cfg)
        center (B,2,200) cell offset ; height (B,1,200) ; dim (B,3,200) log(w,l,h) ;
        rot (B,2,200) sin θ, cos θ ; vel (B,2,200) ; cls (B,10,200)
15  return predictions, H, peak scores Ĥ[k_q, c_q]

API calls and source code (Algorithm 7):

Algorithm 8 — BEVFusion targets and losses

Why: detection needs three kinds of supervision. The Gaussian heatmap teaches the network where object centres are, so query initialisation in Algorithm 7 starts in the right places. Hungarian matching assigns each ground-truth box to exactly one query, so there are no duplicate detections and no NMS is needed. Focal and L1 losses then train classification and box regression on the matched pairs, with lower weight on the noisier velocity terms.

Algorithm 8: BEVFusion targets and losses
─────────────────────────────────────────────────────────────────────────────────────────────
Input : predictions, H, ground truth G' ℝ[M,9], g
Output: L_det
Libs  : mmdet3d gaussian_radius, draw_heatmap_gaussian ;
        projects/BEVFusion HungarianAssigner3D, TransFusionBBoxCoder ;
        mmdet GaussianFocalLoss, FocalLoss, L1Loss ; scipy linear_sum_assignment
─────────────────────────────────────────────────────────────────────────────────────────────
 1  H* ← zeros(B,10,180,180)
 2  foreach box m do draw a Gaussian at its centre cell in channel g_m,
                    radius = max(2, gaussian_radius(w_m, l_m))
 3  Cost ← 0.15·focal + 0.25·BEV_L1 + 0.25·IoU3D      ▷ cost matrix (200, M) (cfg)
 4  π ← Hungarian(Cost)                                ▷ one query per GT box
 5  t_m ← [Δx, Δy, z, log w, log l, log h, sin θ, cos θ, vx, vy]
                                   ▷ TransFusionBBoxCoder.encode, out_size_factor 8
 6  L_hm  ← GaussianFocal(H, H*)                       ▷ weight 1.0
 7  L_cls ← Focal(γ=2, α=0.25)(cls, g_π)               ▷ unmatched → background ; weight 1.0
 8  L_box ← Σ w ⊙ |t̂_π − t|,  w = [1,1,1,1,1,1,1,1,0.2,0.2]   ▷ weight 0.25 (cfg)
 9  L_det ← L_hm + L_cls + 0.25·L_box
                              ▷ optional: BCE depth loss on δ against LiDAR depth
10  return L_det

API calls and source code (Algorithm 8):

Algorithm 9 — BEVFusion end to end (training and inference)

Why: this ties the pieces together and follows BEVFusion's two-stage recipe: train the LiDAR detector first (it converges quickly and benefits from GT-paste augmentation), then add the camera branch and fine-tune briefly. Training everything from scratch is slower and tends to under-use the camera branch.

Algorithm 9: BEVFusion end to end
─────────────────────────────────────────────────────────────────────────────────────────────
Input : dataset 𝒟 (nuScenes train / val infos)
Output: trained model ; boxes (B,200,9), scores (B,200), labels (B,200)
Libs  : mmengine Runner, AdamW ; nuScenes detection eval via mmdet3d NuScenesMetric
─────────────────────────────────────────────────────────────────────────────────────────────
 1  ▷ Stage 1, LiDAR only (bevfusion_lidar_voxel0075_second_secfpn_…):
    ▷   Algorithms 5 → 7 without camera, 20 epochs, AdamW + cyclic LR,
    ▷   GT-paste on and switched off after epoch 15 (DisableObjectSampleHook)
 2  ▷ Stage 2, camera + LiDAR (bevfusion_lidar-cam_voxel0075_second_secfpn_…):
    ▷   init from stage 1 + Swin-T, GT-paste off, 6 epochs, AdamW lr 2e-4
 3  foreach batch in 𝒟 do
 4      Algorithm 2 → Algorithm 3
 5      F_cam, δ ← Alg. 4 ; F_lidar ← Alg. 5 ; F_radar ← Alg. 6 if use_radar
 6      predictions ← Alg. 7 ; L_det ← Alg. 8 ; backprop ; AdamW step
 7  end
 8  ▷ Inference
 9  score ← sigmoid(cls) · Ĥ[k_q, c_q]
10  boxes ← TransFusionBBoxCoder.decode(center, height, dim, rot, vel)   ▷ (B,200,9)
11  keep boxes inside post_center_range [−61.2, −61.2, −10, 61.2, 61.2, 10] ; no NMS needed
12  evaluate mAP / NDS

API calls and source code (Algorithm 9):

BEVFusion — notes for assembling it

1. Augmentation consistency. Any image or 3D augmentation must update A_img and A_lidar (Algorithm 3), otherwise camera and LiDAR BEV features stop lining up.
2. Memory. The LSS frustum (B·N_c, 80, 118, 32, 88) is about 0.6 GB in fp32 at B = 1, so use the bev_pool CUDA op rather than a scatter in PyTorch.
3. Occupancy from the BEVFusion map. The fused BEV (B, 256, 180, 180) has no height axis: to predict occupancy, lift it with a channel-to-height head (Conv2d 256 → C·Z, reshape to (B, C, 180, 180, Z)) and resample 0.6 m / 180² to 0.5 m / 200² with grid_sample. See SurroundOcc for a camera-only occupancy model.
4. Sanity checks. Overlay LiDAR projected into each camera, compare camera BEV and LiDAR BEV after augmentation, and draw GT boxes in the same frame before training.