Feature Pyramid Networks: the maths, the architecture, and a ResNet-50 → FPN → YOLO detector

A detector has to find a 12-pixel drone and a 400-pixel truck in the same frame. A plain CNN gives you one feature map at stride 32, where the drone is a third of one cell. Feature Pyramid Networks (Lin, Dollár, Girshick, He, Hariharan and Belongie, CVPR 2017) fixed this with a small, almost entirely linear piece of network that turns any backbone’s internal hierarchy into a pyramid with strong features at every scale. Nearly every detector since, from Mask R-CNN and RetinaNet to the YOLO family, has some descendant of it as its neck.

This post goes through the paper’s construction and its maths, then builds a concrete detector: a ResNet-50 backbone, an FPN neck and a YOLO-style head, with every tensor shape written out and working torchvision code.

1. The scale problem and four ways to attack it

A convolutional backbone trades resolution for meaning. Early stages are high resolution but only know about edges and texture. Late stages know about objects but have been downsampled 32×. The paper frames the options with four sketches, redrawn here.

• (a) Image pyramid. Resize the image to several scales and run the whole network on each. Every level gets strong features, but inference cost multiplies by the number of scales, and training this way is usually impractical.
• (b) Single feature map. Predict only from the last stage (classic Faster R-CNN, YOLOv1). Fast, but small objects are smeared into a single coarse cell.
• (c) Pyramidal feature hierarchy. Predict from several stages of the backbone (SSD). The multi-scale maps are free, but the high-resolution ones come from shallow layers with weak semantics, so they do not help much with small objects. SSD actually starts from a fairly deep layer for this reason.
• (d) FPN. Keep (c)’s free hierarchy, then add a top-down pathway that pushes the semantics of the deep layers back down into the high-resolution ones, and predict from every level of the result.

2. The construction, as equations

2.1 Bottom-up pathway

This is just the backbone’s forward pass. ResNet has stages whose outputs halve in resolution. FPN takes the last residual block of stages conv2 to conv5 and calls them \(C_2, C_3, C_4, C_5\). With \(\ell\) the level index, level \(\ell\) has stride \(2^\ell\) relative to the input:

The stem output (conv1, stride 2) is left out because its memory footprint is large and its features are too low-level to be worth it.

2.2 Lateral connections

Each \(C_\ell\) has a different channel count, so it is first projected to a common width \(d\) (256 in the paper) with a \(1\times1\) convolution. A \(1\times1\) convolution is a matrix applied independently at every pixel:

It does not mix spatial positions. Its only job is to put every level in the same \(d\)-dimensional space so the levels can be added together.

2.3 Top-down pathway

Start from the coarsest level and walk down. Writing \(U\) for 2× nearest-neighbour upsampling and \(K_\ell\) for a \(3\times3\) convolution:

Two details are easy to get wrong when reimplementing. First, the map that is upsampled into the next level is \(M_{\ell+1}\), the sum before the \(3\times3\) conv, not the output \(P_{\ell+1}\). torchvision’s implementation does exactly this (its last_inner variable). Second, there are no non-linearities anywhere in these extra layers. The authors report that adding them made little difference, and the neck is a purely linear map of the backbone features.

2.4 Nearest-neighbour upsampling as an operator

Nearest-neighbour 2× upsampling copies each coarse value into a \(2\times2\) block:

It has no parameters. Its adjoint \(U^\top\) sums each \(2\times2\) block of a fine gradient back onto one coarse cell, which is how gradients travel up the top-down path during training. The blocky copy is also why the \(3\times3\) conv \(K_\ell\) exists: the paper uses it to reduce the aliasing that upsampling introduces before the map is used for prediction.

2.5 Unrolling the recursion

Because every operation in the neck is linear, the recursion can be unrolled into a closed form:

Each output level is a smoothed sum of its own lateral projection and the projections of every deeper level, upsampled to its resolution. \(P_2\) contains information from all four backbone stages; \(P_5\) only from \(C_5\). That is the precise sense in which FPN gives high-resolution levels strong semantics.

The gradient view is just as useful. A loss on a fine level reaches every deeper stage:

So \(C_5\) is trained by the small-object losses on \(P_2\) as well as by the large-object losses on \(P_5\). In a plain pyramidal hierarchy (option c) it would only see the latter.

2.6 Why every level has the same width

Fixing \(d=256\) at every level means one prediction head can be shared across the whole pyramid. The paper does this for both the RPN and the Fast R-CNN head, and found that separate per-level heads performed about the same. A shared head means the same weights must recognise a car at stride 8 and stride 32, which is the feature-pyramid version of the scale invariance an image pyramid gives you.

2.7 Assigning objects to levels

An object of size \(s\) pixels covers \(s / 2^\ell\) cells at level \(\ell\). To keep that footprint roughly constant, the level should grow with \(\log_2 s\). The paper uses this rule in two places.

RPN anchors. One anchor area per level: \(32^2, 64^2, 128^2, 256^2, 512^2\) on \(P_2\) to \(P_6\), each with aspect ratios 1:2, 1:1 and 2:1, for 15 anchor types. The anchor side is always 8 × the stride. \(P_6\) is a stride-2 subsampling of \(P_5\) and exists only to host the \(512^2\) anchors.

RoI pooling. A region of width \(w\) and height \(h\) (in input pixels) is pooled from level

clamped to \([2, 5]\). The constants come from ImageNet: a 224-pixel region is the canonical pre-training size and maps to \(C_4\), which the single-scale ResNet Faster R-CNN baseline also used. Halve the region and it drops one level.

Some numbers: \(\sqrt{wh} = 224 \rightarrow P_4\), \(112 \rightarrow P_3\), \(448 \rightarrow P_5\), and a 50-pixel box gives \(\lfloor 4 – 2.16 \rfloor = 1\), clamped to \(P_2\).

3. What the paper measured

All ablations below are ResNet-50 on COCO minival. RPN proposals are scored by average recall at 1000 proposals (AR_1k) and on small objects (AR_s).

Each ablation removes one idea and loses most of the gain. Without the top-down path, the fine levels lack semantics. Without the laterals, the top-down maps are semantically strong but their locations have been through many rounds of down- and upsampling and are imprecise. Using only \(P_2\) (which still benefits from top-down information) recovers part of the gain, but predicting at every level is better and far cheaper in anchors.

In the full Faster R-CNN, FPN raised COCO minival AP from 31.6 to 33.9 against a strong single-scale baseline with the same backbone, and AP@0.5 from 53.1 to 56.9. With ResNet-101 the single model reached 36.2 AP on test-dev (18.2 on small objects), ahead of every single-model entry the paper compared against, including the COCO 2016 challenge winners, without extra bells and whistles. It was also faster: about 0.15 s per image on an M40 versus 0.32 s for the single-scale baseline, partly because the RoI head became much lighter.

4. Every tensor shape, ResNet-50 + FPN at 640 × 640

The neck is light in parameters (3.34 M for P2–P5, against roughly 23.5 M for the ResNet-50 trunk) but not in compute, and the compute is concentrated at the finest level:

The \(3\times3\) output convs dominate, and they scale with the number of cells. \(P_2\) has 16× the cells of \(P_4\), so it costs 16.8 GMAC on its own, 2.6× the 6.4 GMAC of \(P_3\) to \(P_5\) together. On an edge device the feature maps themselves are the bigger problem: \(P_2\) in FP16 is 13 MB per image before the head touches it, and memory bandwidth, not TOPS, is usually the limit. This is the main reason real-time detectors stop at stride 8.

One implementation detail: torchvision upsamples with F.interpolate(..., size=C_l.shape[-2:], mode="nearest") rather than a fixed factor of 2. With an input that is not a multiple of 32, say 610 pixels, \(C_4\) is 39×39 while \(2\times\) \(C_5\) would be 40×40, and the add would fail. Upsampling to the lateral’s exact size avoids that.

5. FPN as a neck

Modern detectors are described as three parts:

• Backbone: turns pixels into a feature hierarchy \(C_3, C_4, C_5\). ResNet, CSPDarknet, ConvNeXt, a ViT with an adapter, DINOv2.
• Neck: mixes the hierarchy into a pyramid \(P_3, P_4, P_5\) with equal width. FPN is the original; PANet adds a second, bottom-up path after FPN’s top-down one; BiFPN (EfficientDet) repeats that with learned weighted sums. YOLOv3 used an FPN-style top-down neck with concatenation instead of addition, and YOLOv4 onward use FPN + PAN.
• Head: predicts boxes and classes on every pyramid level. Two-stage (RPN + RoI head), or dense single-stage (RetinaNet, FCOS, YOLO).

Because FPN’s contract is simple, a dict of maps in and a dict of 256-channel maps out, the three parts can be swapped independently. That is what the rest of this post does.

6. ResNet-50 → FPN → YOLO head

Here is a complete single-stage detector. It uses only \(P_3\) to \(P_5\), like the YOLO family, and an anchor-free decoupled head in the style of YOLOX and YOLOv8 (simplified: plain box regression rather than YOLOv8’s distribution focal loss).

The output count is worth committing to memory: at 640 × 640, the head produces one prediction per cell of each level, \(80^2 + 40^2 + 20^2 = 8400\) in total, each with \(4 + 1 + 80 = 85\) numbers for COCO. At 1280 × 1280 that becomes 33,600, which is why input size dominates NMS and post-processing time on edge hardware.

6.1 The head and the box decoding

For a cell \((i, j)\) at level stride \(s\), the head predicts four raw distances \(\hat{t}\), an objectness logit \(\hat{o}\) and class logits \(\hat{z}_k\). The cell centre and the box are

Multiplying by the stride is the dense-head version of the FPN level rule: the same raw output means a box twice as big one level up, so the head can share its sense of “object-sized” across levels. The classification and objectness biases are initialised to \(b = -\log\frac{1-\pi}{\pi} \approx -4.6\) with \(\pi = 0.01\), so every cell starts by predicting “background with 1% confidence” and the loss is not swamped by thousands of easy negatives in the first iterations (the trick from RetinaNet).

7. Code with the torchvision FPN API

torchvision gives you the neck as torchvision.ops.FeaturePyramidNetwork, plus helpers that wire it to a ResNet. Two ways to build the backbone + neck, then the full detector.

7.1 Explicit: create_feature_extractor + FeaturePyramidNetwork

import torch
from torchvision.models import resnet50, ResNet50_Weights
from torchvision.models.feature_extraction import create_feature_extractor
from torchvision.ops import FeaturePyramidNetwork

# 1) Backbone: pull C3, C4, C5 out of a stock ResNet-50
backbone = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
body = create_feature_extractor(
    backbone,
    return_nodes={"layer2": "p3", "layer3": "p4", "layer4": "p5"},  # keys we want downstream
)

# 2) Neck: in_channels must match the backbone, ordered fine -> coarse
fpn = FeaturePyramidNetwork(in_channels_list=[512, 1024, 2048], out_channels=256)

x = torch.randn(2, 3, 640, 640)
feats = body(x)                 # dict: p3, p4, p5 (still C3, C4, C5 here)
for k, v in feats.items():
    print("C", k, tuple(v.shape))
# C p3 (2, 512, 80, 80)
# C p4 (2, 1024, 40, 40)
# C p5 (2, 2048, 20, 20)

pyramid = fpn(feats)            # OrderedDict with the same keys
for k, v in pyramid.items():
    print("P", k, tuple(v.shape))
# P p3 (2, 256, 80, 80)
# P p4 (2, 256, 40, 40)
# P p5 (2, 256, 20, 20)

FeaturePyramidNetwork(in_channels_list, out_channels, extra_blocks=None, norm_layer=None) builds one \(1\times1\) inner block and one \(3\times3\) layer block per input level, and its forward takes a dict ordered from the finest map to the coarsest and returns an OrderedDict with the same keys. Pass norm_layer=nn.BatchNorm2d if you want normalised neck convs (the paper’s neck has none), or extra_blocks=LastLevelMaxPool() / LastLevelP6P7(...) for the P6 or RetinaNet-style P6–P7 levels.

7.2 One call: resnet_fpn_backbone

import torch
from torchvision.models import ResNet50_Weights
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone

bb = resnet_fpn_backbone(
    backbone_name="resnet50",
    weights=ResNet50_Weights.IMAGENET1K_V2,
    trainable_layers=5,          # 0..5 stages unfrozen; default 3
    returned_layers=[2, 3, 4],   # layer2..layer4 -> C3..C5 (default [1, 2, 3, 4])
)
print(bb.out_channels)           # 256

out = bb(torch.randn(2, 3, 640, 640))
for k, v in out.items():
    print(k, tuple(v.shape))
# 0    (2, 256, 80, 80)   P3
# 1    (2, 256, 40, 40)   P4
# 2    (2, 256, 20, 20)   P5
# pool (2, 256, 10, 10)   P6, added by the default LastLevelMaxPool

Three defaults to know about. The backbone uses FrozenBatchNorm2d (fine for fine-tuning with small batches; pass norm_layer=nn.BatchNorm2d to train BN). Only the last 3 stages are trainable unless you change trainable_layers. And extra_blocks=None here means “use the default”, which is LastLevelMaxPool, so the output always has a "pool" P6 entry and the keys are the strings "0", "1", "2", not the layer names. The returned BackboneWithFPN has an out_channels attribute, which is what torchvision’s own FasterRCNN, RetinaNet and FCOS constructors read.

7.3 The full detector: backbone, FPN neck, YOLO head, decoding

import math
import torch
import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights
from torchvision.models.feature_extraction import create_feature_extractor
from torchvision.ops import FeaturePyramidNetwork, Conv2dNormActivation, batched_nms


def conv_bn_silu(c_in, c_out):
    return Conv2dNormActivation(c_in, c_out, kernel_size=3, activation_layer=nn.SiLU)


class YOLOHead(nn.Module):
    """Anchor-free, decoupled YOLO-style head (YOLOX / YOLOv8 family, simplified).

    One head per pyramid level. Output per level: [B, 4 + 1 + num_classes, H, W]
    channels = (l, t, r, b) distances, objectness, class logits.
    """

    def __init__(self, num_classes=80, in_ch=256, width=256, num_levels=3, prior=0.01):
        super().__init__()
        self.nc = num_classes
        self.cls_convs = nn.ModuleList()
        self.reg_convs = nn.ModuleList()
        self.cls_pred = nn.ModuleList()
        self.box_pred = nn.ModuleList()
        self.obj_pred = nn.ModuleList()
        for _ in range(num_levels):
            self.cls_convs.append(nn.Sequential(conv_bn_silu(in_ch, width), conv_bn_silu(width, width)))
            self.reg_convs.append(nn.Sequential(conv_bn_silu(in_ch, width), conv_bn_silu(width, width)))
            self.cls_pred.append(nn.Conv2d(width, num_classes, 1))
            self.box_pred.append(nn.Conv2d(width, 4, 1))
            self.obj_pred.append(nn.Conv2d(width, 1, 1))
        # start every cell at p = 0.01 so early training is not swamped by easy negatives
        b = -math.log((1 - prior) / prior)
        for m in list(self.cls_pred) + list(self.obj_pred):
            nn.init.constant_(m.bias, b)

    def forward(self, feats):
        outs = []
        for i, f in enumerate(feats):
            c = self.cls_convs[i](f)
            r = self.reg_convs[i](f)
            outs.append(torch.cat([self.box_pred[i](r), self.obj_pred[i](r), self.cls_pred[i](c)], dim=1))
        return outs


class ResNet50FPNYOLO(nn.Module):
    strides = (8, 16, 32)

    def __init__(self, num_classes=80):
        super().__init__()
        self.backbone = create_feature_extractor(
            resnet50(weights=ResNet50_Weights.IMAGENET1K_V2),
            return_nodes={"layer2": "c3", "layer3": "c4", "layer4": "c5"},
        )
        self.neck = FeaturePyramidNetwork([512, 1024, 2048], 256)
        self.head = YOLOHead(num_classes, num_levels=3)

    def forward(self, x):
        c = self.backbone(x)            # c3, c4, c5
        p = self.neck(c)                # p3, p4, p5 (same keys, 256 channels each)
        return self.head(list(p.values()))


@torch.no_grad()
def decode(outs, strides=(8, 16, 32), conf_thr=0.25, iou_thr=0.65, max_det=300):
    boxes, scores = [], []
    for out, s in zip(outs, strides):
        B, C, H, W = out.shape
        out = out.permute(0, 2, 3, 1).reshape(B, H * W, C)              # [B, HW, 5+nc]
        ys, xs = torch.meshgrid(torch.arange(H, device=out.device),
                                torch.arange(W, device=out.device), indexing="ij")
        ctr = torch.stack([(xs + 0.5) * s, (ys + 0.5) * s], -1).reshape(1, H * W, 2).to(out.dtype)
        ltrb = out[..., :4].clamp(-10, 10).exp() * s                    # distances in pixels
        boxes.append(torch.cat([ctr - ltrb[..., :2], ctr + ltrb[..., 2:]], -1))
        scores.append(out[..., 4:5].sigmoid() * out[..., 5:].sigmoid())
    boxes = torch.cat(boxes, 1)                                          # [B, 8400, 4]
    scores = torch.cat(scores, 1)                                        # [B, 8400, nc]

    results = []
    for b in range(boxes.shape[0]):
        conf, cls = scores[b].max(-1)
        keep = conf > conf_thr
        bx, conf, cls = boxes[b][keep], conf[keep], cls[keep]
        k = batched_nms(bx, conf, cls, iou_thr)[:max_det]
        results.append({"boxes": bx[k], "scores": conf[k], "labels": cls[k]})
    return results


if __name__ == "__main__":
    model = ResNet50FPNYOLO(num_classes=80).eval()
    x = torch.randn(2, 3, 640, 640)
    outs = model(x)
    for o in outs:
        print(tuple(o.shape))
    # (2, 85, 80, 80)
    # (2, 85, 40, 40)
    # (2, 85, 20, 20)
    dets = decode(outs)
    print(len(dets), dets[0]["boxes"].shape)   # 2, (N, 4)
    print(sum(p.numel() for p in model.parameters()) / 1e6, "M params")

The shapes follow the diagrams exactly: [2, 512, 80, 80], [2, 1024, 40, 40] and [2, 2048, 20, 20] out of the backbone, three [2, 256, H, W] maps out of the neck, three [2, 85, H, W] maps out of the head, and [2, 8400, 4] boxes with [2, 8400, 80] class scores before NMS. By the parameter arithmetic, the model is about 33 M parameters: 23.5 M backbone, 2.7 M neck (P3–P5) and 7.1 M in the three heads.

To train it you need two more pieces, both standard: a label assignment rule that decides which cells are responsible for each ground-truth box (centre sampling as in FCOS, SimOTA as in YOLOX, or task-aligned assignment as in YOLOv8), and a loss: BCE on objectness and classes, plus an IoU-family loss on the decoded boxes (torchvision.ops.generalized_box_iou_loss or complete_box_iou_loss). Remember the backbone expects ImageNet normalisation (Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])).

8. Practical notes for deployment

• Keep input sides a multiple of 32. The size-based upsampling makes other sizes work, but they waste compute on padding-like edge cells and complicate engine building.
• The neck exports cleanly. Nearest upsampling becomes an ONNX Resize op with mode="nearest", and the adds and convs fuse well in TensorRT. Export at a fixed resolution to get a static engine.
• Choose levels by your object sizes. For small, distant targets you may want \(P_2\), but budget for it: it is the most expensive level in the network, in compute and in bandwidth. A common compromise is a light \(P_2\) head with fewer channels.
• Count your outputs. The number of pre-NMS predictions, not the backbone, often sets the latency floor at large input sizes.

References

• T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie. Feature Pyramid Networks for Object Detection. CVPR 2017.
• S. Liu et al. Path Aggregation Network for Instance Segmentation (PANet). CVPR 2018.
• M. Tan, R. Pang, Q. Le. EfficientDet: Scalable and Efficient Object Detection (BiFPN). CVPR 2020.
• Z. Ge et al. YOLOX: Exceeding YOLO Series in 2021. arXiv 2021.
• T.-Y. Lin et al. Focal Loss for Dense Object Detection (RetinaNet). ICCV 2017.
• torchvision: torchvision.ops.FeaturePyramidNetwork and torchvision.models.detection.backbone_utils.