Perception Stack

FastSAM — SAM's speed problem, solved

FastSAM (CASIA-IVA-Lab, 2023) reproduces SAM's "segment everything" at roughly 50× the speed by recasting it as a familiar, efficient task: class-agnostic instance segmentation with YOLOv8-seg, followed by a prompt-selection stage. It trades a little quality for real-time throughput.

Symbols used on this page are defined in the Nomenclature → deep learning & training.

FastSAM paper (arXiv:2306.12156) 5-notebook deep dive in deeplearning/notebooks/fastSAM/

The YOLOv8-seg pipeline

image (B,3,1024,1024)
  → Backbone (CSP/C2f): P3(/8), P4(/16), P5(/32)
  → Neck (PAN-FPN): multi-scale fusion
  → Segment head:  box(DFL 4·16) · cls(1) · mask-coeff(32) per anchor
  → Proto branch:  k=32 prototype masks at input/4 (256×256)

The clever part is prototype masks: instead of predicting a full mask per instance, the model learns 32 shared prototype masks and, per detection, a 32-vector of coefficients. The instance mask is a linear combination passed through a sigmoid, then cropped to the box:

$$ \text{mask}_i = \sigma\!\Big(\underbrace{\mathbf{c}_i}_{(1\times32)}\;\underbrace{P}_{(32\times HW)}\Big), \qquad P=\text{prototypes} $$

Training losses

YOLOv8-seg is trained with a TaskAlignedAssigner and four losses:

  • CIoU box regression + DFL (distribution focal loss) for precise box edges,
  • BCE for objectness/class,
  • mask BCE on the assembled prototype masks.

FastSAM is trained on just 2% of SA-1B, yet recovers most of SAM's everything-mode quality.

Prompting after the fact

FastSAM first segments everything, then selects masks matching your prompt: a point keeps masks containing it, a box keeps masks with high IoU, and text uses CLIP to score each mask crop against the phrase. The heavy lifting (segmentation) is done once; prompting is cheap post-processing.

Under the hood: it's a detector

Because FastSAM is YOLO-based, it inherits the anchor/NMS machinery of classical detectors — many raw candidate masks are pruned by non-max suppression before prompt selection:

VariantBackboneParamsSize
FastSAM-sYOLOv8s-seg~11.8M23 MB
FastSAM-xYOLOv8x-seg~68M138 MB
SAM vs FastSAM. SAM is a promptable transformer optimized for mask quality and zero-shot flexibility; FastSAM is a real-time detector optimized for throughput. Choose SAM for interactive precision, FastSAM for video-rate "segment everything."

Full notebook deep-dive

The complete, code-free explanations, math, and figures extracted from the model's Jupyter notebooks. The runnable notebooks are bundled with this site — browse the FastSAM notebooks (local copies).

FastSAM 01 - Overview & Architecture

FastSAM (Fast Segment Anything, Zhao et al., CASIA-IVA-Lab, 2023) achieves SAM-style 'segment everything' at a fraction of the cost by decoupling the task into two stages:

  1. All-instance segmentation with a YOLOv8-seg CNN detector (trained on just 2% of SA-1B), producing class-agnostic masks for every object.
  2. Prompt-guided selection: given a point / box / text prompt, pick the matching mask(s) from stage 1.

The result runs ~50x faster than SAM's ViT-H image encoder while reaching comparable zero-shot mask quality, because a CNN at 640-1024 px is dramatically cheaper than a heavy ViT.

FastSAM vs SAM at a glance

SAM (ViT-H)FastSAM (YOLOv8x-seg)
BackboneViT-H (632M params)CSP-Darknet / C2f (~68M)
Paradigmpromptable transformerdetect-then-select CNN
Masksdecoder per promptprototype masks + per-instance coeffs
'Everything' speedslow (dense grid of prompts)single forward pass
Training datafull SA-1B (1.1B masks)2% of SA-1B

FastSAM comes in two sizes: FastSAM-s (YOLOv8s-seg, ~11.8M params, 23MB) and FastSAM-x (YOLOv8x-seg, ~68M params, 138MB).

The YOLOv8-seg pipeline

`` image (B,3,H,W) H=W=1024 for FastSAM (dynamic, /32) | +-----v------+ Backbone (CSP/C2f): P3 (/8), P4 (/16), P5 (/32) | backbone | multi-scale features +-----+------+ | P3,P4,P5 +-----v------+ Neck (PAN-FPN): top-down + bottom-up fusion | neck | +-----+------+ | +-----v----------------------------+ | Segment head (per FPN level): | | - box branch (DFL, 4reg_max) | | - cls branch (1 class: object) | | - mask-coef (nm=32 per anchor)| | Proto branch (from P3): | | - k=32 prototype masks (H/4,W/4)| +-----------------------------------+ | masks = sigmoid( coeffs(N,32) @ protos(32, H/4W/4) ) -> (N, H/4, W/4) -> upsample/crop to boxes -> (N,H,W) ``

The prototype-mask trick (from YOLACT) is the key: instead of a full per-instance mask branch, the head predicts k=32 shared prototype masks once, plus 32 mask coefficients per detected instance. A linear combination + sigmoid yields each instance mask - very cheap.

Load the real FastSAM model and inspect it

Loaded FastSAM-s.pt
task           : segment
num parameters : 11,790,483
num classes    : 1 (class-agnostic: 1 -> object)
top module types: {'Conv2d': 76, 'Conv': 66, 'BatchNorm2d': 66, 'ModuleList': 11, 'Sequential': 10, 'Bottleneck': 10, 'C2f': 8, 'Concat': 4, 'Upsample': 2, 'SegmentationModel': 1}

The prototype-mask assembly, illustrated

We synthesize k=32 prototypes and a set of instance coefficients to show exactly how the final masks are formed - the same math the real head uses.

protos : (32, 160, 160) (k, Hp, Wp)
coeffs : (3, 32) (N_instances, k)
masks  : (3, 160, 160) = sigmoid(coeffs @ protos)
saved 01_prototype_masks.png

Next: NB02 traces the exact tensor shape at every stage of the network on a real image.

FastSAM 02 - Architecture Deep-Dive: Tensor Sizes & Features

We push a real image through FastSAM and record the tensor shape at each stage: preprocessing, the three backbone scales (P3/P4/P5), the neck, and the segment head outputs (boxes, mask coefficients, prototypes). Then we visualize backbone feature maps and the raw prototype masks.

Preprocessing

Ultralytics letterboxes the image to a multiple of the stride (32) - commonly 1024x1024 for FastSAM - normalizes to [0,1], and forms pixel_values (B,3,1024,1024).

raw image : (480, 640, 3) from /mnt/mydata/huggingface/sample_coco.jpg
letterboxed input target: (1, 3, 1024, 1024)
strides -> feature grids:
  P3  stride  8 -> 128x128 grid
  P4  stride 16 -> 64x64 grid
  P5  stride 32 -> 32x32 grid

Hook the real network and record shapes

We register forward hooks on the backbone/neck/head modules and read off the tensor shapes produced for our image. This is the ground truth for FastSAM-s.

layer | type | output shape
   0 | Conv           | (1, 32, 512, 512)
   1 | Conv           | (1, 64, 256, 256)
   2 | C2f            | (1, 64, 256, 256)
   3 | Conv           | (1, 128, 128, 128)
   4 | C2f            | (1, 128, 128, 128)
   5 | Conv           | (1, 256, 64, 64)
   6 | C2f            | (1, 256, 64, 64)
   7 | Conv           | (1, 512, 32, 32)
   8 | C2f            | (1, 512, 32, 32)
   9 | SPPF           | (1, 512, 32, 32)
  10 | Upsample       | (1, 512, 64, 64)
  11 | Concat         | (1, 768, 64, 64)
  12 | C2f            | (1, 256, 64, 64)
  13 | Upsample       | (1, 256, 128, 128)
  14 | Concat         | (1, 384, 128, 128)
  15 | C2f            | (1, 128, 128, 128)
  16 | Conv           | (1, 128, 64, 64)
  17 | Concat         | (1, 384, 64, 64)
  18 | C2f            | (1, 256, 64, 64)
  19 | Conv           | (1, 256, 32, 32)
  20 | Concat         | (1, 768, 32, 32)
  21 | C2f            | (1, 512, 32, 32)
  22 | Segment        | ['tuple', 'dict']

The segment-head output shapes

OutputShape (FastSAM-s @1024)Meaning
protos(1, 32, 256, 256)32 prototype masks at input/4
box (DFL)(1, 64, A)4 sides x reg_max(16) distributions
cls(1, 1, A)single 'object' class logit
mask coeffs(1, 32, A)32 proto coeffs per anchor

A = total anchors summed over the 3 FPN levels (128^2 + 64^2 + 32^2 = 21504 at 1024). After NMS, only the kept N instances survive; each instance mask = sigmoid(coeffs_N x protos) cropped to its box.

Visualize backbone feature maps + prototypes on the real image

detections: 11
saved 02_feature_maps.png

Next: NB03 covers how FastSAM is trained and the exact loss functions.

FastSAM 03 - Training Approach & Loss Functions

FastSAM is trained exactly like a YOLOv8 instance-segmentation model, but on a class-agnostic version of SA-1B (only 2% of it, ~46k images). Every SA-1B mask becomes a single-class ('object') instance with a tight bounding box + polygon mask. The model then learns to detect + segment all objects in one pass.

Inputs and targets

  • Input: letterboxed image (B,3,1024,1024).
  • Targets (per image): a list of instances, each with a box (cx,cy,w,h) normalized, class 0, and a binary mask (stored as a polygon in the YOLO-seg label .txt).
  • Assignment: YOLOv8 uses TaskAlignedAssigner to match anchors to GT by a combined score s = p^alpha * IoU^beta (classification confidence x localization quality), replacing hand-tuned anchor matching.

The four loss terms

YOLOv8-seg total loss is a weighted sum:

$$\mathcal{L} = \lambda_{box}\mathcal{L}_{CIoU} + \lambda_{cls}\mathcal{L}_{BCE} + \lambda_{dfl}\mathcal{L}_{DFL} + \lambda_{seg}\mathcal{L}_{mask}$$

  1. Box - CIoU loss on matched anchors (complete IoU: overlap + center distance + aspect ratio).
  2. Classification - BCE on the objectness/class logit.
  3. DFL (Distribution Focal Loss) - the box regression is a distribution over reg_max=16 bins per side; DFL sharpens it toward the continuous target.
  4. Mask loss - BCE between the assembled instance mask sigmoid(coeffs @ protos) (cropped to the GT box) and the GT mask, averaged over matched instances.

Default gains (ultralytics): box=7.5, cls=0.5, dfl=1.5, seg mask gain within the box term.

CIoU(pred,gt) = 0.7808 -> loss = 0.2192
DFL loss example = 3.7718
mask BCE = 0.1389

Visualize the DFL idea

Each box side is predicted as a softmax distribution over 16 bins; the final coordinate is the distribution's expectation. DFL pushes probability mass onto the two bins straddling the true value.

saved 03_dfl.png

Training recipe (FastSAM defaults)

  • Optimizer: SGD/AdamW, cosine LR, warmup, EMA of weights.
  • Heavy augmentation: mosaic, mixup, HSV jitter, random flip/scale (mosaic disabled for the last ~10 epochs).
  • 100 epochs on the 2% SA-1B subset at imgsz=1024.
  • Because there is 1 class, the model is purely a region/instance proposer - semantic labels come later from the prompt stage.

Next: NB04 runs real inference in everything/box/point/text modes; NB05 fine-tunes on aerial imagery.

FastSAM 04 - Inference (Everything / Box / Point / Text)

FastSAM's ultralytics API runs stage-1 all-instance segmentation once, then the FastSAMPrompt helper selects masks by prompt. We demonstrate all four modes on a real image.

image: (480, 640, 3) source: /mnt/mydata/huggingface/sample_coco.jpg
FastSAM ready; stage-1 instances: 11

Mode 1 - 'Everything' (all instances in one pass)

saved 04_everything.png

Mode 2 - Box prompt

FastSAMPrompt.box_prompt keeps the stage-1 mask with highest IoU to the given box.

saved 04_box_prompt.png

Mode 3 - Point prompt

saved 04_point_prompt.png

Mode 4 - Text prompt (CLIP-guided)

text_prompt embeds the stage-1 mask crops and the text with CLIP and keeps the best-matching mask. Requires the clip package + a CLIP checkpoint; if unavailable we note it and skip gracefully.

saved 04_text_prompt.png; text_prompt_ran = False

Next: NB05 fine-tunes FastSAM on the harshinde/Aerial-Imagery dataset.

FastSAM 05 - Fine-Tuning on Aerial Imagery

We fine-tune FastSAM (a YOLOv8-seg model) on harshinde/Aerial-Imagery, a semantic-segmentation dataset of drone/aerial tiles with per-pixel masks. Because FastSAM trains via the YOLOv8-seg pipeline, we must:

  1. Download the dataset (image + PNG mask pairs) from the HF Hub.
  2. Convert each semantic mask into YOLO-seg polygon labels (class x1 y1 x2 y2 ... normalized).
  3. Write an ultralytics data.yaml and train with model.train(...).

The dataset layout is Tile N/images/.jpg + Tile N/masks/.png. Masks are color/greyscale-coded classes; we treat each connected component of a foreground class as one instance.

Step 1 - Download the dataset from the HF Hub

We snapshot the repo into /mnt/mydata/huggingface/aerial_imagery. Uses HF_TOKEN if present; skips gracefully (schematic synthetic tiles) if offline.

downloaded to /mnt/mydata/huggingface/aerial_imagery
image dirs found: 8
AERIAL_DIR = /mnt/mydata/huggingface/aerial_imagery

Step 2 - Convert semantic masks to YOLO-seg polygon labels

For each mask PNG we threshold to foreground, find connected components (via cv2.findContours), and write one polygon line per component: class_id x1 y1 x2 y2 ... with coordinates normalized to [0,1]. Tiny components are dropped. Images+labels are split train/val and laid out in the ultralytics folder convention.

image/mask pairs: 72
per-split (images, polygons): {'val': (14, 106), 'train': (58, 307)}

Visualize a converted sample (image + polygons)

saved 05_label_sample.png

Step 3 - Write data.yaml and fine-tune

One class (object). We start from FastSAM-s.pt and run a short fine-tune (few epochs, small imgsz) so the notebook completes quickly; bump epochs/imgsz for real training. Ultralytics writes runs under FASTSAM_RUNS_DIR.

path: /mnt/mydata/huggingface/aerial_yolo
train: images/train
val: images/val
names:
  0: object
Ultralytics 8.4.115 🚀 Python-3.10.12 torch-2.3.0+cu121 CUDA:0 (NVIDIA RTX 6000 Ada Generation, 48639MiB)
engine/trainer: agnostic_nms=False, amp=True, angle=1.0, augment=False, auto_augment=randaugment, batch=4, bgr=0.0, box=7.5, cache=False, cfg=None, channels_last=False, classes=None, close_mosaic=10, cls=0.5, cls_pw=0.0, cls_remap=True, compile=False, conf=None, copy_paste=0.0, copy_paste_mode=flip, cos_lr=False, cutmix=0.0, data=/mnt/mydata/huggingface/aerial_yolo/data.yaml, degrees=0.0, deterministic=True, device=0, dfl=1.5, dgrad=0.5, dis=6.0, distill_model=None, dlam=1.0, dlog=1.0, dnn=False, dropout=0.0, dynamic=False, embed=None, end2end=None, epochs=2, erasing=0.4, exist_ok=True, fliplr=0.5, flipud=0.0, format=torchscript, fraction=1.0, freeze=None, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, imgsz=256, iou=0.7, keras=False, kobj=1.0, line_width=None, lr0=0.01, lrf=0.01, mask_ratio=4, max_det=300, mixup=0.0, mode=train, model=/mnt/mydata/huggingface/fastsam/FastSAM-s.pt, momentum=0.937, mosaic=1.0, multi_scale=0.0, name=aerial_ft, nbs=64, nms=False, opset=None, optimize=False, optimizer=auto, overlap_mask=True, patience=100, perspective=0.0, plots=False, pose=12.0, pretrained=True, profile=False, project=/mnt/mydata/huggingface/fastsam_runs, quantize=None, rect=False, resume=False, retina_masks=False, rle=1.0, save=True, save_conf=False, save_crop=False, save_dir=/mnt/mydata/huggingface/fastsam_runs/aerial_ft, save_frames=False, save_json=False, save_period=-1, save_txt=False, scale=0.5, seed=0, shear=0.0, show=False, show_boxes=True, show_conf=True, show_labels=True, simplify=True, single_cls=False, source=None, split=val, stream_buffer=False, task=segment, time=None, tracker=tracktrack.yaml, translate=0.1, val=True, verbose=False, vid_stride=1, visualize=False, warmup_bias_lr=0.1, warmup_epochs=3.0, warmup_momentum=0.8, weight_decay=0.0005, workers=2, workspace=None
Downloading https://ultralytics.com/assets/Arial.ttf to '/tmp/Ultralytics/Arial.ttf': 77% ━━━━━━━━━─── 584.0/755.1KB 5.7MB/s 0.1s<0.0s
Downloading https://ultralytics.com/assets/Arial.ttf to '/tmp/Ultralytics/Arial.ttf': 100% ━━━━━━━━━━━━ 755.1KB 6.0MB/s 0.1s
                   from n params module arguments
  0                  -1  1       928  ultralytics.nn.modules.conv.Conv             [3, 32, 3, 2]
  1                  -1  1     18560  ultralytics.nn.modules.conv.Conv             [32, 64, 3, 2]
  2                  -1  1     29056  ultralytics.nn.modules.block.C2f             [64, 64, 1, True]
  3                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]
  4                  -1  2    197632  ultralytics.nn.modules.block.C2f             [128, 128, 2, True]
  5                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
  6                  -1  2    788480  ultralytics.nn.modules.block.C2f             [256, 256, 2, True]
  7                  -1  1   1180672  ultralytics.nn.modules.conv.Conv             [256, 512, 3, 2]
  8                  -1  1   1838080  ultralytics.nn.modules.block.C2f             [512, 512, 1, True]
  9                  -1  1    656896  ultralytics.nn.modules.block.SPPF            [512, 512, 5]
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 12                  -1  1    591360  ultralytics.nn.modules.block.C2f             [768, 256, 1]
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 15                  -1  1    148224  ultralytics.nn.modules.block.C2f             [384, 128, 1]
 16                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 18                  -1  1    493056  ultralytics.nn.modules.block.C2f             [384, 256, 1]
 19                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 21                  -1  1   1969152  ultralytics.nn.modules.block.C2f             [768, 512, 1]
 22        [15, 18, 21]  1   2770931  ultralytics.nn.modules.head.Segment          [1, 32, 128, 16, None, [128, 256, 512]]
YOLOv8s-seg summary: 152 layers, 11,790,483 parameters, 11,790,467 gradients, 40.2 GFLOPs
Transferred 417/417 items from pretrained weights
Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks...
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 7% ╸─────────── 416.0KB/5.3MB 4.0MB/s 0.1s<1.2s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 21% ━━╸───────── 1.1/5.3MB 7.3MB/s 0.2s<0.6s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 35% ━━━━──────── 1.9/5.3MB 7.5MB/s 0.3s<0.5s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 48% ━━━━━╸────── 2.6/5.3MB 6.9MB/s 0.4s<0.4s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 57% ━━━━━━╸───── 3.1/5.3MB 4.4MB/s 0.5s<0.5s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 72% ━━━━━━━━╸─── 3.8/5.3MB 7.1MB/s 0.6s<0.2s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 85% ━━━━━━━━━━── 4.5/5.3MB 6.7MB/s 0.7s<0.1s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 93% ━━━━━━━━━━━─ 5.0/5.3MB 4.5MB/s 0.8s<0.1s
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26n.pt to 'yolo26n.pt': 100% ━━━━━━━━━━━━ 5.3MB 6.0MB/s 0.9s
AMP: checks passed ✅
train: Fast image access ✅ (ping: 0.0±0.0 ms, read: 3166.4±1040.4 MB/s, size: 150.1 KB)
train: Scanning /mnt/mydata/huggingface/aerial_yolo/labels/train... 58 images, 1 backgrounds, 0 corrupt: 100% ━━━━━━━━━━━━ 58/58 1.6Kit/s 0.0s
train: New cache created: /mnt/mydata/huggingface/aerial_yolo/labels/train.cache
val: Fast image access ✅ (ping: 0.0±0.0 ms, read: 2369.4±1383.3 MB/s, size: 665.1 KB)
val: Scanning /mnt/mydata/huggingface/aerial_yolo/labels/val... 14 images, 0 backgrounds, 0 corrupt: 100% ━━━━━━━━━━━━ 14/14 1.3Kit/s 0.0s
val: New cache created: /mnt/mydata/huggingface/aerial_yolo/labels/val.cache
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically...
optimizer: AdamW(lr=0.002, momentum=0.9) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias(decay=0.0)
Image sizes 256 train, 256 val
Using 2 dataloader workers
Logging results to /mnt/mydata/huggingface/fastsam_runs/aerial_ft
Starting training for 2 epochs...
      Epoch    GPU_mem box_loss seg_loss cls_loss dfl_loss sem_loss  Instances       Size
        1/2     0.574G     0.8909      5.245      1.502      1.168          0         33        256: 0% ──────────── 0/15  1.5s
        1/2     0.592G      1.077      6.167      1.904      1.241          0         33        256: 13% ━╸────────── 2/15 4.4it/s 1.7s<2.9s
        1/2     0.592G      1.162      6.572      1.962      1.301          0         25        256: 26% ━━━───────── 4/15 8.3it/s 1.8s<1.3s
        1/2     0.592G      1.346      7.064      2.011      1.422          0         37        256: 40% ━━━━╸─────── 6/15 10.3it/s 1.9s<0.9s
        1/2     0.592G      1.379      6.798      1.939      1.413          0         61        256: 53% ━━━━━━────── 8/15 12.6it/s 2.0s<0.6s
        1/2     0.592G       1.37      6.916      1.921       1.41          0         41        256: 66% ━━━━━━━━──── 10/15 12.9it/s 2.2s<0.4s
        1/2     0.592G      1.388      6.766      1.914      1.404          0         50        256: 80% ━━━━━━━━━╸── 12/15 14.2it/s 2.3s<0.2s
        1/2     0.592G      1.394      6.907      1.961      1.414          0         11        256: 93% ━━━━━━━━━━━─ 14/15 10.9it/s 2.9s<0.1s
        1/2     0.592G      1.394      6.907      1.961      1.414          0         11        256: 100% ━━━━━━━━━━━━ 15/15 5.2it/s 2.9s
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95)     Mask(P          R      mAP50  mAP50-95): 50% ━━━━━━────── 1/2 1.0it/s 0.3s<1.0s
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95)     Mask(P          R      mAP50  mAP50-95): 100% ━━━━━━━━━━━━ 2/2 3.9it/s 0.5s
                   all         14        106      0.194      0.292      0.138     0.0827      0.189      0.264     0.0896     0.0427
      Epoch    GPU_mem box_loss seg_loss cls_loss dfl_loss sem_loss  Instances       Size
        2/2     0.592G      1.298      6.457      1.882      1.323          0         31        256: 6% ╸─────────── 1/15 2.6it/s 0.1s<5.5s
        2/2     0.592G      1.358      7.375      1.857      1.351          0         34        256: 20% ━━────────── 3/15 6.3it/s 0.3s<1.9s
        2/2     0.592G      1.441      7.283      1.984       1.37          0         31        256: 33% ━━━━──────── 5/15 9.3it/s 0.4s<1.1s
        2/2     0.592G      1.472      7.005      2.047      1.414          0         37        256: 46% ━━━━━╸────── 7/15 10.7it/s 0.5s<0.7s
        2/2     0.592G       1.48      7.152      2.056      1.425          0         44        256: 60% ━━━━━━━───── 9/15 12.2it/s 0.6s<0.5s
        2/2     0.592G      1.473      7.125      2.043      1.426          0         45        256: 73% ━━━━━━━━╸─── 11/15 12.8it/s 0.8s<0.3s
        2/2     0.592G      1.447      7.042      2.038       1.42          0         31        256: 86% ━━━━━━━━━━── 13/15 13.6it/s 0.9s<0.1s
        2/2     0.592G      1.473      6.954       2.03      1.425          0         38        256: 100% ━━━━━━━━━━━━ 15/15 15.5it/s 1.0s
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95)     Mask(P          R      mAP50  mAP50-95): 100% ━━━━━━━━━━━━ 2/2 18.5it/s 0.1s
                   all         14        106      0.234      0.368      0.164        0.1      0.191      0.292     0.0979     0.0474
2 epochs completed in 0.001 hours.
Optimizer stripped from /mnt/mydata/huggingface/fastsam_runs/aerial_ft/weights/last.pt, 23.8MB
Optimizer stripped from /mnt/mydata/huggingface/fastsam_runs/aerial_ft/weights/best.pt, 23.8MB
Validating /mnt/mydata/huggingface/fastsam_runs/aerial_ft/weights/best.pt...
Ultralytics 8.4.115 🚀 Python-3.10.12 torch-2.3.0+cu121 CUDA:0 (NVIDIA RTX 6000 Ada Generation, 48639MiB)
YOLOv8s-seg summary (fused): 86 layers, 11,779,987 parameters, 0 gradients, 39.9 GFLOPs
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95)     Mask(P          R      mAP50  mAP50-95): 100% ━━━━━━━━━━━━ 2/2 26.6it/s 0.1s
                   all         14        106      0.235      0.368      0.166      0.102      0.193      0.292      0.101     0.0484
Speed: 0.1ms preprocess, 1.3ms inference, 0.0ms loss, 1.1ms postprocess per image
training complete; results dir: /mnt/mydata/huggingface/fastsam_runs/aerial_ft

Step 4 - Inference with the fine-tuned model

We run the fine-tuned checkpoint (best.pt) on a held-out val tile and overlay the predicted instance masks - now specialized to the aerial domain.

saved 05_finetuned_inference.png; best = /mnt/mydata/huggingface/fastsam_runs/aerial_ft/weights/best.pt

Notes & next steps

  • This is a short demonstration fine-tune. For a real run: raise FASTSAM_FT_EPOCHS (e.g. 50-100), FASTSAM_FT_IMGSZ (e.g. 1024), and enable augmentation/plots.
  • The mask->polygon conversion treats each connected foreground component as one instance and a single object class. If the aerial masks encode multiple semantic classes (distinct pixel values), map each value to its own class id and expand names in data.yaml.
  • Because FastSAM is class-agnostic by design, fine-tuning adapts the region proposals to the aerial domain (rooftops/roads/vehicles), improving 'everything' segmentation on overhead imagery.