{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GroundingDINO: Swin Transformer Backbone\n",
"\n",
"## Notebook 02 of 14\n",
"\n",
"This notebook covers the **Swin Transformer** visual backbone used in GroundingDINO, including:\n",
"- Architecture details and design choices\n",
"- Tensor shapes at every layer\n",
"- Window attention mechanism\n",
"- Multi-scale feature extraction\n",
"- Live implementation and visualization\n",
"\n",
"---\n",
"\n",
"## Table of Contents\n",
"1. [Why Swin Transformer?](#why-swin)\n",
"2. [Swin Transformer Architecture](#architecture)\n",
"3. [Patch Partition & Linear Embedding](#patch-partition)\n",
"4. [Window-Based Self-Attention (W-MSA)](#window-attention)\n",
"5. [Shifted Window Attention (SW-MSA)](#shifted-window)\n",
"6. [Swin Transformer Block](#swin-block)\n",
"7. [Hierarchical Feature Maps](#hierarchical)\n",
"8. [Tensor Shapes Through the Network](#tensor-shapes)\n",
"9. [Implementation & Visualization](#implementation)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Why Swin Transformer? \n",
"\n",
"GroundingDINO uses **Swin Transformer** as its visual backbone because:\n",
"\n",
"| Property | Benefit for Detection |\n",
"|----------|----------------------|\n",
"| **Hierarchical features** | Multi-scale representations for objects of different sizes |\n",
"| **Local window attention** | Efficient O(n) complexity vs O(n²) for global attention |\n",
"| **Shifted windows** | Cross-window connections without global attention |\n",
"| **High resolution** | Maintains spatial detail needed for precise localization |\n",
"| **Pre-trained on ImageNet** | Strong visual representations |\n",
"\n",
"### Swin Variants Used in GroundingDINO\n",
"\n",
"| Model | Backbone | Params | COCO AP (zero-shot) |\n",
"|-------|----------|--------|--------------------|\n",
"| GroundingDINO-T | Swin-T | 172M | 48.4 |\n",
"| GroundingDINO-B | Swin-B | 253M | 56.7 |\n",
"| GroundingDINO-L | Swin-L | 341M | 57.2 |\n",
"\n",
"### Key Difference from ViT\n",
"```\n",
"ViT: Image → [16×16 patches] → Single-scale features → Transformer\n",
"Swin: Image → [4×4 patches] → Hierarchical features → Transformer\n",
" (4 stages, 4× downsampling each)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.patches as patches\n",
"from einops import rearrange\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"print(f\"PyTorch version: {torch.__version__}\")\n",
"print(f\"CUDA available: {torch.cuda.is_available()}\")\n",
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"print(f\"Using device: {device}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Swin Transformer Architecture \n",
"\n",
"### Overall Structure\n",
"\n",
"For an input image of size $H \\times W \\times 3$:\n",
"\n",
"```\n",
"Input: H×W×3\n",
" ↓\n",
"Patch Partition (4×4): H/4 × W/4 × 48\n",
" ↓\n",
"Linear Embedding: H/4 × W/4 × C (C=96 for Swin-T)\n",
" ↓\n",
"Stage 1: H/4 × W/4 × C (2 Swin blocks)\n",
" ↓ Patch Merging\n",
"Stage 2: H/8 × W/8 × 2C (2 Swin blocks)\n",
" ↓ Patch Merging\n",
"Stage 3: H/16 × W/16 × 4C (6 Swin blocks)\n",
" ↓ Patch Merging\n",
"Stage 4: H/32 × W/32 × 8C (2 Swin blocks)\n",
"```\n",
"\n",
"For Swin-T (C=96) with 800×1333 input:\n",
"- Stage 1: 200×334×96\n",
"- Stage 2: 100×167×192\n",
"- Stage 3: 50×84×384\n",
"- Stage 4: 25×42×768\n",
"\n",
"GroundingDINO uses **Stages 2, 3, 4** as multi-scale features (FPN-like)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize Swin Transformer architecture\n",
"fig, ax = plt.subplots(figsize=(18, 10))\n",
"ax.set_xlim(0, 18)\n",
"ax.set_ylim(0, 10)\n",
"ax.axis('off')\n",
"\n",
"def draw_stage(ax, x, y, w, h, label, sublabel, color, fontsize=9):\n",
" rect = patches.FancyBboxPatch((x, y), w, h, boxstyle='round,pad=0.1',\n",
" facecolor=color, edgecolor='black', linewidth=2)\n",
" ax.add_patch(rect)\n",
" ax.text(x + w/2, y + h/2 + 0.2, label, ha='center', va='center', \n",
" fontsize=fontsize, fontweight='bold')\n",
" ax.text(x + w/2, y + h/2 - 0.3, sublabel, ha='center', va='center', \n",
" fontsize=8, color='gray')\n",
"\n",
"# Input\n",
"draw_stage(ax, 0.2, 4.0, 1.8, 2.0, 'Input\\nImage', '800×1333×3', '#E3F2FD')\n",
"\n",
"# Patch Partition\n",
"draw_stage(ax, 2.5, 4.0, 2.0, 2.0, 'Patch\\nPartition', '200×334×48\\n(4×4 patches)', '#FFF9C4')\n",
"ax.annotate('', xy=(2.5, 5.0), xytext=(2.0, 5.0), arrowprops=dict(arrowstyle='->', lw=2))\n",
"\n",
"# Linear Embedding\n",
"draw_stage(ax, 5.0, 4.0, 2.0, 2.0, 'Linear\\nEmbedding', '200×334×96\\n(C=96)', '#F8BBD0')\n",
"ax.annotate('', xy=(5.0, 5.0), xytext=(4.5, 5.0), arrowprops=dict(arrowstyle='->', lw=2))\n",
"\n",
"# Stage 1\n",
"draw_stage(ax, 7.5, 5.5, 2.0, 3.0, 'Stage 1\\n(2 blocks)', '200×334×96', '#C8E6C9')\n",
"ax.annotate('', xy=(7.5, 7.0), xytext=(7.0, 5.0), arrowprops=dict(arrowstyle='->', lw=2, color='green'))\n",
"\n",
"# Patch Merging 1\n",
"ax.text(8.5, 5.2, 'Patch Merging\\n(2× downsample)', ha='center', fontsize=8, color='red')\n",
"\n",
"# Stage 2\n",
"draw_stage(ax, 7.5, 2.5, 2.0, 2.5, 'Stage 2\\n(2 blocks)', '100×167×192', '#BBDEFB')\n",
"ax.annotate('', xy=(8.5, 5.5), xytext=(8.5, 5.0), arrowprops=dict(arrowstyle='->', lw=2, color='blue'))\n",
"\n",
"# Stage 3\n",
"draw_stage(ax, 10.5, 5.5, 2.0, 3.0, 'Stage 3\\n(6 blocks)', '50×84×384', '#E1BEE7')\n",
"ax.annotate('', xy=(10.5, 7.0), xytext=(9.5, 4.0), arrowprops=dict(arrowstyle='->', lw=2, color='purple'))\n",
"\n",
"# Stage 4\n",
"draw_stage(ax, 10.5, 2.5, 2.0, 2.5, 'Stage 4\\n(2 blocks)', '25×42×768', '#FFCC80')\n",
"ax.annotate('', xy=(11.5, 5.5), xytext=(11.5, 5.0), arrowprops=dict(arrowstyle='->', lw=2, color='orange'))\n",
"\n",
"# Multi-scale output\n",
"draw_stage(ax, 13.5, 3.0, 3.5, 4.0, 'Multi-Scale\\nFeatures\\n(for GroundingDINO)', \n",
" 'C2: 100×167×192\\nC3: 50×84×384\\nC4: 25×42×768', '#E8F5E9', fontsize=10)\n",
"\n",
"ax.annotate('', xy=(13.5, 6.5), xytext=(12.5, 7.0), arrowprops=dict(arrowstyle='->', lw=2, color='green'))\n",
"ax.annotate('', xy=(13.5, 5.0), xytext=(12.5, 5.5), arrowprops=dict(arrowstyle='->', lw=2, color='purple'))\n",
"ax.annotate('', xy=(13.5, 3.5), xytext=(12.5, 3.5), arrowprops=dict(arrowstyle='->', lw=2, color='orange'))\n",
"\n",
"ax.set_title('Swin Transformer Architecture in GroundingDINO\\n(Input: 800×1333, Swin-T, C=96)', \n",
" fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('swin_architecture.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Patch Partition & Linear Embedding \n",
"\n",
"### Patch Partition\n",
"The image is split into **non-overlapping 4×4 patches**:\n",
"\n",
"$$\\text{Input}: H \\times W \\times 3 \\rightarrow \\frac{H}{4} \\times \\frac{W}{4} \\times (4 \\times 4 \\times 3) = \\frac{H}{4} \\times \\frac{W}{4} \\times 48$$\n",
"\n",
"### Linear Embedding\n",
"A linear layer projects the 48-dim patch features to C-dim:\n",
"\n",
"$$\\frac{H}{4} \\times \\frac{W}{4} \\times 48 \\xrightarrow{\\text{Linear}} \\frac{H}{4} \\times \\frac{W}{4} \\times C$$\n",
"\n",
"For Swin-T: C=96, Swin-S/B: C=96, Swin-L: C=192\n",
"\n",
"### Patch Merging (Downsampling)\n",
"Between stages, spatial resolution is halved and channels doubled:\n",
"\n",
"$$\\frac{H}{4} \\times \\frac{W}{4} \\times C \\xrightarrow{\\text{Patch Merge}} \\frac{H}{8} \\times \\frac{W}{8} \\times 2C$$\n",
"\n",
"Implementation: Take 2×2 neighboring patches → concatenate → linear projection"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Implement Patch Partition and visualize\n",
"class PatchPartition(nn.Module):\n",
" \"\"\"Split image into non-overlapping 4x4 patches.\"\"\"\n",
" def __init__(self, patch_size=4):\n",
" super().__init__()\n",
" self.patch_size = patch_size\n",
" \n",
" def forward(self, x):\n",
" # x: B, C, H, W\n",
" B, C, H, W = x.shape\n",
" p = self.patch_size\n",
" # Reshape to patches\n",
" x = x.view(B, C, H//p, p, W//p, p)\n",
" x = x.permute(0, 2, 4, 3, 5, 1) # B, H/p, W/p, p, p, C\n",
" x = x.reshape(B, H//p, W//p, p*p*C) # B, H/p, W/p, p*p*C\n",
" return x\n",
"\n",
"class PatchMerging(nn.Module):\n",
" \"\"\"Merge 2x2 patches to downsample by 2x.\"\"\"\n",
" def __init__(self, dim):\n",
" super().__init__()\n",
" self.norm = nn.LayerNorm(4 * dim)\n",
" self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)\n",
" \n",
" def forward(self, x):\n",
" # x: B, H, W, C\n",
" B, H, W, C = x.shape\n",
" # Take 2x2 neighboring patches\n",
" x0 = x[:, 0::2, 0::2, :] # B, H/2, W/2, C\n",
" x1 = x[:, 1::2, 0::2, :] # B, H/2, W/2, C\n",
" x2 = x[:, 0::2, 1::2, :] # B, H/2, W/2, C\n",
" x3 = x[:, 1::2, 1::2, :] # B, H/2, W/2, C\n",
" x = torch.cat([x0, x1, x2, x3], dim=-1) # B, H/2, W/2, 4C\n",
" x = self.norm(x)\n",
" x = self.reduction(x) # B, H/2, W/2, 2C\n",
" return x\n",
"\n",
"# Test with a sample input\n",
"B, C, H, W = 1, 3, 64, 64\n",
"x = torch.randn(B, C, H, W)\n",
"\n",
"patch_partition = PatchPartition(patch_size=4)\n",
"x_patched = patch_partition(x)\n",
"print(f\"Input shape: {x.shape}\")\n",
"print(f\"After patch partition: {x_patched.shape} (B, H/4, W/4, 4*4*3=48)\")\n",
"\n",
"# Linear embedding\n",
"linear_embed = nn.Linear(48, 96) # C=96 for Swin-T\n",
"x_embedded = linear_embed(x_patched)\n",
"print(f\"After linear embed: {x_embedded.shape} (B, H/4, W/4, C=96)\")\n",
"\n",
"# Patch merging\n",
"patch_merge = PatchMerging(dim=96)\n",
"x_merged = patch_merge(x_embedded)\n",
"print(f\"After patch merging: {x_merged.shape} (B, H/8, W/8, 2C=192)\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize patch partition\n",
"fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n",
"\n",
"# Create a sample image\n",
"np.random.seed(42)\n",
"img = np.random.rand(32, 32, 3)\n",
"# Add some structure\n",
"img[8:16, 8:16] = [0.9, 0.2, 0.2] # red square\n",
"img[16:24, 16:24] = [0.2, 0.9, 0.2] # green square\n",
"\n",
"# Original image\n",
"axes[0].imshow(img)\n",
"axes[0].set_title('Original Image\\n32×32×3', fontsize=12, fontweight='bold')\n",
"axes[0].axis('off')\n",
"\n",
"# Show patch grid\n",
"axes[1].imshow(img)\n",
"patch_size = 4\n",
"for i in range(0, 32, patch_size):\n",
" axes[1].axhline(y=i-0.5, color='yellow', linewidth=1.5)\n",
" axes[1].axvline(x=i-0.5, color='yellow', linewidth=1.5)\n",
"axes[1].set_title(f'Patch Grid (4×4 patches)\\n{32//patch_size}×{32//patch_size} = {(32//patch_size)**2} patches', \n",
" fontsize=12, fontweight='bold')\n",
"axes[1].axis('off')\n",
"\n",
"# Show flattened patches\n",
"patches_img = img.reshape(32//patch_size, patch_size, 32//patch_size, patch_size, 3)\n",
"patches_img = patches_img.transpose(0, 2, 1, 3, 4) # H/p, W/p, p, p, C\n",
"patches_flat = patches_img.reshape(32//patch_size, 32//patch_size, -1) # H/p, W/p, p*p*C\n",
"\n",
"# Show mean color of each patch\n",
"patch_means = patches_flat.reshape(8, 8, 4, 4, 3).mean(axis=(2,3))\n",
"axes[2].imshow(patch_means)\n",
"axes[2].set_title(f'Patch Embeddings (mean color)\\n8×8×48 → 8×8×96 (after linear)', \n",
" fontsize=12, fontweight='bold')\n",
"axes[2].axis('off')\n",
"\n",
"plt.suptitle('Patch Partition Process', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('patch_partition.png', dpi=150, bbox_inches='tight')\n",
"plt.show()\n",
"\n",
"print(f\"\\nPatch partition details:\")\n",
"print(f\" Input: 32×32×3\")\n",
"print(f\" Patch size: 4×4\")\n",
"print(f\" Number of patches: {(32//4)**2} = 64\")\n",
"print(f\" Patch feature dim: 4×4×3 = 48\")\n",
"print(f\" After linear embed: 8×8×96\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Window-Based Self-Attention (W-MSA) \n",
"\n",
"### The Problem with Global Attention\n",
"Standard self-attention has $O(N^2)$ complexity where $N$ is the number of tokens.\n",
"For a 800×1333 image with 4×4 patches: $N = 200 \\times 334 = 66,800$ tokens → $O(4.5 \\times 10^9)$ operations!\n",
"\n",
"### Window Attention Solution\n",
"Partition the feature map into **non-overlapping windows** of size $M \\times M$ (default M=7):\n",
"\n",
"$$\\text{Complexity: } O\\left(\\frac{HW}{M^2} \\cdot M^4 \\cdot d\\right) = O(HW \\cdot M^2 \\cdot d)$$\n",
"\n",
"This is **linear** in image size!\n",
"\n",
"### Window Attention Formula\n",
"\n",
"For each window of $M^2$ tokens:\n",
"$$\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}} + B\\right)V$$\n",
"\n",
"where $B$ is the **relative position bias** — a learnable bias based on relative positions within the window.\n",
"\n",
"### Relative Position Bias\n",
"For a window of size $M \\times M$, relative positions range from $[-(M-1), M-1]$.\n",
"A learnable bias table $\\hat{B} \\in \\mathbb{R}^{(2M-1) \\times (2M-1)}$ is used:\n",
"\n",
"$$B_{ij} = \\hat{B}[\\Delta x_{ij} + M - 1, \\Delta y_{ij} + M - 1]$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class WindowAttention(nn.Module):\n",
" \"\"\"Window-based Multi-head Self-Attention with relative position bias.\"\"\"\n",
" \n",
" def __init__(self, dim, window_size, num_heads):\n",
" super().__init__()\n",
" self.dim = dim\n",
" self.window_size = window_size # M\n",
" self.num_heads = num_heads\n",
" self.head_dim = dim // num_heads\n",
" self.scale = self.head_dim ** -0.5\n",
" \n",
" # QKV projection\n",
" self.qkv = nn.Linear(dim, dim * 3, bias=True)\n",
" self.proj = nn.Linear(dim, dim)\n",
" \n",
" # Relative position bias table\n",
" # (2*M-1) * (2*M-1) entries\n",
" self.relative_position_bias_table = nn.Parameter(\n",
" torch.zeros((2 * window_size - 1) * (2 * window_size - 1), num_heads)\n",
" )\n",
" nn.init.trunc_normal_(self.relative_position_bias_table, std=0.02)\n",
" \n",
" # Compute relative position index\n",
" coords_h = torch.arange(window_size)\n",
" coords_w = torch.arange(window_size)\n",
" coords = torch.stack(torch.meshgrid(coords_h, coords_w, indexing='ij')) # 2, M, M\n",
" coords_flatten = torch.flatten(coords, 1) # 2, M*M\n",
" \n",
" relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, M*M, M*M\n",
" relative_coords = relative_coords.permute(1, 2, 0).contiguous() # M*M, M*M, 2\n",
" relative_coords[:, :, 0] += window_size - 1 # shift to start from 0\n",
" relative_coords[:, :, 1] += window_size - 1\n",
" relative_coords[:, :, 0] *= 2 * window_size - 1\n",
" relative_position_index = relative_coords.sum(-1) # M*M, M*M\n",
" self.register_buffer('relative_position_index', relative_position_index)\n",
" \n",
" def forward(self, x, mask=None):\n",
" # x: num_windows*B, M*M, C\n",
" B_, N, C = x.shape\n",
" \n",
" # QKV\n",
" qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, self.head_dim)\n",
" qkv = qkv.permute(2, 0, 3, 1, 4) # 3, B_, num_heads, N, head_dim\n",
" q, k, v = qkv.unbind(0) # each: B_, num_heads, N, head_dim\n",
" \n",
" # Attention\n",
" attn = (q @ k.transpose(-2, -1)) * self.scale # B_, num_heads, N, N\n",
" \n",
" # Add relative position bias\n",
" relative_position_bias = self.relative_position_bias_table[\n",
" self.relative_position_index.view(-1)\n",
" ].view(self.window_size**2, self.window_size**2, -1) # M*M, M*M, num_heads\n",
" relative_position_bias = relative_position_bias.permute(2, 0, 1).unsqueeze(0) # 1, num_heads, M*M, M*M\n",
" attn = attn + relative_position_bias\n",
" \n",
" if mask is not None:\n",
" attn = attn + mask.unsqueeze(1)\n",
" \n",
" attn = torch.softmax(attn, dim=-1)\n",
" \n",
" # Output\n",
" x = (attn @ v).transpose(1, 2).reshape(B_, N, C)\n",
" x = self.proj(x)\n",
" \n",
" return x, attn\n",
"\n",
"# Test\n",
"M = 7 # window size\n",
"C = 96 # channels\n",
"num_heads = 3\n",
"num_windows = 4\n",
"\n",
"window_attn = WindowAttention(dim=C, window_size=M, num_heads=num_heads)\n",
"x_window = torch.randn(num_windows, M*M, C) # num_windows, M*M, C\n",
"out, attn_weights = window_attn(x_window)\n",
"\n",
"print(f\"Window Attention:\")\n",
"print(f\" Input: {x_window.shape} (num_windows, M*M, C)\")\n",
"print(f\" Output: {out.shape} (num_windows, M*M, C)\")\n",
"print(f\" Attn: {attn_weights.shape} (num_windows, num_heads, M*M, M*M)\")\n",
"print(f\"\\nRelative position bias table: {window_attn.relative_position_bias_table.shape}\")\n",
"print(f\" Size: (2M-1)*(2M-1) × num_heads = {(2*M-1)**2} × {num_heads}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize window partitioning\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"H, W = 28, 28 # feature map size (after patch partition of 112x112 image)\n",
"M = 7 # window size\n",
"\n",
"# Feature map\n",
"feature_map = np.random.rand(H, W)\n",
"\n",
"# Window partition visualization\n",
"ax1 = axes[0]\n",
"im = ax1.imshow(feature_map, cmap='viridis', aspect='equal')\n",
"for i in range(0, H, M):\n",
" ax1.axhline(y=i-0.5, color='red', linewidth=2)\n",
" ax1.axvline(x=i-0.5, color='red', linewidth=2)\n",
"ax1.set_title(f'Feature Map with Window Partition\\n{H}×{W}, Window={M}×{M}', fontsize=11, fontweight='bold')\n",
"num_windows = (H//M) * (W//M)\n",
"ax1.text(W//2, H//2, f'{num_windows} windows\\n({H//M}×{W//M})', \n",
" ha='center', va='center', fontsize=12, color='white', fontweight='bold',\n",
" bbox=dict(boxstyle='round', facecolor='black', alpha=0.7))\n",
"plt.colorbar(im, ax=ax1)\n",
"\n",
"# Attention within one window\n",
"ax2 = axes[1]\n",
"# Simulate attention weights for one window\n",
"np.random.seed(42)\n",
"attn_sim = np.random.rand(M*M, M*M)\n",
"attn_sim = np.exp(attn_sim) / np.exp(attn_sim).sum(axis=-1, keepdims=True)\n",
"im2 = ax2.imshow(attn_sim, cmap='hot', aspect='equal')\n",
"ax2.set_title(f'Attention Map (1 window, 1 head)\\n{M*M}×{M*M} = 49×49', fontsize=11, fontweight='bold')\n",
"ax2.set_xlabel('Key position (within window)')\n",
"ax2.set_ylabel('Query position (within window)')\n",
"plt.colorbar(im2, ax=ax2)\n",
"\n",
"# Relative position bias\n",
"ax3 = axes[2]\n",
"# Create relative position bias visualization\n",
"bias_table = np.random.randn(2*M-1, 2*M-1) * 0.1\n",
"im3 = ax3.imshow(bias_table, cmap='RdBu', aspect='equal')\n",
"ax3.set_title(f'Relative Position Bias Table\\n(2M-1)×(2M-1) = {2*M-1}×{2*M-1}', fontsize=11, fontweight='bold')\n",
"ax3.set_xlabel('Relative column offset')\n",
"ax3.set_ylabel('Relative row offset')\n",
"# Mark center (0,0 relative position)\n",
"ax3.plot(M-1, M-1, 'g*', markersize=15, label='(0,0) center')\n",
"ax3.legend()\n",
"plt.colorbar(im3, ax=ax3)\n",
"\n",
"plt.suptitle('Window-Based Self-Attention Visualization', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('window_attention.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Shifted Window Attention (SW-MSA) \n",
"\n",
"### The Problem with Regular Windows\n",
"Regular window attention has **no cross-window connections** — tokens in different windows never interact!\n",
"\n",
"### Shifted Window Solution\n",
"Alternate between:\n",
"1. **W-MSA**: Regular window partition\n",
"2. **SW-MSA**: Shifted window partition (shift by $\\lfloor M/2 \\rfloor$ in both dimensions)\n",
"\n",
"```\n",
"Layer 1 (W-MSA): Layer 2 (SW-MSA):\n",
"┌──┬──┬──┐ ┌─┬──┬──┬─┐\n",
"│ │ │ │ │ │ │ │ │\n",
"├──┼──┼──┤ → ├─┼──┼──┼─┤\n",
"│ │ │ │ │ │ │ │ │\n",
"├──┼──┼──┤ ├─┼──┼──┼─┤\n",
"│ │ │ │ │ │ │ │ │\n",
"└──┴──┴──┘ └─┴──┴──┴─┘\n",
"```\n",
"\n",
"### Cyclic Shift for Efficiency\n",
"Instead of creating irregular windows at boundaries, use **cyclic shift** + **attention masking**:\n",
"1. Cyclically shift the feature map by $(-\\lfloor M/2 \\rfloor, -\\lfloor M/2 \\rfloor)$\n",
"2. Apply regular window partition\n",
"3. Use attention mask to prevent cross-region attention\n",
"4. Reverse cyclic shift"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize shifted window attention\n",
"fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n",
"\n",
"H, W = 14, 14\n",
"M = 7\n",
"shift = M // 2 # = 3\n",
"\n",
"# Create a feature map with distinct regions\n",
"feature_map = np.zeros((H, W))\n",
"for i in range(H):\n",
" for j in range(W):\n",
" feature_map[i, j] = (i // M) * 2 + (j // M) # 0, 1, 2, 3 for 4 windows\n",
"\n",
"# Row 1: Regular windows\n",
"ax = axes[0, 0]\n",
"cmap = plt.cm.get_cmap('Set3', 4)\n",
"ax.imshow(feature_map, cmap=cmap, vmin=0, vmax=3, aspect='equal')\n",
"for i in range(0, H+1, M):\n",
" ax.axhline(y=i-0.5, color='black', linewidth=2)\n",
" ax.axvline(x=i-0.5, color='black', linewidth=2)\n",
"ax.set_title('W-MSA: Regular Windows\\n(4 windows, no cross-window)', fontsize=11, fontweight='bold')\n",
"for wi in range(2):\n",
" for wj in range(2):\n",
" ax.text(wj*M + M//2, wi*M + M//2, f'W{wi*2+wj}', \n",
" ha='center', va='center', fontsize=14, fontweight='bold')\n",
"\n",
"# Shifted feature map\n",
"shifted_map = np.roll(np.roll(feature_map, -shift, axis=0), -shift, axis=1)\n",
"ax = axes[0, 1]\n",
"ax.imshow(shifted_map, cmap=cmap, vmin=0, vmax=3, aspect='equal')\n",
"for i in range(0, H+1, M):\n",
" ax.axhline(y=i-0.5, color='black', linewidth=2)\n",
" ax.axvline(x=i-0.5, color='black', linewidth=2)\n",
"ax.set_title(f'SW-MSA: After Cyclic Shift by ({-shift},{-shift})\\n(Cross-window connections!)', fontsize=11, fontweight='bold')\n",
"\n",
"# Attention mask for shifted windows\n",
"ax = axes[0, 2]\n",
"# Create attention mask for one shifted window (top-left after shift)\n",
"mask = np.zeros((M*M, M*M))\n",
"# Tokens from different original regions should not attend to each other\n",
"# Simplified visualization\n",
"for i in range(M*M):\n",
" row_i, col_i = i // M, i % M\n",
" region_i = (row_i >= shift) * 2 + (col_i >= shift)\n",
" for j in range(M*M):\n",
" row_j, col_j = j // M, j % M\n",
" region_j = (row_j >= shift) * 2 + (col_j >= shift)\n",
" if region_i != region_j:\n",
" mask[i, j] = -100 # mask out\n",
"\n",
"ax.imshow(mask, cmap='RdYlGn', aspect='equal', vmin=-100, vmax=0)\n",
"ax.set_title('Attention Mask for Shifted Window\\n(Green=attend, Red=masked)', fontsize=11, fontweight='bold')\n",
"ax.set_xlabel('Key position')\n",
"ax.set_ylabel('Query position')\n",
"\n",
"# Row 2: Swin block structure\n",
"ax = axes[1, 0]\n",
"ax.axis('off')\n",
"ax.set_xlim(0, 10)\n",
"ax.set_ylim(0, 10)\n",
"\n",
"# Draw Swin block\n",
"blocks = [\n",
" (3, 8.5, 4, 0.8, 'Input x', '#E3F2FD'),\n",
" (3, 7.2, 4, 0.8, 'LayerNorm', '#FFF9C4'),\n",
" (3, 5.9, 4, 0.8, 'W-MSA (or SW-MSA)', '#C8E6C9'),\n",
" (3, 4.6, 4, 0.8, 'Residual + x', '#F8BBD0'),\n",
" (3, 3.3, 4, 0.8, 'LayerNorm', '#FFF9C4'),\n",
" (3, 2.0, 4, 0.8, 'MLP (FFN)', '#E1BEE7'),\n",
" (3, 0.7, 4, 0.8, 'Residual + Output', '#F8BBD0'),\n",
"]\n",
"for x, y, w, h, label, color in blocks:\n",
" rect = patches.FancyBboxPatch((x, y), w, h, boxstyle='round,pad=0.05',\n",
" facecolor=color, edgecolor='black', linewidth=1.5)\n",
" ax.add_patch(rect)\n",
" ax.text(x + w/2, y + h/2, label, ha='center', va='center', fontsize=10, fontweight='bold')\n",
"\n",
"for i in range(len(blocks)-1):\n",
" ax.annotate('', xy=(5, blocks[i+1][1] + blocks[i+1][3]), \n",
" xytext=(5, blocks[i][1]),\n",
" arrowprops=dict(arrowstyle='->', lw=1.5))\n",
"\n",
"ax.set_title('Swin Transformer Block Structure', fontsize=11, fontweight='bold')\n",
"\n",
"# Complexity comparison\n",
"ax = axes[1, 1]\n",
"H_vals = np.array([56, 112, 224, 448, 896])\n",
"W_vals = H_vals\n",
"M_val = 7\n",
"d = 96\n",
"\n",
"global_complexity = H_vals * W_vals * (H_vals * W_vals) * d / 1e9\n",
"window_complexity = H_vals * W_vals * M_val**2 * d / 1e9\n",
"\n",
"ax.semilogy(H_vals, global_complexity, 'r-o', linewidth=2, label='Global Attention O(N²)', markersize=8)\n",
"ax.semilogy(H_vals, window_complexity, 'b-o', linewidth=2, label='Window Attention O(N)', markersize=8)\n",
"ax.set_xlabel('Feature Map Size H=W', fontsize=11)\n",
"ax.set_ylabel('FLOPs (Giga)', fontsize=11)\n",
"ax.set_title('Complexity: Global vs Window Attention', fontsize=11, fontweight='bold')\n",
"ax.legend()\n",
"ax.grid(True, alpha=0.3)\n",
"\n",
"# Swin-T configuration\n",
"ax = axes[1, 2]\n",
"ax.axis('off')\n",
"config_data = [\n",
" ['Parameter', 'Swin-T', 'Swin-S', 'Swin-B', 'Swin-L'],\n",
" ['C (channels)', '96', '96', '128', '192'],\n",
" ['Stage 1 blocks', '2', '2', '2', '2'],\n",
" ['Stage 2 blocks', '2', '2', '2', '2'],\n",
" ['Stage 3 blocks', '6', '18', '18', '18'],\n",
" ['Stage 4 blocks', '2', '2', '2', '2'],\n",
" ['Heads (S1-S4)', '3,6,12,24', '3,6,12,24', '4,8,16,32', '6,12,24,48'],\n",
" ['Params (M)', '28', '50', '88', '197'],\n",
"]\n",
"table = ax.table(cellText=config_data[1:], colLabels=config_data[0], \n",
" cellLoc='center', loc='center')\n",
"table.auto_set_font_size(False)\n",
"table.set_fontsize(9)\n",
"table.scale(1, 2)\n",
"for j in range(5):\n",
" table[0, j].set_facecolor('#1565C0')\n",
" table[0, j].set_text_props(color='white', fontweight='bold')\n",
"ax.set_title('Swin Transformer Variants', fontsize=11, fontweight='bold')\n",
"\n",
"plt.suptitle('Shifted Window Attention & Swin Block Details', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('shifted_window_attention.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Swin Transformer Block \n",
"\n",
"Each Swin Transformer block consists of:\n",
"\n",
"$$\\hat{z}^l = \\text{W-MSA}(\\text{LN}(z^{l-1})) + z^{l-1}$$\n",
"$$z^l = \\text{MLP}(\\text{LN}(\\hat{z}^l)) + \\hat{z}^l$$\n",
"\n",
"$$\\hat{z}^{l+1} = \\text{SW-MSA}(\\text{LN}(z^l)) + z^l$$\n",
"$$z^{l+1} = \\text{MLP}(\\text{LN}(\\hat{z}^{l+1})) + \\hat{z}^{l+1}$$\n",
"\n",
"### MLP Structure\n",
"The MLP has 2 layers with GELU activation:\n",
"$$\\text{MLP}(x) = \\text{Linear}(\\text{GELU}(\\text{Linear}(x)))$$\n",
"\n",
"Hidden dimension = 4 × C (expansion ratio = 4)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def window_partition(x, window_size):\n",
" \"\"\"Partition feature map into windows.\"\"\"\n",
" B, H, W, C = x.shape\n",
" x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)\n",
" windows = x.permute(0, 1, 3, 2, 4, 5).contiguous()\n",
" windows = windows.view(-1, window_size, window_size, C)\n",
" return windows # num_windows*B, M, M, C\n",
"\n",
"def window_reverse(windows, window_size, H, W):\n",
" \"\"\"Reverse window partition.\"\"\"\n",
" B = int(windows.shape[0] / (H * W / window_size / window_size))\n",
" x = windows.view(B, H // window_size, W // window_size, window_size, window_size, -1)\n",
" x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, -1)\n",
" return x\n",
"\n",
"class SwinTransformerBlock(nn.Module):\n",
" \"\"\"Swin Transformer Block with W-MSA or SW-MSA.\"\"\"\n",
" \n",
" def __init__(self, dim, num_heads, window_size=7, shift_size=0, mlp_ratio=4.0):\n",
" super().__init__()\n",
" self.dim = dim\n",
" self.num_heads = num_heads\n",
" self.window_size = window_size\n",
" self.shift_size = shift_size\n",
" \n",
" self.norm1 = nn.LayerNorm(dim)\n",
" self.attn = WindowAttention(dim, window_size, num_heads)\n",
" self.norm2 = nn.LayerNorm(dim)\n",
" \n",
" mlp_hidden = int(dim * mlp_ratio)\n",
" self.mlp = nn.Sequential(\n",
" nn.Linear(dim, mlp_hidden),\n",
" nn.GELU(),\n",
" nn.Linear(mlp_hidden, dim)\n",
" )\n",
" \n",
" def forward(self, x):\n",
" # x: B, H, W, C\n",
" B, H, W, C = x.shape\n",
" shortcut = x\n",
" \n",
" # LayerNorm\n",
" x = self.norm1(x)\n",
" \n",
" # Cyclic shift\n",
" if self.shift_size > 0:\n",
" x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))\n",
" \n",
" # Window partition\n",
" x_windows = window_partition(x, self.window_size) # num_windows*B, M, M, C\n",
" x_windows = x_windows.view(-1, self.window_size**2, C) # num_windows*B, M*M, C\n",
" \n",
" # Window attention\n",
" attn_windows, attn_weights = self.attn(x_windows)\n",
" \n",
" # Reverse window partition\n",
" attn_windows = attn_windows.view(-1, self.window_size, self.window_size, C)\n",
" x = window_reverse(attn_windows, self.window_size, H, W) # B, H, W, C\n",
" \n",
" # Reverse cyclic shift\n",
" if self.shift_size > 0:\n",
" x = torch.roll(x, shifts=(self.shift_size, self.shift_size), dims=(1, 2))\n",
" \n",
" # Residual + MLP\n",
" x = shortcut + x\n",
" x = x + self.mlp(self.norm2(x))\n",
" \n",
" return x, attn_weights\n",
"\n",
"# Test Swin block\n",
"B, H, W, C = 1, 28, 28, 96\n",
"num_heads = 3\n",
"window_size = 7\n",
"\n",
"# W-MSA block\n",
"block_wmsa = SwinTransformerBlock(C, num_heads, window_size, shift_size=0)\n",
"# SW-MSA block\n",
"block_swmsa = SwinTransformerBlock(C, num_heads, window_size, shift_size=window_size//2)\n",
"\n",
"x = torch.randn(B, H, W, C)\n",
"out_wmsa, attn_wmsa = block_wmsa(x)\n",
"out_swmsa, attn_swmsa = block_swmsa(x)\n",
"\n",
"print(\"Swin Transformer Block:\")\n",
"print(f\" Input: {x.shape} (B, H, W, C)\")\n",
"print(f\" W-MSA output: {out_wmsa.shape}\")\n",
"print(f\" SW-MSA output: {out_swmsa.shape}\")\n",
"print(f\" Attention weights: {attn_wmsa.shape} (num_windows*B, num_heads, M*M, M*M)\")\n",
"print(f\"\\nMLP hidden dim: {int(C * 4)} (4× expansion)\")\n",
"print(f\"Number of windows: {(H//window_size) * (W//window_size)} = {H//window_size}×{W//window_size}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Hierarchical Feature Maps \n",
"\n",
"GroundingDINO uses **multi-scale features** from Swin Transformer stages 2, 3, 4.\n",
"These are fed into the Feature Pyramid Network (FPN) or directly to the Feature Enhancer.\n",
"\n",
"### Feature Scales\n",
"\n",
"For input 800×1333 (typical detection input):\n",
"\n",
"| Stage | Stride | Feature Size | Channels | Receptive Field |\n",
"|-------|--------|-------------|----------|----------------|\n",
"| Stage 2 | 8 | 100×167 | 192 | Small objects |\n",
"| Stage 3 | 16 | 50×84 | 384 | Medium objects |\n",
"| Stage 4 | 32 | 25×42 | 768 | Large objects |\n",
"\n",
"### Projection to Common Dimension\n",
"All features are projected to 256-dim for the Feature Enhancer:\n",
"$$\\text{Stage}_i \\xrightarrow{\\text{Conv1×1}} 256\\text{-dim}$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Simulate multi-scale feature extraction\n",
"class MultiScaleFeatureExtractor(nn.Module):\n",
" \"\"\"Simplified multi-scale feature extractor mimicking Swin output.\"\"\"\n",
" \n",
" def __init__(self, C=96):\n",
" super().__init__()\n",
" # Project each scale to 256-dim\n",
" self.proj2 = nn.Conv2d(2*C, 256, 1) # Stage 2: 192 → 256\n",
" self.proj3 = nn.Conv2d(4*C, 256, 1) # Stage 3: 384 → 256\n",
" self.proj4 = nn.Conv2d(8*C, 256, 1) # Stage 4: 768 → 256\n",
" \n",
" def forward(self, c2, c3, c4):\n",
" # c2: B, 192, H/8, W/8\n",
" # c3: B, 384, H/16, W/16\n",
" # c4: B, 768, H/32, W/32\n",
" p2 = self.proj2(c2) # B, 256, H/8, W/8\n",
" p3 = self.proj3(c3) # B, 256, H/16, W/16\n",
" p4 = self.proj4(c4) # B, 256, H/32, W/32\n",
" return p2, p3, p4\n",
"\n",
"# Simulate Swin outputs for 800×1333 input\n",
"B = 1\n",
"H, W = 800, 1333\n",
"C = 96\n",
"\n",
"# Simulated Swin stage outputs\n",
"c2 = torch.randn(B, 2*C, H//8, W//8) # Stage 2\n",
"c3 = torch.randn(B, 4*C, H//16, W//16) # Stage 3\n",
"c4 = torch.randn(B, 8*C, H//32, W//32) # Stage 4\n",
"\n",
"extractor = MultiScaleFeatureExtractor(C)\n",
"p2, p3, p4 = extractor(c2, c3, c4)\n",
"\n",
"print(\"Multi-Scale Feature Extraction (Swin-T, 800×1333 input):\")\n",
"print(f\"\\nSwin Stage Outputs:\")\n",
"print(f\" Stage 2 (stride 8): {c2.shape} → {c2.shape[2]}×{c2.shape[3]} spatial\")\n",
"print(f\" Stage 3 (stride 16): {c3.shape} → {c3.shape[2]}×{c3.shape[3]} spatial\")\n",
"print(f\" Stage 4 (stride 32): {c4.shape} → {c4.shape[2]}×{c4.shape[3]} spatial\")\n",
"\n",
"print(f\"\\nAfter Projection to 256-dim:\")\n",
"print(f\" P2: {p2.shape}\")\n",
"print(f\" P3: {p3.shape}\")\n",
"print(f\" P4: {p4.shape}\")\n",
"\n",
"# Total number of visual tokens\n",
"total_tokens = p2.shape[2]*p2.shape[3] + p3.shape[2]*p3.shape[3] + p4.shape[2]*p4.shape[3]\n",
"print(f\"\\nTotal visual tokens: {p2.shape[2]*p2.shape[3]} + {p3.shape[2]*p3.shape[3]} + {p4.shape[2]*p4.shape[3]} = {total_tokens}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Tensor Shapes Through the Network \n",
"\n",
"Complete tensor shape tracking for Swin-T with 800×1333 input:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"# Complete tensor shape table\n",
"shapes = [\n",
" ('Input Image', '1 × 3 × 800 × 1333', '3.2M', 'RGB image'),\n",
" ('Patch Partition (4×4)', '1 × 200 × 334 × 48', '3.2M', '4×4×3=48 per patch'),\n",
" ('Linear Embedding', '1 × 200 × 334 × 96', '6.4M', 'C=96 for Swin-T'),\n",
" ('Stage 1 Block 1 (W-MSA)', '1 × 200 × 334 × 96', '6.4M', 'Window size=7'),\n",
" ('Stage 1 Block 2 (SW-MSA)', '1 × 200 × 334 × 96', '6.4M', 'Shift=3'),\n",
" ('Patch Merging 1', '1 × 100 × 167 × 192', '3.2M', '2× downsample, 2× channels'),\n",
" ('Stage 2 Block 1 (W-MSA)', '1 × 100 × 167 × 192', '3.2M', 'num_heads=6'),\n",
" ('Stage 2 Block 2 (SW-MSA)', '1 × 100 × 167 × 192', '3.2M', '→ C2 output'),\n",
" ('Patch Merging 2', '1 × 50 × 84 × 384', '1.6M', '2× downsample'),\n",
" ('Stage 3 Blocks 1-6', '1 × 50 × 84 × 384', '1.6M', 'num_heads=12, 6 blocks'),\n",
" ('Patch Merging 3', '1 × 25 × 42 × 768', '0.8M', '→ C3 output'),\n",
" ('Stage 4 Block 1 (W-MSA)', '1 × 25 × 42 × 768', '0.8M', 'num_heads=24'),\n",
" ('Stage 4 Block 2 (SW-MSA)', '1 × 25 × 42 × 768', '0.8M', '→ C4 output'),\n",
" ('Proj C2 → 256', '1 × 256 × 100 × 167', '4.3M', 'Conv1×1'),\n",
" ('Proj C3 → 256', '1 × 256 × 50 × 84', '1.1M', 'Conv1×1'),\n",
" ('Proj C4 → 256', '1 × 256 × 25 × 42', '0.27M', 'Conv1×1'),\n",
" ('Flatten & Concat', '1 × 22,000 × 256', '5.6M', 'All scales flattened'),\n",
"]\n",
"\n",
"df = pd.DataFrame(shapes, columns=['Layer', 'Shape', 'Elements', 'Notes'])\n",
"\n",
"fig, ax = plt.subplots(figsize=(18, 10))\n",
"ax.axis('off')\n",
"table = ax.table(cellText=df.values, colLabels=df.columns, cellLoc='left', loc='center')\n",
"table.auto_set_font_size(False)\n",
"table.set_fontsize(9)\n",
"table.scale(1, 2.2)\n",
"\n",
"# Style\n",
"for j in range(4):\n",
" table[0, j].set_facecolor('#1565C0')\n",
" table[0, j].set_text_props(color='white', fontweight='bold')\n",
"\n",
"# Color code by stage\n",
"stage_colors = {\n",
" 0: '#E3F2FD', 1: '#E3F2FD', 2: '#E3F2FD',\n",
" 3: '#C8E6C9', 4: '#C8E6C9', 5: '#FFF9C4',\n",
" 6: '#BBDEFB', 7: '#BBDEFB', 8: '#FFF9C4',\n",
" 9: '#E1BEE7', 10: '#FFF9C4',\n",
" 11: '#FFCC80', 12: '#FFCC80',\n",
" 13: '#F8BBD0', 14: '#F8BBD0', 15: '#F8BBD0',\n",
" 16: '#E8F5E9',\n",
"}\n",
"for i in range(1, len(shapes)+1):\n",
" color = stage_colors.get(i-1, 'white')\n",
" for j in range(4):\n",
" table[i, j].set_facecolor(color)\n",
"\n",
"plt.title('Swin-T Tensor Shapes: 800×1333 Input', fontsize=14, fontweight='bold', pad=20)\n",
"plt.tight_layout()\n",
"plt.savefig('swin_tensor_shapes.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Implementation & Visualization \n",
"\n",
"Let's use the actual Swin Transformer from `timm` to visualize real feature maps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Install timm if needed\n",
"try:\n",
" import timm\n",
" print(f\"timm version: {timm.__version__}\")\n",
"except ImportError:\n",
" import subprocess\n",
" subprocess.run(['pip', 'install', 'timm', '-q'])\n",
" import timm\n",
"\n",
"from PIL import Image\n",
"import torchvision.transforms as T\n",
"import requests\n",
"from io import BytesIO"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load Swin-T from timm\n",
"swin_model = timm.create_model('swin_tiny_patch4_window7_224', pretrained=True, features_only=True)\n",
"swin_model.eval()\n",
"print(\"Swin-T loaded successfully!\")\n",
"print(f\"Output feature channels: {swin_model.feature_info.channels()}\")\n",
"print(f\"Output feature strides: {swin_model.feature_info.reduction()}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Download a sample image\n",
"import urllib.request\n",
"import os\n",
"\n",
"img_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n",
"img_path = \"/mnt/mydata/huggingface/sample_coco.jpg\"\n",
"os.makedirs(\"/mnt/mydata/huggingface\", exist_ok=True)\n",
"\n",
"try:\n",
" urllib.request.urlretrieve(img_url, img_path)\n",
" img = Image.open(img_path).convert('RGB')\n",
" print(f\"Downloaded image: {img.size}\")\n",
"except Exception as e:\n",
" print(f\"Could not download: {e}\")\n",
" # Create synthetic image\n",
" img = Image.fromarray(np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8))\n",
" print(\"Using synthetic image\")\n",
"\n",
"# Preprocess\n",
"transform = T.Compose([\n",
" T.Resize((224, 224)),\n",
" T.ToTensor(),\n",
" T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
"])\n",
"\n",
"img_tensor = transform(img).unsqueeze(0) # 1, 3, 224, 224\n",
"print(f\"Input tensor: {img_tensor.shape}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Extract multi-scale features\n",
"with torch.no_grad():\n",
" features = swin_model(img_tensor)\n",
"\n",
"print(\"Swin-T Multi-Scale Features:\")\n",
"for i, feat in enumerate(features):\n",
" print(f\" Stage {i+1}: {feat.shape} (B, C, H, W)\")\n",
"\n",
"# Visualize feature maps\n",
"fig, axes = plt.subplots(2, 4, figsize=(20, 10))\n",
"\n",
"# Original image\n",
"img_display = np.array(img.resize((224, 224)))\n",
"axes[0, 0].imshow(img_display)\n",
"axes[0, 0].set_title('Input Image\\n224×224×3', fontsize=11, fontweight='bold')\n",
"axes[0, 0].axis('off')\n",
"\n",
"# Feature maps from each stage\n",
"stage_names = ['Stage 1\\n56×56×96', 'Stage 2\\n28×28×192', 'Stage 3\\n14×14×384', 'Stage 4\\n7×7×768']\n",
"for i, (feat, name) in enumerate(zip(features, stage_names)):\n",
" # Show mean across channels\n",
" feat_np = feat[0].mean(0).detach().numpy() # H, W\n",
" im = axes[0, i+1 if i < 3 else 0].imshow(feat_np, cmap='viridis') if i < 3 else None\n",
" if i < 3:\n",
" axes[0, i+1].imshow(feat_np, cmap='viridis')\n",
" axes[0, i+1].set_title(f'{name}\\n(mean across channels)', fontsize=10, fontweight='bold')\n",
" axes[0, i+1].axis('off')\n",
"\n",
"# Stage 4 in bottom row\n",
"feat4_np = features[3][0].mean(0).detach().numpy()\n",
"axes[1, 0].imshow(feat4_np, cmap='viridis')\n",
"axes[1, 0].set_title('Stage 4\\n7×7×768\\n(mean across channels)', fontsize=10, fontweight='bold')\n",
"axes[1, 0].axis('off')\n",
"\n",
"# Show individual channels from Stage 3\n",
"feat3 = features[2][0].detach().numpy() # C, H, W\n",
"for i in range(3):\n",
" axes[1, i+1].imshow(feat3[i*50], cmap='hot')\n",
" axes[1, i+1].set_title(f'Stage 3, Channel {i*50}\\n14×14', fontsize=10, fontweight='bold')\n",
" axes[1, i+1].axis('off')\n",
"\n",
"plt.suptitle('Swin Transformer Multi-Scale Feature Maps', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('swin_feature_maps.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Parameter count analysis\n",
"def count_params(model):\n",
" return sum(p.numel() for p in model.parameters())\n",
"\n",
"def count_trainable_params(model):\n",
" return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
"\n",
"print(f\"Swin-T Parameter Count:\")\n",
"print(f\" Total: {count_params(swin_model):,}\")\n",
"print(f\" Trainable: {count_trainable_params(swin_model):,}\")\n",
"\n",
"# Breakdown by stage\n",
"print(f\"\\nParameter breakdown:\")\n",
"for name, module in swin_model.named_children():\n",
" params = count_params(module)\n",
" print(f\" {name}: {params:,} ({params/count_params(swin_model)*100:.1f}%)\")\n",
"\n",
"# Memory usage\n",
"print(f\"\\nMemory usage (float32):\")\n",
"for i, feat in enumerate(features):\n",
" mem_mb = feat.numel() * 4 / 1024 / 1024\n",
" print(f\" Stage {i+1} features: {feat.shape} → {mem_mb:.2f} MB\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"The Swin Transformer backbone in GroundingDINO:\n",
"\n",
"1. **Processes** the input image through 4×4 patch partition → linear embedding\n",
"2. **Applies** hierarchical window attention (W-MSA + SW-MSA) in 4 stages\n",
"3. **Produces** multi-scale features at strides 8, 16, 32\n",
"4. **Projects** all scales to 256-dim for the Feature Enhancer\n",
"\n",
"Key tensor shapes (800×1333 input, Swin-T):\n",
"- Input: `1 × 3 × 800 × 1333`\n",
"- C2 (stride 8): `1 × 192 × 100 × 167`\n",
"- C3 (stride 16): `1 × 384 × 50 × 84`\n",
"- C4 (stride 32): `1 × 768 × 25 × 42`\n",
"- After projection: `1 × 256 × H_i × W_i` for each scale\n",
"\n",
"**Next**: [Notebook 03 - BERT Text Encoder](03_bert_text_encoder.ipynb)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}