{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GroundingDINO: Feature Enhancer (Cross-Modal Fusion)\n",
"\n",
"## Notebook 04 of 14\n",
"\n",
"This notebook covers the **Feature Enhancer** — the first stage of cross-modal fusion in GroundingDINO:\n",
"- Architecture of the Feature Enhancer\n",
"- Image-to-text and text-to-image cross-attention\n",
"- Deformable self-attention for image features\n",
"- Tensor shapes at every step\n",
"- Visualization of cross-modal attention patterns\n",
"\n",
"---\n",
"\n",
"## Table of Contents\n",
"1. [What is the Feature Enhancer?](#what-is)\n",
"2. [Architecture Overview](#architecture)\n",
"3. [Deformable Self-Attention (Image)](#deformable-attn)\n",
"4. [Cross-Modal Attention (Image↔Text)](#cross-modal)\n",
"5. [Feature Enhancer Layer](#enhancer-layer)\n",
"6. [Tensor Shapes](#tensor-shapes)\n",
"7. [Implementation & Visualization](#implementation)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. What is the Feature Enhancer? \n",
"\n",
"The Feature Enhancer is the **first fusion stage** in GroundingDINO. It takes:\n",
"- **Image features** from Swin Transformer (multi-scale, projected to 256-dim)\n",
"- **Text features** from BERT (token-level, projected to 256-dim)\n",
"\n",
"And produces **enhanced features** where each modality has been informed by the other.\n",
"\n",
"### Why is this needed?\n",
"\n",
"Without fusion, the image features don't \"know\" what text is being queried. The Feature Enhancer allows:\n",
"- Image features to attend to relevant text tokens\n",
"- Text features to attend to relevant image regions\n",
"- Both modalities to be aligned in a shared feature space\n",
"\n",
"### Key Design Choices\n",
"\n",
"| Component | Choice | Reason |\n",
"|-----------|--------|--------|\n",
"| Image self-attention | **Deformable** | Efficient for multi-scale features |\n",
"| Text self-attention | **Standard** | Text sequences are short |\n",
"| Cross-modal attention | **Standard** | Bidirectional image↔text |\n",
"| Number of layers | **6** | Balance between capacity and efficiency |\n",
"\n",
"### Comparison with GLIP\n",
"GLIP uses a single cross-modal fusion layer. GroundingDINO uses **6 layers** of deep fusion, enabling much richer cross-modal interactions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.patches as patches\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"print(f\"Using device: {device}\")\n",
"\n",
"# Feature dimensions\n",
"D = 256 # feature dimension\n",
"print(f\"Feature dimension: {D}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Architecture Overview \n",
"\n",
"### Feature Enhancer Structure\n",
"\n",
"The Feature Enhancer consists of **N=6 identical layers**, each containing:\n",
"\n",
"```\n",
"Image Features (B, Nv, 256) Text Features (B, L, 256)\n",
" │ │\n",
" ▼ ▼\n",
" Deformable Self-Attn Standard Self-Attn\n",
" (image → image) (text → text)\n",
" │ │\n",
" ▼ ▼\n",
" Image-to-Text Cross-Attn Text-to-Image Cross-Attn\n",
" (image queries, text keys) (text queries, image keys)\n",
" │ │\n",
" ▼ ▼\n",
" FFN (image) FFN (text)\n",
" │ │\n",
" ▼ ▼\n",
"Enhanced Image Features Enhanced Text Features\n",
"(B, Nv, 256) (B, L, 256)\n",
"```\n",
"\n",
"Where:\n",
"- $N_v$ = total visual tokens = sum of all scale tokens\n",
"- $L$ = text sequence length\n",
"- For 800×1333 input: $N_v = 100 \\times 167 + 50 \\times 84 + 25 \\times 42 = 16700 + 4200 + 1050 = 21950$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize Feature Enhancer architecture\n",
"fig, ax = plt.subplots(figsize=(18, 12))\n",
"ax.set_xlim(0, 18)\n",
"ax.set_ylim(0, 12)\n",
"ax.axis('off')\n",
"\n",
"def draw_box(ax, x, y, w, h, label, color, fontsize=9, alpha=1.0):\n",
" rect = patches.FancyBboxPatch((x, y), w, h, boxstyle='round,pad=0.1',\n",
" facecolor=color, edgecolor='black', linewidth=1.5, alpha=alpha)\n",
" ax.add_patch(rect)\n",
" ax.text(x + w/2, y + h/2, label, ha='center', va='center', \n",
" fontsize=fontsize, fontweight='bold', multialignment='center')\n",
"\n",
"def arrow(ax, x1, y1, x2, y2, color='black', lw=2):\n",
" ax.annotate('', xy=(x2, y2), xytext=(x1, y1),\n",
" arrowprops=dict(arrowstyle='->', color=color, lw=lw))\n",
"\n",
"# Inputs\n",
"draw_box(ax, 0.5, 10.0, 5.5, 1.5, 'Image Features\\n(B, Nv, 256)\\nNv = 21,950 tokens', '#BBDEFB', 10)\n",
"draw_box(ax, 12.0, 10.0, 5.5, 1.5, 'Text Features\\n(B, L, 256)\\nL = seq length', '#FFF176', 10)\n",
"\n",
"# Feature Enhancer box\n",
"rect = patches.FancyBboxPatch((0.3, 1.0), 17.4, 8.5, boxstyle='round,pad=0.2',\n",
" facecolor='#F3E5F5', edgecolor='purple', linewidth=3, alpha=0.3)\n",
"ax.add_patch(rect)\n",
"ax.text(9, 9.3, 'Feature Enhancer (6 Layers)', ha='center', fontsize=13, \n",
" fontweight='bold', color='purple')\n",
"\n",
"# Layer components - Image side\n",
"draw_box(ax, 0.5, 7.0, 5.5, 1.2, 'Deformable Self-Attention\\n(Image → Image)', '#C8E6C9', 9)\n",
"draw_box(ax, 0.5, 5.0, 5.5, 1.2, 'Image-to-Text Cross-Attention\\n(Q=Image, K/V=Text)', '#FFCC80', 9)\n",
"draw_box(ax, 0.5, 3.0, 5.5, 1.2, 'FFN (Image)\\n256 → 2048 → 256', '#F8BBD0', 9)\n",
"\n",
"# Layer components - Text side\n",
"draw_box(ax, 12.0, 7.0, 5.5, 1.2, 'Standard Self-Attention\\n(Text → Text)', '#C8E6C9', 9)\n",
"draw_box(ax, 12.0, 5.0, 5.5, 1.2, 'Text-to-Image Cross-Attention\\n(Q=Text, K/V=Image)', '#FFCC80', 9)\n",
"draw_box(ax, 12.0, 3.0, 5.5, 1.2, 'FFN (Text)\\n256 → 2048 → 256', '#F8BBD0', 9)\n",
"\n",
"# Arrows - Image side\n",
"arrow(ax, 3.25, 10.0, 3.25, 8.2, '#1565C0')\n",
"arrow(ax, 3.25, 7.0, 3.25, 6.2, '#1565C0')\n",
"arrow(ax, 3.25, 5.0, 3.25, 4.2, '#1565C0')\n",
"\n",
"# Arrows - Text side\n",
"arrow(ax, 14.75, 10.0, 14.75, 8.2, '#F57F17')\n",
"arrow(ax, 14.75, 7.0, 14.75, 6.2, '#F57F17')\n",
"arrow(ax, 14.75, 5.0, 14.75, 4.2, '#F57F17')\n",
"\n",
"# Cross-modal arrows\n",
"ax.annotate('', xy=(12.0, 5.6), xytext=(6.0, 7.6),\n",
" arrowprops=dict(arrowstyle='->', color='red', lw=2.5, \n",
" connectionstyle='arc3,rad=-0.3'))\n",
"ax.text(9, 7.0, 'Image features\\nattend to text', ha='center', fontsize=9, color='red',\n",
" bbox=dict(boxstyle='round', facecolor='white', edgecolor='red', alpha=0.8))\n",
"\n",
"ax.annotate('', xy=(6.0, 5.6), xytext=(12.0, 7.6),\n",
" arrowprops=dict(arrowstyle='->', color='blue', lw=2.5,\n",
" connectionstyle='arc3,rad=0.3'))\n",
"ax.text(9, 6.0, 'Text features\\nattend to image', ha='center', fontsize=9, color='blue',\n",
" bbox=dict(boxstyle='round', facecolor='white', edgecolor='blue', alpha=0.8))\n",
"\n",
"# Outputs\n",
"draw_box(ax, 0.5, 1.2, 5.5, 1.5, 'Enhanced Image Features\\n(B, Nv, 256)', '#E3F2FD', 10)\n",
"draw_box(ax, 12.0, 1.2, 5.5, 1.5, 'Enhanced Text Features\\n(B, L, 256)', '#FFFDE7', 10)\n",
"arrow(ax, 3.25, 3.0, 3.25, 2.7, '#1565C0')\n",
"arrow(ax, 14.75, 3.0, 14.75, 2.7, '#F57F17')\n",
"\n",
"ax.set_title('Feature Enhancer Architecture (Stage 1 Fusion)', fontsize=15, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('feature_enhancer_architecture.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Deformable Self-Attention (Image) \n",
"\n",
"### Why Deformable Attention?\n",
"\n",
"Standard attention over all $N_v \\approx 22,000$ visual tokens would be $O(N_v^2) \\approx 5 \\times 10^8$ — too expensive!\n",
"\n",
"**Deformable attention** samples only **K reference points** per query:\n",
"\n",
"$$\\text{DeformAttn}(q, p, x) = \\sum_{m=1}^{M} W_m \\left[ \\sum_{k=1}^{K} A_{mqk} \\cdot W_m' x(p + \\Delta p_{mqk}) \\right]$$\n",
"\n",
"where:\n",
"- $q$ = query feature\n",
"- $p$ = reference point (2D location)\n",
"- $\\Delta p_{mqk}$ = learned offset for head $m$, query $q$, sample $k$\n",
"- $A_{mqk}$ = attention weight (scalar)\n",
"- $K$ = number of sampling points (default 4)\n",
"- $M$ = number of heads (default 8)\n",
"\n",
"### Complexity\n",
"- Standard attention: $O(N_v^2 \\cdot D)$\n",
"- Deformable attention: $O(N_v \\cdot K \\cdot D)$ — **linear** in $N_v$!\n",
"\n",
"### Multi-Scale Deformable Attention\n",
"For multi-scale features, reference points are sampled from **all scales**:\n",
"$$\\text{MSDeformAttn}(q, \\hat{p}, \\{x^l\\}) = \\sum_{m=1}^{M} W_m \\left[ \\sum_{l=1}^{L} \\sum_{k=1}^{K} A_{mlqk} \\cdot W_m' x^l(\\phi_l(\\hat{p}) + \\Delta p_{mlqk}) \\right]$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class SimplifiedDeformableAttention(nn.Module):\n",
" \"\"\"Simplified deformable attention for illustration.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, n_heads=8, n_points=4):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" self.n_heads = n_heads\n",
" self.n_points = n_points\n",
" self.head_dim = d_model // n_heads\n",
" \n",
" # Sampling offset predictor\n",
" self.sampling_offsets = nn.Linear(d_model, n_heads * n_points * 2)\n",
" # Attention weight predictor\n",
" self.attention_weights = nn.Linear(d_model, n_heads * n_points)\n",
" # Value projection\n",
" self.value_proj = nn.Linear(d_model, d_model)\n",
" # Output projection\n",
" self.output_proj = nn.Linear(d_model, d_model)\n",
" \n",
" def forward(self, query, reference_points, input_flatten, input_spatial_shapes):\n",
" \"\"\"\n",
" query: (B, Nq, C) - query features\n",
" reference_points: (B, Nq, 2) - normalized reference points [0,1]\n",
" input_flatten: (B, Nv, C) - flattened multi-scale features\n",
" input_spatial_shapes: (L, 2) - spatial shapes of each scale\n",
" \"\"\"\n",
" B, Nq, C = query.shape\n",
" B, Nv, C = input_flatten.shape\n",
" \n",
" # Predict sampling offsets\n",
" offsets = self.sampling_offsets(query) # B, Nq, n_heads*n_points*2\n",
" offsets = offsets.view(B, Nq, self.n_heads, self.n_points, 2) # B, Nq, H, K, 2\n",
" \n",
" # Predict attention weights\n",
" attn_weights = self.attention_weights(query) # B, Nq, n_heads*n_points\n",
" attn_weights = attn_weights.view(B, Nq, self.n_heads, self.n_points) # B, Nq, H, K\n",
" attn_weights = F.softmax(attn_weights, dim=-1) # normalize over K points\n",
" \n",
" # Value projection\n",
" values = self.value_proj(input_flatten) # B, Nv, C\n",
" \n",
" # Simplified: use attention weights to aggregate (skip actual bilinear sampling)\n",
" # In practice, bilinear interpolation is used at offset locations\n",
" # Here we approximate with a weighted sum over random sampled positions\n",
" sampled_values = values[:, :Nq, :] # B, Nq, C (simplified)\n",
" \n",
" output = self.output_proj(sampled_values) # B, Nq, C\n",
" \n",
" return output, offsets, attn_weights\n",
"\n",
"# Test\n",
"B = 1\n",
"Nv = 1000 # simplified visual tokens\n",
"Nq = 100 # query tokens\n",
"D = 256\n",
"\n",
"deform_attn = SimplifiedDeformableAttention(d_model=D, n_heads=8, n_points=4)\n",
"query = torch.randn(B, Nq, D)\n",
"ref_pts = torch.rand(B, Nq, 2) # normalized [0,1]\n",
"input_flat = torch.randn(B, Nv, D)\n",
"spatial_shapes = torch.tensor([[50, 84], [25, 42]])\n",
"\n",
"out, offsets, attn_w = deform_attn(query, ref_pts, input_flat, spatial_shapes)\n",
"\n",
"print(\"Deformable Self-Attention:\")\n",
"print(f\" Query input: {query.shape} (B, Nq, D)\")\n",
"print(f\" Reference points: {ref_pts.shape} (B, Nq, 2)\")\n",
"print(f\" Input features: {input_flat.shape} (B, Nv, D)\")\n",
"print(f\" Output: {out.shape} (B, Nq, D)\")\n",
"print(f\" Sampling offsets: {offsets.shape} (B, Nq, n_heads, n_points, 2)\")\n",
"print(f\" Attn weights: {attn_w.shape} (B, Nq, n_heads, n_points)\")\n",
"print(f\"\\nComplexity: O(Nq × K × D) = O({Nq} × 4 × {D}) vs O(Nq × Nv × D) = O({Nq} × {Nv} × {D})\")\n",
"print(f\"Speedup: {Nv // 4}×\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize deformable attention sampling\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"# Feature map\n",
"H, W = 25, 42\n",
"feature_map = np.random.rand(H, W)\n",
"\n",
"# Standard attention: attends to ALL positions\n",
"ax1 = axes[0]\n",
"ax1.imshow(feature_map, cmap='Blues', aspect='equal')\n",
"# Show all positions being attended\n",
"for i in range(0, H, 3):\n",
" for j in range(0, W, 3):\n",
" ax1.plot(j, i, 'r.', markersize=3, alpha=0.5)\n",
"ax1.set_title(f'Standard Attention\\nAttends to ALL {H*W} positions\\nO(N²) complexity', \n",
" fontsize=11, fontweight='bold')\n",
"ax1.axis('off')\n",
"\n",
"# Deformable attention: attends to K=4 sampled positions\n",
"ax2 = axes[1]\n",
"ax2.imshow(feature_map, cmap='Blues', aspect='equal')\n",
"\n",
"# Show reference point and sampled offsets\n",
"np.random.seed(42)\n",
"ref_y, ref_x = H//2, W//2 # reference point\n",
"ax2.plot(ref_x, ref_y, 'r*', markersize=15, label='Reference point', zorder=5)\n",
"\n",
"# 8 heads × 4 points = 32 sampling locations\n",
"colors = plt.cm.Set1(np.linspace(0, 1, 8))\n",
"for head in range(8):\n",
" for k in range(4):\n",
" offset_y = np.random.randn() * 3\n",
" offset_x = np.random.randn() * 5\n",
" sample_y = ref_y + offset_y\n",
" sample_x = ref_x + offset_x\n",
" ax2.plot(sample_x, sample_y, 'o', color=colors[head], markersize=8, \n",
" alpha=0.8, zorder=4)\n",
" ax2.plot([ref_x, sample_x], [ref_y, sample_y], '-', \n",
" color=colors[head], alpha=0.4, linewidth=1)\n",
"\n",
"ax2.set_title(f'Deformable Attention\\nSamples K=4 points per head (8 heads)\\nO(N×K) complexity', \n",
" fontsize=11, fontweight='bold')\n",
"ax2.legend(loc='upper right', fontsize=8)\n",
"ax2.axis('off')\n",
"\n",
"# Complexity comparison\n",
"ax3 = axes[2]\n",
"Nv_vals = np.array([100, 500, 1000, 5000, 10000, 22000])\n",
"K = 4\n",
"D = 256\n",
"\n",
"standard_ops = Nv_vals**2 * D / 1e9\n",
"deform_ops = Nv_vals * K * D / 1e9\n",
"\n",
"ax3.loglog(Nv_vals, standard_ops, 'r-o', linewidth=2, label=f'Standard O(N²D)', markersize=8)\n",
"ax3.loglog(Nv_vals, deform_ops, 'b-o', linewidth=2, label=f'Deformable O(NKD), K={K}', markersize=8)\n",
"ax3.axvline(x=22000, color='green', linestyle='--', linewidth=2, label='GroundingDINO Nv')\n",
"ax3.set_xlabel('Number of visual tokens Nv', fontsize=11)\n",
"ax3.set_ylabel('GFLOPs', fontsize=11)\n",
"ax3.set_title('Attention Complexity Comparison', fontsize=11, fontweight='bold')\n",
"ax3.legend(fontsize=9)\n",
"ax3.grid(True, alpha=0.3)\n",
"\n",
"plt.suptitle('Deformable vs Standard Attention', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('deformable_attention.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Cross-Modal Attention (Image↔Text) \n",
"\n",
"### Image-to-Text Cross-Attention\n",
"Image features attend to text features:\n",
"$$\\text{ImgToText}(F_v, F_t) = \\text{softmax}\\left(\\frac{F_v W_Q (F_t W_K)^T}{\\sqrt{d_k}}\\right) F_t W_V$$\n",
"\n",
"- Query: Image features $(B, N_v, 256)$\n",
"- Key/Value: Text features $(B, L, 256)$\n",
"- Output: $(B, N_v, 256)$ — image features enriched with text context\n",
"\n",
"### Text-to-Image Cross-Attention\n",
"Text features attend to image features:\n",
"$$\\text{TextToImg}(F_t, F_v) = \\text{softmax}\\left(\\frac{F_t W_Q (F_v W_K)^T}{\\sqrt{d_k}}\\right) F_v W_V$$\n",
"\n",
"- Query: Text features $(B, L, 256)$\n",
"- Key/Value: Image features $(B, N_v, 256)$\n",
"- Output: $(B, L, 256)$ — text features enriched with visual context\n",
"\n",
"### Attention Mask\n",
"The text attention mask is used to prevent attending to padding tokens."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class CrossModalAttention(nn.Module):\n",
" \"\"\"Cross-modal attention between image and text features.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, n_heads=8):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" self.n_heads = n_heads\n",
" self.head_dim = d_model // n_heads\n",
" self.scale = self.head_dim ** -0.5\n",
" \n",
" # Image-to-Text\n",
" self.img_q = nn.Linear(d_model, d_model)\n",
" self.txt_k = nn.Linear(d_model, d_model)\n",
" self.txt_v = nn.Linear(d_model, d_model)\n",
" self.img_out = nn.Linear(d_model, d_model)\n",
" \n",
" # Text-to-Image\n",
" self.txt_q = nn.Linear(d_model, d_model)\n",
" self.img_k = nn.Linear(d_model, d_model)\n",
" self.img_v = nn.Linear(d_model, d_model)\n",
" self.txt_out = nn.Linear(d_model, d_model)\n",
" \n",
" def _attn(self, q, k, v, mask=None):\n",
" B, Nq, D = q.shape\n",
" H = self.n_heads\n",
" d_k = self.head_dim\n",
" Nk = k.shape[1]\n",
" \n",
" q = q.view(B, Nq, H, d_k).transpose(1, 2) # B, H, Nq, d_k\n",
" k = k.view(B, Nk, H, d_k).transpose(1, 2) # B, H, Nk, d_k\n",
" v = v.view(B, Nk, H, d_k).transpose(1, 2) # B, H, Nk, d_k\n",
" \n",
" attn = (q @ k.transpose(-2, -1)) * self.scale # B, H, Nq, Nk\n",
" \n",
" if mask is not None:\n",
" # mask: B, Nk → B, 1, 1, Nk\n",
" attn = attn.masked_fill(~mask.bool().unsqueeze(1).unsqueeze(2), float('-inf'))\n",
" \n",
" attn = F.softmax(attn, dim=-1)\n",
" attn = torch.nan_to_num(attn, nan=0.0) # handle all-masked rows\n",
" \n",
" out = (attn @ v).transpose(1, 2).contiguous().view(B, Nq, D) # B, Nq, D\n",
" return out, attn\n",
" \n",
" def forward(self, img_feat, txt_feat, txt_mask=None):\n",
" \"\"\"\n",
" img_feat: (B, Nv, D)\n",
" txt_feat: (B, L, D)\n",
" txt_mask: (B, L) - 1 for real tokens, 0 for padding\n",
" \"\"\"\n",
" # Image-to-Text: image queries attend to text\n",
" q_img = self.img_q(img_feat) # B, Nv, D\n",
" k_txt = self.txt_k(txt_feat) # B, L, D\n",
" v_txt = self.txt_v(txt_feat) # B, L, D\n",
" img_enhanced, img2txt_attn = self._attn(q_img, k_txt, v_txt, txt_mask)\n",
" img_enhanced = self.img_out(img_enhanced) # B, Nv, D\n",
" \n",
" # Text-to-Image: text queries attend to image\n",
" q_txt = self.txt_q(txt_feat) # B, L, D\n",
" k_img = self.img_k(img_feat) # B, Nv, D\n",
" v_img = self.img_v(img_feat) # B, Nv, D\n",
" txt_enhanced, txt2img_attn = self._attn(q_txt, k_img, v_img)\n",
" txt_enhanced = self.txt_out(txt_enhanced) # B, L, D\n",
" \n",
" return img_enhanced, txt_enhanced, img2txt_attn, txt2img_attn\n",
"\n",
"# Test\n",
"B = 1\n",
"Nv = 500 # simplified\n",
"L = 15\n",
"D = 256\n",
"\n",
"cross_attn = CrossModalAttention(d_model=D, n_heads=8)\n",
"img_feat = torch.randn(B, Nv, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"txt_mask = torch.ones(B, L)\n",
"txt_mask[0, 12:] = 0 # padding\n",
"\n",
"img_enh, txt_enh, i2t_attn, t2i_attn = cross_attn(img_feat, txt_feat, txt_mask)\n",
"\n",
"print(\"Cross-Modal Attention:\")\n",
"print(f\" Image input: {img_feat.shape} (B, Nv, D)\")\n",
"print(f\" Text input: {txt_feat.shape} (B, L, D)\")\n",
"print(f\" Image enhanced: {img_enh.shape} (B, Nv, D)\")\n",
"print(f\" Text enhanced: {txt_enh.shape} (B, L, D)\")\n",
"print(f\" Img→Txt attention: {i2t_attn.shape} (B, H, Nv, L)\")\n",
"print(f\" Txt→Img attention: {t2i_attn.shape} (B, H, L, Nv)\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize cross-modal attention patterns\n",
"fig, axes = plt.subplots(1, 2, figsize=(18, 6))\n",
"\n",
"# Image-to-Text attention (simplified: 20 image tokens × 15 text tokens)\n",
"Nv_small = 20\n",
"L_small = 12\n",
"tokens = ['[CLS]', 'cat', '.', 'dog', '.', 'bird', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n",
"\n",
"# Simulate attention: image regions attend to relevant text tokens\n",
"np.random.seed(42)\n",
"i2t_sim = np.random.rand(Nv_small, L_small) * 0.1\n",
"# Make some image regions strongly attend to specific tokens\n",
"i2t_sim[:7, 1] = np.random.rand(7) * 0.8 + 0.2 # first 7 regions → 'cat'\n",
"i2t_sim[7:14, 3] = np.random.rand(7) * 0.8 + 0.2 # next 7 regions → 'dog'\n",
"i2t_sim[14:, 5] = np.random.rand(6) * 0.8 + 0.2 # last 6 regions → 'bird'\n",
"# Normalize\n",
"i2t_sim = i2t_sim / i2t_sim.sum(axis=-1, keepdims=True)\n",
"\n",
"ax1 = axes[0]\n",
"im1 = ax1.imshow(i2t_sim, cmap='hot', aspect='auto', vmin=0)\n",
"ax1.set_xticks(range(L_small))\n",
"ax1.set_xticklabels(tokens, rotation=45, ha='right', fontsize=10)\n",
"ax1.set_yticks(range(Nv_small))\n",
"ax1.set_yticklabels([f'Img_{i}' for i in range(Nv_small)], fontsize=8)\n",
"ax1.set_title('Image-to-Text Cross-Attention\\n(Image queries attend to text tokens)', \n",
" fontsize=11, fontweight='bold')\n",
"ax1.set_xlabel('Text tokens (keys)')\n",
"ax1.set_ylabel('Image regions (queries)')\n",
"plt.colorbar(im1, ax=ax1)\n",
"\n",
"# Add region labels\n",
"ax1.text(-0.5, 3, 'Cat\\nregions', ha='right', va='center', fontsize=9, color='red', fontweight='bold')\n",
"ax1.text(-0.5, 10, 'Dog\\nregions', ha='right', va='center', fontsize=9, color='blue', fontweight='bold')\n",
"ax1.text(-0.5, 17, 'Bird\\nregions', ha='right', va='center', fontsize=9, color='green', fontweight='bold')\n",
"\n",
"# Text-to-Image attention\n",
"t2i_sim = i2t_sim.T # L × Nv\n",
"\n",
"ax2 = axes[1]\n",
"im2 = ax2.imshow(t2i_sim, cmap='Blues', aspect='auto', vmin=0)\n",
"ax2.set_yticks(range(L_small))\n",
"ax2.set_yticklabels(tokens, fontsize=10)\n",
"ax2.set_xticks(range(Nv_small))\n",
"ax2.set_xticklabels([f'I{i}' for i in range(Nv_small)], rotation=45, ha='right', fontsize=8)\n",
"ax2.set_title('Text-to-Image Cross-Attention\\n(Text tokens attend to image regions)', \n",
" fontsize=11, fontweight='bold')\n",
"ax2.set_ylabel('Text tokens (queries)')\n",
"ax2.set_xlabel('Image regions (keys)')\n",
"plt.colorbar(im2, ax=ax2)\n",
"\n",
"plt.suptitle('Cross-Modal Attention Patterns in Feature Enhancer', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('cross_modal_attention.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Feature Enhancer Layer \n",
"\n",
"### Complete Layer Structure\n",
"\n",
"Each Feature Enhancer layer processes both modalities in parallel:\n",
"\n",
"**Image branch:**\n",
"$$\\hat{F}_v = F_v + \\text{DeformSelfAttn}(\\text{LN}(F_v))$$\n",
"$$\\hat{F}_v = \\hat{F}_v + \\text{ImgToTextCrossAttn}(\\text{LN}(\\hat{F}_v), F_t)$$\n",
"$$F_v' = \\hat{F}_v + \\text{FFN}(\\text{LN}(\\hat{F}_v))$$\n",
"\n",
"**Text branch:**\n",
"$$\\hat{F}_t = F_t + \\text{SelfAttn}(\\text{LN}(F_t))$$\n",
"$$\\hat{F}_t = \\hat{F}_t + \\text{TextToImgCrossAttn}(\\text{LN}(\\hat{F}_t), F_v)$$\n",
"$$F_t' = \\hat{F}_t + \\text{FFN}(\\text{LN}(\\hat{F}_t))$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class FeatureEnhancerLayer(nn.Module):\n",
" \"\"\"Single Feature Enhancer layer.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, n_heads=8, ffn_dim=2048, dropout=0.1):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" \n",
" # Image self-attention (simplified as standard for illustration)\n",
" self.img_self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)\n",
" self.img_norm1 = nn.LayerNorm(d_model)\n",
" \n",
" # Text self-attention\n",
" self.txt_self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)\n",
" self.txt_norm1 = nn.LayerNorm(d_model)\n",
" \n",
" # Cross-modal attention\n",
" self.cross_attn = CrossModalAttention(d_model, n_heads)\n",
" self.img_norm2 = nn.LayerNorm(d_model)\n",
" self.txt_norm2 = nn.LayerNorm(d_model)\n",
" \n",
" # FFN for image\n",
" self.img_ffn = nn.Sequential(\n",
" nn.Linear(d_model, ffn_dim),\n",
" nn.ReLU(),\n",
" nn.Dropout(dropout),\n",
" nn.Linear(ffn_dim, d_model),\n",
" nn.Dropout(dropout)\n",
" )\n",
" self.img_norm3 = nn.LayerNorm(d_model)\n",
" \n",
" # FFN for text\n",
" self.txt_ffn = nn.Sequential(\n",
" nn.Linear(d_model, ffn_dim),\n",
" nn.ReLU(),\n",
" nn.Dropout(dropout),\n",
" nn.Linear(ffn_dim, d_model),\n",
" nn.Dropout(dropout)\n",
" )\n",
" self.txt_norm3 = nn.LayerNorm(d_model)\n",
" \n",
" def forward(self, img_feat, txt_feat, txt_key_padding_mask=None):\n",
" \"\"\"\n",
" img_feat: (B, Nv, D)\n",
" txt_feat: (B, L, D)\n",
" txt_key_padding_mask: (B, L) - True for padding positions\n",
" \"\"\"\n",
" # === Image branch ===\n",
" # Self-attention\n",
" img_norm = self.img_norm1(img_feat)\n",
" img_sa, _ = self.img_self_attn(img_norm, img_norm, img_norm)\n",
" img_feat = img_feat + img_sa\n",
" \n",
" # Cross-modal attention\n",
" img_norm = self.img_norm2(img_feat)\n",
" txt_norm = self.txt_norm1(txt_feat)\n",
" txt_sa, _ = self.txt_self_attn(txt_norm, txt_norm, txt_norm,\n",
" key_padding_mask=txt_key_padding_mask)\n",
" txt_feat = txt_feat + txt_sa\n",
" \n",
" txt_mask = None\n",
" if txt_key_padding_mask is not None:\n",
" txt_mask = (~txt_key_padding_mask).float()\n",
" \n",
" img_cross, txt_cross, i2t_attn, t2i_attn = self.cross_attn(\n",
" img_norm, self.txt_norm2(txt_feat), txt_mask\n",
" )\n",
" img_feat = img_feat + img_cross\n",
" txt_feat = txt_feat + txt_cross\n",
" \n",
" # FFN\n",
" img_feat = img_feat + self.img_ffn(self.img_norm3(img_feat))\n",
" txt_feat = txt_feat + self.txt_ffn(self.txt_norm3(txt_feat))\n",
" \n",
" return img_feat, txt_feat, i2t_attn, t2i_attn\n",
"\n",
"\n",
"class FeatureEnhancer(nn.Module):\n",
" \"\"\"Full Feature Enhancer with N layers.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, n_heads=8, ffn_dim=2048, n_layers=6):\n",
" super().__init__()\n",
" self.layers = nn.ModuleList([\n",
" FeatureEnhancerLayer(d_model, n_heads, ffn_dim)\n",
" for _ in range(n_layers)\n",
" ])\n",
" \n",
" def forward(self, img_feat, txt_feat, txt_key_padding_mask=None):\n",
" all_i2t_attns = []\n",
" all_t2i_attns = []\n",
" \n",
" for layer in self.layers:\n",
" img_feat, txt_feat, i2t, t2i = layer(img_feat, txt_feat, txt_key_padding_mask)\n",
" all_i2t_attns.append(i2t)\n",
" all_t2i_attns.append(t2i)\n",
" \n",
" return img_feat, txt_feat, all_i2t_attns, all_t2i_attns\n",
"\n",
"# Test\n",
"B = 1\n",
"Nv = 200 # simplified\n",
"L = 12\n",
"D = 256\n",
"\n",
"enhancer = FeatureEnhancer(d_model=D, n_heads=8, ffn_dim=2048, n_layers=6)\n",
"img_feat = torch.randn(B, Nv, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"txt_pad_mask = torch.zeros(B, L, dtype=torch.bool)\n",
"txt_pad_mask[0, 10:] = True # last 2 tokens are padding\n",
"\n",
"with torch.no_grad():\n",
" img_out, txt_out, i2t_attns, t2i_attns = enhancer(img_feat, txt_feat, txt_pad_mask)\n",
"\n",
"print(\"Feature Enhancer (6 layers):\")\n",
"print(f\" Image input: {img_feat.shape} (B, Nv, D)\")\n",
"print(f\" Text input: {txt_feat.shape} (B, L, D)\")\n",
"print(f\" Image output: {img_out.shape} (B, Nv, D)\")\n",
"print(f\" Text output: {txt_out.shape} (B, L, D)\")\n",
"print(f\" Img→Txt attns: {len(i2t_attns)} layers, each {i2t_attns[0].shape}\")\n",
"print(f\" Txt→Img attns: {len(t2i_attns)} layers, each {t2i_attns[0].shape}\")\n",
"\n",
"total_params = sum(p.numel() for p in enhancer.parameters())\n",
"print(f\"\\nFeature Enhancer parameters: {total_params:,} ({total_params/1e6:.1f}M)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Tensor Shapes \n",
"\n",
"Complete tensor shape tracking through the Feature Enhancer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"# For 800×1333 input, Swin-T\n",
"Nv = 100*167 + 50*84 + 25*42 # = 16700 + 4200 + 1050 = 21950\n",
"L = 20 # example text length\n",
"D = 256\n",
"H = 8 # attention heads\n",
"K = 4 # deformable sampling points\n",
"\n",
"shapes = [\n",
" ('Image features (input)', f'1 × {Nv} × 256', 'Flattened multi-scale features'),\n",
" ('Text features (input)', f'1 × {L} × 256', 'BERT output projected to 256'),\n",
" ('--- Layer 1 ---', '', ''),\n",
" ('Img DeformSelfAttn Q', f'1 × {Nv} × 256', 'Query for deformable attn'),\n",
" ('Img DeformSelfAttn offsets', f'1 × {Nv} × {H} × {K} × 2', 'Sampling offsets'),\n",
" ('Img DeformSelfAttn weights', f'1 × {Nv} × {H} × {K}', 'Attention weights'),\n",
" ('Img after DeformSelfAttn', f'1 × {Nv} × 256', 'Image self-attended'),\n",
" ('Txt SelfAttn Q/K/V', f'1 × {H} × {L} × 32', 'Text self-attention (d_k=32)'),\n",
" ('Txt SelfAttn weights', f'1 × {H} × {L} × {L}', 'Text self-attention map'),\n",
" ('Txt after SelfAttn', f'1 × {L} × 256', 'Text self-attended'),\n",
" ('Img→Txt CrossAttn Q', f'1 × {H} × {Nv} × 32', 'Image queries'),\n",
" ('Img→Txt CrossAttn K/V', f'1 × {H} × {L} × 32', 'Text keys/values'),\n",
" ('Img→Txt CrossAttn weights', f'1 × {H} × {Nv} × {L}', 'Image-to-text attention'),\n",
" ('Img after CrossAttn', f'1 × {Nv} × 256', 'Image enriched with text'),\n",
" ('Txt→Img CrossAttn weights', f'1 × {H} × {L} × {Nv}', 'Text-to-image attention'),\n",
" ('Txt after CrossAttn', f'1 × {L} × 256', 'Text enriched with image'),\n",
" ('Img FFN hidden', f'1 × {Nv} × 2048', '4× expansion'),\n",
" ('Img after FFN', f'1 × {Nv} × 256', 'Image layer output'),\n",
" ('Txt FFN hidden', f'1 × {L} × 2048', '4× expansion'),\n",
" ('Txt after FFN', f'1 × {L} × 256', 'Text layer output'),\n",
" ('--- After 6 Layers ---', '', ''),\n",
" ('Enhanced Image Features', f'1 × {Nv} × 256', 'Ready for query selection'),\n",
" ('Enhanced Text Features', f'1 × {L} × 256', 'Ready for decoder'),\n",
"]\n",
"\n",
"df = pd.DataFrame(shapes, columns=['Component', 'Shape', 'Description'])\n",
"\n",
"fig, ax = plt.subplots(figsize=(18, 12))\n",
"ax.axis('off')\n",
"table = ax.table(cellText=df.values, colLabels=df.columns, cellLoc='left', loc='center')\n",
"table.auto_set_font_size(False)\n",
"table.set_fontsize(8)\n",
"table.scale(1, 1.8)\n",
"\n",
"for j in range(3):\n",
" table[0, j].set_facecolor('#1565C0')\n",
" table[0, j].set_text_props(color='white', fontweight='bold')\n",
"\n",
"# Color code\n",
"for i in range(1, len(shapes)+1):\n",
" row = shapes[i-1]\n",
" if '---' in row[0]:\n",
" color = '#37474F'\n",
" for j in range(3):\n",
" table[i, j].set_facecolor(color)\n",
" table[i, j].set_text_props(color='white', fontweight='bold')\n",
" elif 'Img' in row[0]:\n",
" for j in range(3):\n",
" table[i, j].set_facecolor('#E3F2FD')\n",
" elif 'Txt' in row[0] or 'Text' in row[0]:\n",
" for j in range(3):\n",
" table[i, j].set_facecolor('#FFFDE7')\n",
" elif 'Enhanced' in row[0]:\n",
" for j in range(3):\n",
" table[i, j].set_facecolor('#E8F5E9')\n",
"\n",
"plt.title('Feature Enhancer Tensor Shapes (800×1333 input, Swin-T)', \n",
" fontsize=13, fontweight='bold', pad=20)\n",
"plt.tight_layout()\n",
"plt.savefig('feature_enhancer_shapes.png', dpi=150, bbox_inches='tight')\n",
"plt.show()\n",
"\n",
"print(f\"Total visual tokens Nv = {Nv:,}\")\n",
"print(f\" Scale 1 (stride 8): 100×167 = {100*167:,}\")\n",
"print(f\" Scale 2 (stride 16): 50×84 = {50*84:,}\")\n",
"print(f\" Scale 3 (stride 32): 25×42 = {25*42:,}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Implementation & Visualization \n",
"\n",
"Let's visualize how features change through the Feature Enhancer layers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Track feature evolution through layers\n",
"B = 1\n",
"Nv = 100\n",
"L = 10\n",
"D = 256\n",
"\n",
"# Initialize with random features\n",
"torch.manual_seed(42)\n",
"img_feat_init = torch.randn(B, Nv, D)\n",
"txt_feat_init = torch.randn(B, L, D)\n",
"\n",
"# Track features through each layer\n",
"img_feat = img_feat_init.clone()\n",
"txt_feat = txt_feat_init.clone()\n",
"\n",
"img_norms_per_layer = [img_feat.norm(dim=-1).mean().item()]\n",
"txt_norms_per_layer = [txt_feat.norm(dim=-1).mean().item()]\n",
"img_txt_sim_per_layer = []\n",
"\n",
"# Compute initial similarity\n",
"img_n = F.normalize(img_feat[0], dim=-1) # Nv, D\n",
"txt_n = F.normalize(txt_feat[0], dim=-1) # L, D\n",
"sim = (img_n @ txt_n.T).mean().item() # avg similarity\n",
"img_txt_sim_per_layer.append(sim)\n",
"\n",
"enhancer_single = FeatureEnhancer(d_model=D, n_heads=8, ffn_dim=1024, n_layers=6)\n",
"enhancer_single.eval()\n",
"\n",
"with torch.no_grad():\n",
" for i, layer in enumerate(enhancer_single.layers):\n",
" img_feat, txt_feat, _, _ = layer(img_feat, txt_feat)\n",
" img_norms_per_layer.append(img_feat.norm(dim=-1).mean().item())\n",
" txt_norms_per_layer.append(txt_feat.norm(dim=-1).mean().item())\n",
" \n",
" img_n = F.normalize(img_feat[0], dim=-1)\n",
" txt_n = F.normalize(txt_feat[0], dim=-1)\n",
" sim = (img_n @ txt_n.T).mean().item()\n",
" img_txt_sim_per_layer.append(sim)\n",
"\n",
"# Visualize\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 5))\n",
"\n",
"layers = range(7)\n",
"ax1 = axes[0]\n",
"ax1.plot(layers, img_norms_per_layer, 'b-o', linewidth=2, label='Image features', markersize=8)\n",
"ax1.plot(layers, txt_norms_per_layer, 'r-o', linewidth=2, label='Text features', markersize=8)\n",
"ax1.set_xlabel('Layer (0=input)')\n",
"ax1.set_ylabel('Mean Feature Norm')\n",
"ax1.set_title('Feature Norm Evolution\\nThrough Feature Enhancer', fontsize=11, fontweight='bold')\n",
"ax1.legend()\n",
"ax1.grid(True, alpha=0.3)\n",
"ax1.set_xticks(layers)\n",
"ax1.set_xticklabels(['Input'] + [f'L{i+1}' for i in range(6)])\n",
"\n",
"ax2 = axes[1]\n",
"ax2.plot(range(7), img_txt_sim_per_layer, 'g-o', linewidth=2, markersize=8)\n",
"ax2.set_xlabel('Layer (0=input)')\n",
"ax2.set_ylabel('Mean Cosine Similarity')\n",
"ax2.set_title('Image-Text Similarity\\nThrough Feature Enhancer', fontsize=11, fontweight='bold')\n",
"ax2.grid(True, alpha=0.3)\n",
"ax2.set_xticks(range(7))\n",
"ax2.set_xticklabels(['Input'] + [f'L{i+1}' for i in range(6)])\n",
"\n",
"# Feature distribution\n",
"ax3 = axes[2]\n",
"img_init_flat = img_feat_init[0].numpy().flatten()\n",
"img_final_flat = img_feat[0].detach().numpy().flatten()\n",
"ax3.hist(img_init_flat, bins=50, alpha=0.6, label='Image features (input)', color='blue', density=True)\n",
"ax3.hist(img_final_flat, bins=50, alpha=0.6, label='Image features (output)', color='green', density=True)\n",
"ax3.set_xlabel('Feature value')\n",
"ax3.set_ylabel('Density')\n",
"ax3.set_title('Feature Distribution\\nBefore vs After Feature Enhancer', fontsize=11, fontweight='bold')\n",
"ax3.legend()\n",
"ax3.grid(True, alpha=0.3)\n",
"\n",
"plt.suptitle('Feature Enhancer: Feature Evolution Analysis', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('feature_enhancer_evolution.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"The Feature Enhancer in GroundingDINO:\n",
"\n",
"1. **Takes** multi-scale image features `(B, Nv, 256)` and text features `(B, L, 256)`\n",
"2. **Applies** 6 layers of bidirectional cross-modal fusion\n",
"3. **Uses** deformable self-attention for efficient image processing\n",
"4. **Produces** enhanced features where each modality is informed by the other\n",
"\n",
"Key tensor shapes:\n",
"- Image input/output: `(B, 21950, 256)` for 800×1333 input\n",
"- Text input/output: `(B, L, 256)`\n",
"- Cross-modal attention: `(B, 8, Nv, L)` and `(B, 8, L, Nv)`\n",
"\n",
"This is **Stage 1 of 3** fusion stages in GroundingDINO.\n",
"\n",
"**Next**: [Notebook 05 - Language-Guided Query Selection](05_language_guided_query_selection.ipynb)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}