{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GroundingDINO: Cross-Modal Decoder\n",
"\n",
"## Notebook 06 of 14\n",
"\n",
"This notebook covers the **Cross-Modal Decoder** — Stage 3 of fusion in GroundingDINO:\n",
"- Decoder architecture with 6 layers\n",
"- Self-attention, image cross-attention, text cross-attention\n",
"- Iterative box refinement\n",
"- Detection head (box regression + contrastive score)\n",
"- Tensor shapes at every step\n",
"\n",
"---\n",
"\n",
"## Table of Contents\n",
"1. [Decoder Overview](#overview)\n",
"2. [Decoder Layer Structure](#layer-structure)\n",
"3. [Self-Attention on Queries](#self-attn)\n",
"4. [Image Cross-Attention (Deformable)](#img-cross-attn)\n",
"5. [Text Cross-Attention](#txt-cross-attn)\n",
"6. [Iterative Box Refinement](#box-refinement)\n",
"7. [Detection Head](#detection-head)\n",
"8. [Tensor Shapes](#tensor-shapes)\n",
"9. [Implementation & Visualization](#implementation)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Decoder Overview \n",
"\n",
"The Cross-Modal Decoder is the **final stage** of GroundingDINO. It takes:\n",
"- **Queries** from Language-Guided Query Selection: `(B, 900, 256)`\n",
"- **Anchor boxes**: `(B, 900, 4)`\n",
"- **Enhanced image features**: `(B, Nv, 256)`\n",
"- **Enhanced text features**: `(B, L, 256)`\n",
"\n",
"And produces:\n",
"- **Bounding boxes**: `(B, 900, 4)` — refined box predictions\n",
"- **Contrastive scores**: `(B, 900, L)` — similarity to each text token\n",
"\n",
"### Key Innovation: Text Cross-Attention in Decoder\n",
"Unlike standard DINO decoder (which only has image cross-attention), GroundingDINO adds **text cross-attention** at every decoder layer:\n",
"\n",
"```\n",
"Standard DINO Decoder Layer:\n",
" Self-Attn → Image Cross-Attn → FFN\n",
"\n",
"GroundingDINO Decoder Layer:\n",
" Self-Attn → Image Cross-Attn → Text Cross-Attn → FFN\n",
"```\n",
"\n",
"This allows each query to continuously refine its understanding of both the visual content AND the text query."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.patches as patches\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"print(f\"Using device: {device}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize decoder architecture\n",
"fig, ax = plt.subplots(figsize=(16, 14))\n",
"ax.set_xlim(0, 16)\n",
"ax.set_ylim(0, 14)\n",
"ax.axis('off')\n",
"\n",
"def draw_box(ax, x, y, w, h, label, color, fontsize=9):\n",
" rect = patches.FancyBboxPatch((x, y), w, h, boxstyle='round,pad=0.1',\n",
" facecolor=color, edgecolor='black', linewidth=1.5)\n",
" ax.add_patch(rect)\n",
" ax.text(x + w/2, y + h/2, label, ha='center', va='center', \n",
" fontsize=fontsize, fontweight='bold', multialignment='center')\n",
"\n",
"def arrow(ax, x1, y1, x2, y2, color='black', lw=2):\n",
" ax.annotate('', xy=(x2, y2), xytext=(x1, y1),\n",
" arrowprops=dict(arrowstyle='->', color=color, lw=lw))\n",
"\n",
"# Inputs\n",
"draw_box(ax, 0.5, 12.5, 4.5, 1.0, 'Queries (B, 900, 256)\\n+ Anchors (B, 900, 4)', '#C8E6C9', 9)\n",
"draw_box(ax, 5.5, 12.5, 4.5, 1.0, 'Image Features\\n(B, Nv, 256)', '#BBDEFB', 9)\n",
"draw_box(ax, 10.5, 12.5, 4.5, 1.0, 'Text Features\\n(B, L, 256)', '#FFF176', 9)\n",
"\n",
"# Decoder layer box\n",
"rect = patches.FancyBboxPatch((0.3, 2.5), 15.4, 9.5, boxstyle='round,pad=0.2',\n",
" facecolor='#F3E5F5', edgecolor='purple', linewidth=3, alpha=0.3)\n",
"ax.add_patch(rect)\n",
"ax.text(8, 11.8, 'Cross-Modal Decoder Layer (×6)', ha='center', fontsize=13, \n",
" fontweight='bold', color='purple')\n",
"\n",
"# Layer components\n",
"components = [\n",
" (3.5, 10.0, 9.0, 1.2, 'Self-Attention on Queries\\n(B, 900, 256) → (B, 900, 256)', '#C8E6C9'),\n",
" (3.5, 8.0, 9.0, 1.2, 'Image Cross-Attention (Deformable)\\nQ=queries, K/V=image features → (B, 900, 256)', '#BBDEFB'),\n",
" (3.5, 6.0, 9.0, 1.2, 'Text Cross-Attention\\nQ=queries, K/V=text features → (B, 900, 256)', '#FFF176'),\n",
" (3.5, 4.0, 9.0, 1.2, 'FFN (256 → 2048 → 256)\\n+ LayerNorm + Residual', '#F8BBD0'),\n",
"]\n",
"for x, y, w, h, label, color in components:\n",
" draw_box(ax, x, y, w, h, label, color, 10)\n",
"\n",
"# Arrows between components\n",
"for y_from, y_to in [(12.5, 11.2), (10.0, 9.2), (8.0, 7.2), (6.0, 5.2)]:\n",
" arrow(ax, 8, y_from, 8, y_to, 'black', 2)\n",
"\n",
"# Cross-attention inputs\n",
"arrow(ax, 7.75, 12.5, 5.5, 9.2, '#1565C0') # image features → img cross-attn\n",
"arrow(ax, 12.75, 12.5, 10.5, 7.2, '#F57F17') # text features → txt cross-attn\n",
"\n",
"# Outputs\n",
"draw_box(ax, 0.5, 0.5, 6.5, 1.5, 'Box Predictions\\n(B, 900, 4) per layer\\n(iterative refinement)', '#FFCC80', 9)\n",
"draw_box(ax, 8.5, 0.5, 6.5, 1.5, 'Contrastive Scores\\n(B, 900, L)\\n(query-text similarity)', '#E1BEE7', 9)\n",
"\n",
"arrow(ax, 5.5, 2.5, 3.75, 2.0, 'black', 2)\n",
"arrow(ax, 10.5, 2.5, 11.75, 2.0, 'black', 2)\n",
"\n",
"ax.set_title('GroundingDINO Cross-Modal Decoder\\n(Stage 3 Fusion)', fontsize=15, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('cross_modal_decoder.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Decoder Layer Structure \n",
"\n",
"### Mathematical Formulation\n",
"\n",
"For each decoder layer $l$, given queries $Q^l$, image features $F_v$, text features $F_t$:\n",
"\n",
"**Step 1: Self-Attention**\n",
"$$\\hat{Q}^l = Q^l + \\text{SelfAttn}(\\text{LN}(Q^l))$$\n",
"\n",
"**Step 2: Image Cross-Attention (Deformable)**\n",
"$$\\tilde{Q}^l = \\hat{Q}^l + \\text{DeformCrossAttn}(\\text{LN}(\\hat{Q}^l), F_v, A^l)$$\n",
"\n",
"where $A^l$ are the current anchor boxes (updated each layer)\n",
"\n",
"**Step 3: Text Cross-Attention** ← *GroundingDINO's key addition*\n",
"$$\\bar{Q}^l = \\tilde{Q}^l + \\text{TextCrossAttn}(\\text{LN}(\\tilde{Q}^l), F_t)$$\n",
"\n",
"**Step 4: FFN**\n",
"$$Q^{l+1} = \\bar{Q}^l + \\text{FFN}(\\text{LN}(\\bar{Q}^l))$$\n",
"\n",
"**Step 5: Box Refinement**\n",
"$$\\Delta b^l = \\text{BoxHead}(Q^{l+1})$$\n",
"$$A^{l+1} = \\text{sigmoid}(\\text{inverse\\_sigmoid}(A^l) + \\Delta b^l)$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def inverse_sigmoid(x, eps=1e-5):\n",
" \"\"\"Inverse sigmoid function for box refinement.\"\"\"\n",
" x = x.clamp(min=0, max=1)\n",
" x1 = x.clamp(min=eps)\n",
" x2 = (1 - x).clamp(min=eps)\n",
" return torch.log(x1/x2)\n",
"\n",
"class CrossModalDecoderLayer(nn.Module):\n",
" \"\"\"Single Cross-Modal Decoder Layer.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, n_heads=8, ffn_dim=2048, n_points=4, dropout=0.1):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" self.n_heads = n_heads\n",
" self.head_dim = d_model // n_heads\n",
" \n",
" # Self-attention\n",
" self.self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)\n",
" self.norm1 = nn.LayerNorm(d_model)\n",
" \n",
" # Image cross-attention (simplified as standard for illustration)\n",
" self.img_cross_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)\n",
" self.norm2 = nn.LayerNorm(d_model)\n",
" \n",
" # Text cross-attention (GroundingDINO's key addition)\n",
" self.txt_cross_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)\n",
" self.norm3 = nn.LayerNorm(d_model)\n",
" \n",
" # FFN\n",
" self.ffn = nn.Sequential(\n",
" nn.Linear(d_model, ffn_dim),\n",
" nn.ReLU(),\n",
" nn.Dropout(dropout),\n",
" nn.Linear(ffn_dim, d_model),\n",
" nn.Dropout(dropout)\n",
" )\n",
" self.norm4 = nn.LayerNorm(d_model)\n",
" \n",
" # Box refinement head\n",
" self.box_head = nn.Sequential(\n",
" nn.Linear(d_model, d_model),\n",
" nn.ReLU(),\n",
" nn.Linear(d_model, 4)\n",
" )\n",
" \n",
" def forward(self, queries, img_feat, txt_feat, anchors, txt_key_padding_mask=None):\n",
" \"\"\"\n",
" queries: (B, K, D)\n",
" img_feat: (B, Nv, D)\n",
" txt_feat: (B, L, D)\n",
" anchors: (B, K, 4) - current anchor boxes\n",
" \"\"\"\n",
" B, K, D = queries.shape\n",
" \n",
" # Step 1: Self-attention\n",
" q_norm = self.norm1(queries)\n",
" sa_out, sa_weights = self.self_attn(q_norm, q_norm, q_norm)\n",
" queries = queries + sa_out\n",
" \n",
" # Step 2: Image cross-attention\n",
" q_norm = self.norm2(queries)\n",
" img_ca_out, img_ca_weights = self.img_cross_attn(q_norm, img_feat, img_feat)\n",
" queries = queries + img_ca_out\n",
" \n",
" # Step 3: Text cross-attention (GroundingDINO addition)\n",
" q_norm = self.norm3(queries)\n",
" txt_ca_out, txt_ca_weights = self.txt_cross_attn(\n",
" q_norm, txt_feat, txt_feat,\n",
" key_padding_mask=txt_key_padding_mask\n",
" )\n",
" queries = queries + txt_ca_out\n",
" \n",
" # Step 4: FFN\n",
" queries = queries + self.ffn(self.norm4(queries))\n",
" \n",
" # Step 5: Box refinement\n",
" delta_box = self.box_head(queries) # B, K, 4\n",
" new_anchors = torch.sigmoid(inverse_sigmoid(anchors) + delta_box) # B, K, 4\n",
" \n",
" return queries, new_anchors, sa_weights, img_ca_weights, txt_ca_weights\n",
"\n",
"# Test\n",
"B, K, Nv, L, D = 1, 50, 200, 12, 256\n",
"layer = CrossModalDecoderLayer(d_model=D, n_heads=8, ffn_dim=1024)\n",
"\n",
"queries = torch.randn(B, K, D)\n",
"img_feat = torch.randn(B, Nv, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"anchors = torch.rand(B, K, 4) # normalized boxes\n",
"\n",
"out_q, new_anchors, sa_w, img_ca_w, txt_ca_w = layer(queries, img_feat, txt_feat, anchors)\n",
"\n",
"print(\"Cross-Modal Decoder Layer:\")\n",
"print(f\" Input queries: {queries.shape} (B, K, D)\")\n",
"print(f\" Image features: {img_feat.shape} (B, Nv, D)\")\n",
"print(f\" Text features: {txt_feat.shape} (B, L, D)\")\n",
"print(f\" Input anchors: {anchors.shape} (B, K, 4)\")\n",
"print(f\" Output queries: {out_q.shape} (B, K, D)\")\n",
"print(f\" Refined anchors: {new_anchors.shape} (B, K, 4)\")\n",
"print(f\" Self-attn weights: {sa_w.shape} (B, K, K)\")\n",
"print(f\" Img cross-attn: {img_ca_w.shape} (B, K, Nv)\")\n",
"print(f\" Txt cross-attn: {txt_ca_w.shape} (B, K, L)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Self-Attention on Queries \n",
"\n",
"### Purpose\n",
"Self-attention among the 900 queries allows them to:\n",
"- Suppress duplicate detections (NMS-free!)\n",
"- Share information between queries\n",
"- Coordinate to cover different objects\n",
"\n",
"### Tensor Shapes\n",
"```\n",
"Input: (B, 900, 256)\n",
"Q = K = V = queries\n",
"Q, K, V: (B, 8, 900, 32) [8 heads, d_k=32]\n",
"Attn: (B, 8, 900, 900)\n",
"Output: (B, 900, 256)\n",
"```\n",
"\n",
"### Note on Complexity\n",
"Self-attention over 900 queries: $O(900^2) = 810,000$ — manageable!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize self-attention among queries\n",
"K = 30 # simplified\n",
"D = 256\n",
"n_heads = 8\n",
"\n",
"# Simulate query self-attention\n",
"np.random.seed(42)\n",
"# Create structured queries: some similar (potential duplicates)\n",
"queries_sim = np.random.rand(K, D) * 0.1\n",
"# Group 1: queries 0-9 are similar (same object)\n",
"queries_sim[:10] += np.random.rand(D) * 2\n",
"# Group 2: queries 10-19 are similar (another object)\n",
"queries_sim[10:20] += np.random.rand(D) * 2\n",
"# Group 3: queries 20-29 are diverse\n",
"queries_sim[20:] += np.random.rand(10, D)\n",
"\n",
"# Compute self-attention\n",
"q_norm = queries_sim / (np.linalg.norm(queries_sim, axis=-1, keepdims=True) + 1e-8)\n",
"self_attn_sim = q_norm @ q_norm.T\n",
"self_attn_weights = np.exp(self_attn_sim / np.sqrt(D//n_heads))\n",
"self_attn_weights /= self_attn_weights.sum(axis=-1, keepdims=True)\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
"\n",
"ax1 = axes[0]\n",
"im = ax1.imshow(self_attn_weights, cmap='hot', aspect='equal')\n",
"ax1.set_title('Query Self-Attention Weights\\n(Queries attending to each other)', fontsize=11, fontweight='bold')\n",
"ax1.set_xlabel('Key query index')\n",
"ax1.set_ylabel('Query index')\n",
"# Draw group boundaries\n",
"for boundary in [10, 20]:\n",
" ax1.axhline(y=boundary-0.5, color='cyan', linewidth=2)\n",
" ax1.axvline(x=boundary-0.5, color='cyan', linewidth=2)\n",
"ax1.text(5, 5, 'Group 1\\n(same obj)', ha='center', va='center', fontsize=8, color='cyan')\n",
"ax1.text(15, 15, 'Group 2\\n(same obj)', ha='center', va='center', fontsize=8, color='cyan')\n",
"plt.colorbar(im, ax=ax1)\n",
"\n",
"# Show how self-attention suppresses duplicates\n",
"ax2 = axes[1]\n",
"ax2.set_xlim(0, 10)\n",
"ax2.set_ylim(0, 8)\n",
"ax2.axis('off')\n",
"ax2.set_title('Self-Attention: Duplicate Suppression\\n(NMS-free detection)', fontsize=11, fontweight='bold')\n",
"\n",
"# Before self-attention: multiple overlapping boxes\n",
"ax2.text(2.5, 7.5, 'Before Self-Attention', ha='center', fontsize=10, fontweight='bold', color='red')\n",
"for i, (x, y, w, h) in enumerate([(1, 5, 2, 1.5), (1.3, 5.2, 2, 1.5), (1.1, 4.9, 2, 1.5)]):\n",
" rect = patches.Rectangle((x, y), w, h, linewidth=2, edgecolor='red', facecolor='none', alpha=0.7)\n",
" ax2.add_patch(rect)\n",
"ax2.text(2.5, 4.5, '3 overlapping\\ndetections', ha='center', fontsize=9, color='red')\n",
"\n",
"ax2.annotate('', xy=(5.5, 6.0), xytext=(4.0, 6.0),\n",
" arrowprops=dict(arrowstyle='->', color='black', lw=2))\n",
"ax2.text(4.75, 6.3, 'Self-Attn', ha='center', fontsize=9)\n",
"\n",
"# After self-attention: single box\n",
"ax2.text(7.5, 7.5, 'After Self-Attention', ha='center', fontsize=10, fontweight='bold', color='green')\n",
"rect = patches.Rectangle((6, 5), 2, 1.5, linewidth=2.5, edgecolor='green', facecolor='lightgreen', alpha=0.5)\n",
"ax2.add_patch(rect)\n",
"ax2.text(7.5, 4.5, '1 clean\\ndetection', ha='center', fontsize=9, color='green')\n",
"\n",
"plt.suptitle('Query Self-Attention Analysis', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('decoder_self_attention.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Image Cross-Attention (Deformable) \n",
"\n",
"### Purpose\n",
"Each query attends to relevant image regions to gather visual evidence:\n",
"- Query = \"what am I looking for?\"\n",
"- Image features = \"what's in the image?\"\n",
"- Cross-attention = \"find the relevant visual evidence\"\n",
"\n",
"### Deformable Cross-Attention\n",
"Uses the current anchor box as the reference point:\n",
"$$\\text{DeformCrossAttn}(q, a, F_v) = \\sum_{m,k} A_{mqk} \\cdot W_m' F_v(\\phi(a) + \\Delta p_{mqk})$$\n",
"\n",
"where $\\phi(a)$ maps the anchor box center to feature map coordinates.\n",
"\n",
"### Tensor Shapes\n",
"```\n",
"Query: (B, 900, 256)\n",
"Anchors: (B, 900, 4) → reference points\n",
"Offsets: (B, 900, 8, 4, 2) [8 heads, 4 points, 2D offset]\n",
"Weights: (B, 900, 8, 4) [8 heads, 4 points]\n",
"Output: (B, 900, 256)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize image cross-attention\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"# Feature map\n",
"H, W = 25, 42\n",
"feature_map = np.random.rand(H, W)\n",
"# Add some structure\n",
"feature_map[5:10, 5:12] = 0.9 # object 1\n",
"feature_map[15:20, 25:35] = 0.85 # object 2\n",
"\n",
"# Query 1: anchor at object 1\n",
"ax1 = axes[0]\n",
"ax1.imshow(feature_map, cmap='viridis', aspect='equal')\n",
"# Anchor box\n",
"anchor1 = patches.Rectangle((5, 5), 7, 5, linewidth=2, edgecolor='red', facecolor='none')\n",
"ax1.add_patch(anchor1)\n",
"ax1.text(8.5, 7.5, 'Anchor', ha='center', va='center', fontsize=9, color='red', fontweight='bold')\n",
"# Sampling points\n",
"np.random.seed(42)\n",
"for head in range(4):\n",
" for k in range(4):\n",
" sx = 8.5 + np.random.randn() * 3\n",
" sy = 7.5 + np.random.randn() * 2\n",
" ax1.plot(sx, sy, 'c*', markersize=10, zorder=5)\n",
"ax1.set_title('Query 1: Deformable Sampling\\naround anchor box', fontsize=11, fontweight='bold')\n",
"ax1.axis('off')\n",
"\n",
"# Query 2: anchor at object 2\n",
"ax2 = axes[1]\n",
"ax2.imshow(feature_map, cmap='viridis', aspect='equal')\n",
"anchor2 = patches.Rectangle((25, 15), 10, 5, linewidth=2, edgecolor='yellow', facecolor='none')\n",
"ax2.add_patch(anchor2)\n",
"ax2.text(30, 17.5, 'Anchor', ha='center', va='center', fontsize=9, color='yellow', fontweight='bold')\n",
"for head in range(4):\n",
" for k in range(4):\n",
" sx = 30 + np.random.randn() * 4\n",
" sy = 17.5 + np.random.randn() * 2\n",
" ax2.plot(sx, sy, 'y*', markersize=10, zorder=5)\n",
"ax2.set_title('Query 2: Deformable Sampling\\naround different anchor', fontsize=11, fontweight='bold')\n",
"ax2.axis('off')\n",
"\n",
"# Cross-attention weights visualization\n",
"ax3 = axes[2]\n",
"K_small = 20\n",
"Nv_small = H * W\n",
"# Simulate cross-attention weights\n",
"ca_weights = np.random.rand(K_small, Nv_small) * 0.01\n",
"# Each query focuses on its anchor region\n",
"for i in range(K_small):\n",
" anchor_y = np.random.randint(0, H)\n",
" anchor_x = np.random.randint(0, W)\n",
" # High attention near anchor\n",
" for dy in range(-3, 4):\n",
" for dx in range(-5, 6):\n",
" y = anchor_y + dy\n",
" x = anchor_x + dx\n",
" if 0 <= y < H and 0 <= x < W:\n",
" dist = np.sqrt(dy**2 + dx**2)\n",
" ca_weights[i, y*W + x] = np.exp(-dist/2)\n",
"ca_weights /= ca_weights.sum(axis=-1, keepdims=True)\n",
"\n",
"# Show as heatmap (reshaped)\n",
"ca_map = ca_weights[0].reshape(H, W)\n",
"im = ax3.imshow(ca_map, cmap='hot', aspect='equal')\n",
"ax3.set_title('Image Cross-Attention Weights\\n(Query 0 attending to image)', fontsize=11, fontweight='bold')\n",
"plt.colorbar(im, ax=ax3)\n",
"\n",
"plt.suptitle('Image Cross-Attention in Decoder', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('decoder_image_cross_attn.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Text Cross-Attention \n",
"\n",
"### Purpose (GroundingDINO's Key Innovation)\n",
"Each query attends to text tokens to understand what it should detect:\n",
"- Query = \"I found something at this location\"\n",
"- Text features = \"what words describe what I'm looking for?\"\n",
"- Cross-attention = \"which words match what I found?\"\n",
"\n",
"### Tensor Shapes\n",
"```\n",
"Query: (B, 900, 256)\n",
"Text: (B, L, 256)\n",
"Q: (B, 8, 900, 32)\n",
"K/V: (B, 8, L, 32)\n",
"Attn: (B, 8, 900, L) ← each query attends to each text token\n",
"Output: (B, 900, 256)\n",
"```\n",
"\n",
"### Connection to Contrastive Score\n",
"The text cross-attention weights `(B, 8, 900, L)` are related to the final contrastive scores used for detection confidence."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize text cross-attention\n",
"K_small = 15\n",
"L_small = 10\n",
"tokens = ['[CLS]', 'cat', '.', 'dog', '.', 'bird', '[SEP]', '[PAD]', '[PAD]', '[PAD]']\n",
"\n",
"# Simulate text cross-attention weights\n",
"np.random.seed(42)\n",
"txt_ca = np.random.rand(K_small, L_small) * 0.05\n",
"# Queries 0-4: detect cats → attend to 'cat' token\n",
"txt_ca[:5, 1] = np.random.rand(5) * 0.7 + 0.3\n",
"# Queries 5-9: detect dogs → attend to 'dog' token\n",
"txt_ca[5:10, 3] = np.random.rand(5) * 0.7 + 0.3\n",
"# Queries 10-14: detect birds → attend to 'bird' token\n",
"txt_ca[10:, 5] = np.random.rand(5) * 0.7 + 0.3\n",
"# Normalize\n",
"txt_ca[:, 7:] = 0 # padding tokens\n",
"txt_ca /= (txt_ca.sum(axis=-1, keepdims=True) + 1e-8)\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(16, 6))\n",
"\n",
"ax1 = axes[0]\n",
"im = ax1.imshow(txt_ca, cmap='Blues', aspect='auto', vmin=0)\n",
"ax1.set_xticks(range(L_small))\n",
"ax1.set_xticklabels(tokens, rotation=45, ha='right', fontsize=10)\n",
"ax1.set_yticks(range(K_small))\n",
"ax1.set_yticklabels([f'Q{i}' for i in range(K_small)], fontsize=9)\n",
"ax1.set_title('Text Cross-Attention Weights\\n(Each query attends to text tokens)', fontsize=11, fontweight='bold')\n",
"ax1.set_xlabel('Text tokens (keys)')\n",
"ax1.set_ylabel('Queries')\n",
"plt.colorbar(im, ax=ax1)\n",
"\n",
"# Add group labels\n",
"ax1.text(-1.5, 2, 'Cat\\nqueries', ha='right', va='center', fontsize=9, color='red', fontweight='bold')\n",
"ax1.text(-1.5, 7, 'Dog\\nqueries', ha='right', va='center', fontsize=9, color='blue', fontweight='bold')\n",
"ax1.text(-1.5, 12, 'Bird\\nqueries', ha='right', va='center', fontsize=9, color='green', fontweight='bold')\n",
"\n",
"# Show how text attention determines detection category\n",
"ax2 = axes[1]\n",
"ax2.set_xlim(0, 10)\n",
"ax2.set_ylim(0, 8)\n",
"ax2.axis('off')\n",
"ax2.set_title('Text Cross-Attention → Category Assignment', fontsize=11, fontweight='bold')\n",
"\n",
"# Query attending to 'cat'\n",
"ax2.text(2, 7.5, 'Query 2', ha='center', fontsize=10, fontweight='bold')\n",
"rect = patches.FancyBboxPatch((0.5, 6.5), 3, 0.8, boxstyle='round,pad=0.1',\n",
" facecolor='#FFCDD2', edgecolor='red', linewidth=2)\n",
"ax2.add_patch(rect)\n",
"ax2.text(2, 6.9, 'Attends to: \"cat\" (0.85)', ha='center', va='center', fontsize=9)\n",
"ax2.annotate('', xy=(2, 5.8), xytext=(2, 6.5),\n",
" arrowprops=dict(arrowstyle='->', color='red', lw=2))\n",
"rect2 = patches.FancyBboxPatch((0.5, 5.0), 3, 0.7, boxstyle='round,pad=0.1',\n",
" facecolor='#C8E6C9', edgecolor='green', linewidth=2)\n",
"ax2.add_patch(rect2)\n",
"ax2.text(2, 5.35, '→ Detected: CAT (score=0.85)', ha='center', va='center', fontsize=9)\n",
"\n",
"# Query attending to 'dog'\n",
"ax2.text(7.5, 7.5, 'Query 7', ha='center', fontsize=10, fontweight='bold')\n",
"rect3 = patches.FancyBboxPatch((6, 6.5), 3, 0.8, boxstyle='round,pad=0.1',\n",
" facecolor='#BBDEFB', edgecolor='blue', linewidth=2)\n",
"ax2.add_patch(rect3)\n",
"ax2.text(7.5, 6.9, 'Attends to: \"dog\" (0.78)', ha='center', va='center', fontsize=9)\n",
"ax2.annotate('', xy=(7.5, 5.8), xytext=(7.5, 6.5),\n",
" arrowprops=dict(arrowstyle='->', color='blue', lw=2))\n",
"rect4 = patches.FancyBboxPatch((6, 5.0), 3, 0.7, boxstyle='round,pad=0.1',\n",
" facecolor='#C8E6C9', edgecolor='green', linewidth=2)\n",
"ax2.add_patch(rect4)\n",
"ax2.text(7.5, 5.35, '→ Detected: DOG (score=0.78)', ha='center', va='center', fontsize=9)\n",
"\n",
"plt.suptitle('Text Cross-Attention: Linking Queries to Text Categories', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('decoder_text_cross_attn.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Iterative Box Refinement \n",
"\n",
"### How Box Refinement Works\n",
"\n",
"At each decoder layer, the box is refined:\n",
"$$b^{l+1} = \\sigma(\\sigma^{-1}(b^l) + \\Delta b^l)$$\n",
"\n",
"where:\n",
"- $b^l$ = current box prediction (normalized)\n",
"- $\\Delta b^l$ = predicted offset from box head\n",
"- $\\sigma$ = sigmoid function\n",
"- $\\sigma^{-1}$ = inverse sigmoid (logit)\n",
"\n",
"### Why Inverse Sigmoid?\n",
"Working in logit space allows:\n",
"- Unconstrained offsets (can be any real number)\n",
"- Sigmoid ensures output stays in [0,1]\n",
"- Gradients flow better than direct sigmoid\n",
"\n",
"### Box Head Architecture\n",
"```\n",
"Query (256) → Linear(256) → ReLU → Linear(4) → Δbox\n",
"```\n",
"Separate box heads for each decoder layer (6 total)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize iterative box refinement\n",
"fig, axes = plt.subplots(1, 2, figsize=(16, 6))\n",
"\n",
"# Simulate box refinement over 6 layers\n",
"np.random.seed(42)\n",
"gt_box = np.array([0.5, 0.4, 0.3, 0.25]) # ground truth (xc, yc, w, h)\n",
"\n",
"# Initial anchor (from query selection)\n",
"anchor = np.array([0.45, 0.35, 0.4, 0.35]) # noisy initial\n",
"\n",
"boxes_over_layers = [anchor.copy()]\n",
"for layer in range(6):\n",
" # Simulate refinement: gradually approach GT\n",
" noise = np.random.randn(4) * 0.02 * (0.8 ** layer)\n",
" delta = (gt_box - boxes_over_layers[-1]) * 0.4 + noise\n",
" # Apply in logit space\n",
" current = boxes_over_layers[-1]\n",
" logit = np.log(current / (1 - current + 1e-8))\n",
" new_logit = logit + delta\n",
" new_box = 1 / (1 + np.exp(-new_logit))\n",
" boxes_over_layers.append(new_box)\n",
"\n",
"# Plot box evolution\n",
"ax1 = axes[0]\n",
"ax1.set_xlim(0, 1)\n",
"ax1.set_ylim(0, 1)\n",
"ax1.set_facecolor('#f8f8f8')\n",
"ax1.invert_yaxis()\n",
"\n",
"# Draw GT box\n",
"xc, yc, w, h = gt_box\n",
"gt_rect = patches.Rectangle((xc-w/2, yc-h/2), w, h, linewidth=3, \n",
" edgecolor='green', facecolor='lightgreen', alpha=0.3, label='GT box')\n",
"ax1.add_patch(gt_rect)\n",
"\n",
"# Draw boxes at each layer\n",
"colors = plt.cm.Reds(np.linspace(0.3, 1.0, 7))\n",
"for i, (box, color) in enumerate(zip(boxes_over_layers, colors)):\n",
" xc, yc, w, h = box\n",
" rect = patches.Rectangle((xc-w/2, yc-h/2), w, h, linewidth=2,\n",
" edgecolor=color, facecolor='none', \n",
" label=f'Layer {i}' if i > 0 else 'Anchor')\n",
" ax1.add_patch(rect)\n",
" ax1.plot(xc, yc, 'o', color=color, markersize=6)\n",
"\n",
"# Draw trajectory\n",
"centers = np.array([[b[0], b[1]] for b in boxes_over_layers])\n",
"ax1.plot(centers[:, 0], centers[:, 1], 'b--', linewidth=1.5, alpha=0.7)\n",
"\n",
"ax1.legend(loc='upper right', fontsize=8)\n",
"ax1.set_title('Iterative Box Refinement\\n(6 decoder layers)', fontsize=11, fontweight='bold')\n",
"ax1.set_xlabel('X center')\n",
"ax1.set_ylabel('Y center')\n",
"ax1.grid(True, alpha=0.3)\n",
"\n",
"# Plot IoU improvement\n",
"ax2 = axes[1]\n",
"\n",
"def compute_iou(box1, box2):\n",
" x1c, y1c, w1, h1 = box1\n",
" x2c, y2c, w2, h2 = box2\n",
" x1, y1, x2, y2 = x1c-w1/2, y1c-h1/2, x1c+w1/2, y1c+h1/2\n",
" x3, y3, x4, y4 = x2c-w2/2, y2c-h2/2, x2c+w2/2, y2c+h2/2\n",
" inter_x = max(0, min(x2, x4) - max(x1, x3))\n",
" inter_y = max(0, min(y2, y4) - max(y1, y3))\n",
" inter = inter_x * inter_y\n",
" union = w1*h1 + w2*h2 - inter\n",
" return inter / (union + 1e-8)\n",
"\n",
"ious = [compute_iou(b, gt_box) for b in boxes_over_layers]\n",
"layers = range(len(ious))\n",
"ax2.plot(layers, ious, 'b-o', linewidth=2, markersize=10)\n",
"ax2.fill_between(layers, ious, alpha=0.3)\n",
"ax2.set_xlabel('Decoder Layer')\n",
"ax2.set_ylabel('IoU with GT')\n",
"ax2.set_title('IoU Improvement Across Decoder Layers', fontsize=11, fontweight='bold')\n",
"ax2.set_xticks(layers)\n",
"ax2.set_xticklabels(['Anchor'] + [f'L{i+1}' for i in range(6)])\n",
"ax2.set_ylim(0, 1)\n",
"ax2.grid(True, alpha=0.3)\n",
"\n",
"for i, iou in enumerate(ious):\n",
" ax2.text(i, iou + 0.02, f'{iou:.2f}', ha='center', fontsize=9, fontweight='bold')\n",
"\n",
"plt.suptitle('Iterative Box Refinement in Cross-Modal Decoder', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('iterative_box_refinement.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Detection Head \n",
"\n",
"### Two Outputs\n",
"\n",
"**1. Box Regression Head**\n",
"$$\\Delta b = \\text{MLP}(q) \\in \\mathbb{R}^4$$\n",
"Applied at each decoder layer for iterative refinement.\n",
"\n",
"**2. Contrastive Score Head**\n",
"$$s = q \\cdot F_t^T \\in \\mathbb{R}^L$$\n",
"\n",
"The contrastive score is computed as the **dot product** between the query feature and each text token feature:\n",
"- High score at token $j$ → query matches word $j$\n",
"- Used for both detection confidence and phrase grounding\n",
"\n",
"### Final Score Computation\n",
"$$\\text{score}_i = \\max_j \\sigma(s_{ij})$$\n",
"\n",
"The detection confidence is the maximum score over all text tokens."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class DetectionHead(nn.Module):\n",
" \"\"\"Detection head for box regression and contrastive scoring.\"\"\"\n",
" \n",
" def __init__(self, d_model=256):\n",
" super().__init__()\n",
" # Box regression heads (one per decoder layer)\n",
" self.box_heads = nn.ModuleList([\n",
" nn.Sequential(\n",
" nn.Linear(d_model, d_model),\n",
" nn.ReLU(),\n",
" nn.Linear(d_model, 4)\n",
" ) for _ in range(6)\n",
" ])\n",
" \n",
" # Contrastive score projection\n",
" self.score_proj = nn.Linear(d_model, d_model)\n",
" \n",
" def compute_contrastive_scores(self, queries, txt_feat):\n",
" \"\"\"\n",
" queries: (B, K, D)\n",
" txt_feat: (B, L, D)\n",
" Returns: (B, K, L) - contrastive scores\n",
" \"\"\"\n",
" # Project queries\n",
" q_proj = self.score_proj(queries) # B, K, D\n",
" \n",
" # Normalize\n",
" q_norm = F.normalize(q_proj, dim=-1) # B, K, D\n",
" t_norm = F.normalize(txt_feat, dim=-1) # B, L, D\n",
" \n",
" # Dot product: (B, K, D) @ (B, D, L) → (B, K, L)\n",
" scores = torch.bmm(q_norm, t_norm.transpose(1, 2)) # B, K, L\n",
" \n",
" return scores\n",
" \n",
" def get_detection_confidence(self, contrastive_scores, txt_mask=None):\n",
" \"\"\"\n",
" contrastive_scores: (B, K, L)\n",
" txt_mask: (B, L) - 1 for real tokens\n",
" Returns: (B, K) - detection confidence\n",
" \"\"\"\n",
" scores = contrastive_scores # B, K, L\n",
" \n",
" if txt_mask is not None:\n",
" scores = scores.masked_fill(~txt_mask.bool().unsqueeze(1), float('-inf'))\n",
" \n",
" # Max over text tokens\n",
" confidence, _ = torch.sigmoid(scores).max(dim=-1) # B, K\n",
" return confidence\n",
"\n",
"# Test\n",
"B, K, L, D = 1, 50, 12, 256\n",
"head = DetectionHead(d_model=D)\n",
"\n",
"queries = torch.randn(B, K, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"txt_mask = torch.ones(B, L)\n",
"txt_mask[0, 10:] = 0\n",
"\n",
"# Box regression\n",
"delta_box = head.box_heads[0](queries) # B, K, 4\n",
"\n",
"# Contrastive scores\n",
"cont_scores = head.compute_contrastive_scores(queries, txt_feat) # B, K, L\n",
"confidence = head.get_detection_confidence(cont_scores, txt_mask) # B, K\n",
"\n",
"print(\"Detection Head:\")\n",
"print(f\" Input queries: {queries.shape} (B, K, D)\")\n",
"print(f\" Box delta: {delta_box.shape} (B, K, 4)\")\n",
"print(f\" Contrastive scores: {cont_scores.shape} (B, K, L)\")\n",
"print(f\" Detection confidence: {confidence.shape} (B, K)\")\n",
"print(f\"\\nScore statistics:\")\n",
"print(f\" Confidence range: [{confidence.min():.3f}, {confidence.max():.3f}]\")\n",
"print(f\" Mean confidence: {confidence.mean():.3f}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize contrastive scores\n",
"tokens = ['[CLS]', 'cat', '.', 'dog', '.', 'bird', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n",
"K_vis = 20\n",
"L_vis = 12\n",
"\n",
"# Simulate contrastive scores\n",
"np.random.seed(42)\n",
"scores_vis = np.random.rand(K_vis, L_vis) * 0.3\n",
"# Cat detections\n",
"scores_vis[:7, 1] = np.random.rand(7) * 0.5 + 0.5\n",
"# Dog detections\n",
"scores_vis[7:14, 3] = np.random.rand(7) * 0.5 + 0.5\n",
"# Bird detections\n",
"scores_vis[14:, 5] = np.random.rand(6) * 0.5 + 0.5\n",
"# Padding = 0\n",
"scores_vis[:, 7:] = 0\n",
"\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"# Contrastive score matrix\n",
"ax1 = axes[0]\n",
"im = ax1.imshow(scores_vis, cmap='hot', aspect='auto', vmin=0, vmax=1)\n",
"ax1.set_xticks(range(L_vis))\n",
"ax1.set_xticklabels(tokens, rotation=45, ha='right', fontsize=9)\n",
"ax1.set_yticks(range(K_vis))\n",
"ax1.set_yticklabels([f'Q{i}' for i in range(K_vis)], fontsize=8)\n",
"ax1.set_title('Contrastive Scores\\n(B, K, L) - query vs text token', fontsize=11, fontweight='bold')\n",
"plt.colorbar(im, ax=ax1)\n",
"\n",
"# Detection confidence (max over text tokens)\n",
"ax2 = axes[1]\n",
"confidence_vis = scores_vis[:, :7].max(axis=-1) # max over real tokens\n",
"ax2.barh(range(K_vis), confidence_vis, color=plt.cm.RdYlGn(confidence_vis))\n",
"ax2.axvline(x=0.3, color='red', linestyle='--', linewidth=2, label='Threshold=0.3')\n",
"ax2.set_xlabel('Detection Confidence')\n",
"ax2.set_ylabel('Query index')\n",
"ax2.set_title('Detection Confidence\\n(max over text tokens)', fontsize=11, fontweight='bold')\n",
"ax2.legend()\n",
"ax2.grid(True, alpha=0.3)\n",
"\n",
"# Phrase grounding: which token each detection corresponds to\n",
"ax3 = axes[2]\n",
"best_token = scores_vis[:, :7].argmax(axis=-1)\n",
"token_names = [tokens[t] for t in best_token]\n",
"colors_map = {'cat': 'red', 'dog': 'blue', 'bird': 'green', '.': 'gray', '[CLS]': 'gray', '[SEP]': 'gray'}\n",
"bar_colors = [colors_map.get(t, 'gray') for t in token_names]\n",
"ax3.barh(range(K_vis), confidence_vis, color=bar_colors, alpha=0.8)\n",
"ax3.set_xlabel('Detection Confidence')\n",
"ax3.set_ylabel('Query index')\n",
"ax3.set_title('Phrase Grounding\\n(Color = matched text token)', fontsize=11, fontweight='bold')\n",
"\n",
"from matplotlib.patches import Patch\n",
"legend_elements = [Patch(facecolor='red', label='cat'),\n",
" Patch(facecolor='blue', label='dog'),\n",
" Patch(facecolor='green', label='bird')]\n",
"ax3.legend(handles=legend_elements)\n",
"ax3.grid(True, alpha=0.3)\n",
"\n",
"plt.suptitle('Contrastive Scores and Phrase Grounding', fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('contrastive_scores.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Tensor Shapes "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"K = 900\n",
"Nv = 21950\n",
"L = 20\n",
"D = 256\n",
"H = 8\n",
"\n",
"shapes = [\n",
" ('Input Queries', f'1 × {K} × 256', 'From language-guided selection'),\n",
" ('Input Anchors', f'1 × {K} × 4', 'Initial anchor boxes'),\n",
" ('Image Features', f'1 × {Nv} × 256', 'From Feature Enhancer'),\n",
" ('Text Features', f'1 × {L} × 256', 'From Feature Enhancer'),\n",
" ('--- Decoder Layer (×6) ---', '', ''),\n",
" ('Self-Attn Q/K/V', f'1 × {H} × {K} × 32', f'{H} heads, d_k=32'),\n",
" ('Self-Attn weights', f'1 × {H} × {K} × {K}', 'Query-query attention'),\n",
" ('After Self-Attn', f'1 × {K} × 256', 'Queries updated'),\n",
" ('Img CrossAttn offsets', f'1 × {K} × {H} × 4 × 2', 'Deformable sampling'),\n",
" ('Img CrossAttn weights', f'1 × {K} × {H} × 4', 'Per-point weights'),\n",
" ('After Img CrossAttn', f'1 × {K} × 256', 'Visual evidence gathered'),\n",
" ('Txt CrossAttn Q', f'1 × {H} × {K} × 32', 'Query projections'),\n",
" ('Txt CrossAttn K/V', f'1 × {H} × {L} × 32', 'Text projections'),\n",
" ('Txt CrossAttn weights', f'1 × {H} × {K} × {L}', 'Query-text attention'),\n",
" ('After Txt CrossAttn', f'1 × {K} × 256', 'Text context gathered'),\n",
" ('FFN hidden', f'1 × {K} × 2048', '4× expansion'),\n",
" ('After FFN', f'1 × {K} × 256', 'Layer output'),\n",
" ('Box delta', f'1 × {K} × 4', 'Box refinement offset'),\n",
" ('Refined anchors', f'1 × {K} × 4', 'Updated anchor boxes'),\n",
" ('--- Final Output ---', '', ''),\n",
" ('Final queries', f'1 × {K} × 256', 'After 6 decoder layers'),\n",
" ('Contrastive scores', f'1 × {K} × {L}', 'Query-text similarity'),\n",
" ('Detection confidence', f'1 × {K}', 'Max over text tokens'),\n",
" ('Final boxes', f'1 × {K} × 4', 'Refined bounding boxes'),\n",
"]\n",
"\n",
"df = pd.DataFrame(shapes, columns=['Component', 'Shape', 'Description'])\n",
"\n",
"fig, ax = plt.subplots(figsize=(18, 12))\n",
"ax.axis('off')\n",
"table = ax.table(cellText=df.values, colLabels=df.columns, cellLoc='left', loc='center')\n",
"table.auto_set_font_size(False)\n",
"table.set_fontsize(8)\n",
"table.scale(1, 1.8)\n",
"\n",
"for j in range(3):\n",
" table[0, j].set_facecolor('#1565C0')\n",
" table[0, j].set_text_props(color='white', fontweight='bold')\n",
"\n",
"for i in range(1, len(shapes)+1):\n",
" row = shapes[i-1]\n",
" if '---' in row[0]:\n",
" for j in range(3):\n",
" table[i, j].set_facecolor('#37474F')\n",
" table[i, j].set_text_props(color='white', fontweight='bold')\n",
" elif 'Self' in row[0]:\n",
" for j in range(3): table[i, j].set_facecolor('#C8E6C9')\n",
" elif 'Img' in row[0]:\n",
" for j in range(3): table[i, j].set_facecolor('#BBDEFB')\n",
" elif 'Txt' in row[0] or 'Text' in row[0]:\n",
" for j in range(3): table[i, j].set_facecolor('#FFF9C4')\n",
" elif 'FFN' in row[0]:\n",
" for j in range(3): table[i, j].set_facecolor('#F8BBD0')\n",
" elif 'Box' in row[0] or 'anchor' in row[0].lower():\n",
" for j in range(3): table[i, j].set_facecolor('#FFCC80')\n",
" elif 'Final' in row[0] or 'Contrastive' in row[0] or 'Detection' in row[0]:\n",
" for j in range(3): table[i, j].set_facecolor('#E8F5E9')\n",
"\n",
"plt.title('Cross-Modal Decoder Tensor Shapes', fontsize=13, fontweight='bold', pad=20)\n",
"plt.tight_layout()\n",
"plt.savefig('decoder_tensor_shapes.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"The Cross-Modal Decoder in GroundingDINO:\n",
"\n",
"1. **Takes** 900 language-guided queries + image features + text features\n",
"2. **Applies** 6 layers of: Self-Attn → Image Cross-Attn → **Text Cross-Attn** → FFN\n",
"3. **Refines** bounding boxes iteratively at each layer\n",
"4. **Outputs** contrastive scores `(B, 900, L)` and final boxes `(B, 900, 4)`\n",
"\n",
"The **text cross-attention** at every decoder layer is GroundingDINO's key innovation — it ensures queries continuously align with the text query throughout the decoding process.\n",
"\n",
"**Next**: [Notebook 07 - Loss Functions](07_loss_functions.ipynb)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}