{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GroundingDINO: Language-Guided Query Selection\n",
"\n",
"## Notebook 05 of 14\n",
"\n",
"This notebook covers **Language-Guided Query Selection** — Stage 2 of fusion in GroundingDINO:\n",
"- What queries are and why they matter\n",
"- How text guides the selection of visual queries\n",
"- Comparison with DINO's fixed queries\n",
"- Tensor shapes and implementation\n",
"- Visualization of selected queries\n",
"\n",
"---\n",
"\n",
"## Table of Contents\n",
"1. [What are Queries in DETR-based Detectors?](#what-are-queries)\n",
"2. [DINO's Anchor-Based Queries](#dino-queries)\n",
"3. [Language-Guided Query Selection](#lgqs)\n",
"4. [Mixed Query Selection](#mixed-query)\n",
"5. [Tensor Shapes](#tensor-shapes)\n",
"6. [Implementation & Visualization](#implementation)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. What are Queries in DETR-based Detectors? \n",
"\n",
"In DETR-style detectors, **queries** are learnable vectors that the decoder uses to \"ask\" about specific regions of the image.\n",
"\n",
"### DETR Query Mechanism\n",
"```\n",
"N learnable queries (N=300)\n",
" ↓\n",
"Decoder (cross-attends to image features)\n",
" ↓\n",
"N detection predictions (box + class)\n",
"```\n",
"\n",
"### Problem with Fixed Queries\n",
"In standard DETR/DINO:\n",
"- Queries are **fixed** and **text-agnostic**\n",
"- All N queries are used regardless of what text is queried\n",
"- Queries don't know what to look for\n",
"\n",
"### GroundingDINO's Solution\n",
"Select queries that are **relevant to the text query**:\n",
"- Use text features to score image regions\n",
"- Select top-K image regions as queries\n",
"- These queries already \"know\" what to look for\n",
"\n",
"### Why This Matters\n",
"```\n",
"Text: \"cat\"\n",
"→ Select image regions that look like cats\n",
"→ Decoder refines these cat-like regions\n",
"→ Better detection of cats!\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.patches as patches\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"print(f\"Using device: {device}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. DINO's Anchor-Based Queries \n",
"\n",
"### DINO Query Types\n",
"DINO uses two types of queries:\n",
"\n",
"1. **Content queries** $c_q \\in \\mathbb{R}^{256}$: Learnable content embeddings\n",
"2. **Positional queries** (anchors) $a_q \\in \\mathbb{R}^4$: Reference box positions $(x_c, y_c, w, h)$\n",
"\n",
"### Query Initialization in DINO\n",
"```\n",
"Method 1: Learned queries (fixed, text-agnostic)\n",
"Method 2: Mixed query selection (from image features)\n",
"```\n",
"\n",
"### GroundingDINO Extension\n",
"GroundingDINO extends DINO's mixed query selection to be **language-guided**:\n",
"- Score image features by their similarity to text features\n",
"- Select top-K image features as initial queries\n",
"- Use their positions as anchor boxes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize query types\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"# DETR: Fixed learned queries\n",
"ax1 = axes[0]\n",
"ax1.set_xlim(0, 10)\n",
"ax1.set_ylim(0, 10)\n",
"ax1.set_facecolor('#f8f8f8')\n",
"\n",
"# Show fixed query positions (random, text-agnostic)\n",
"np.random.seed(42)\n",
"query_x = np.random.rand(20) * 8 + 1\n",
"query_y = np.random.rand(20) * 8 + 1\n",
"ax1.scatter(query_x, query_y, c='blue', s=100, zorder=5, label='Fixed queries')\n",
"ax1.set_title('DETR: Fixed Learned Queries\\n(Text-agnostic, always same)', fontsize=11, fontweight='bold')\n",
"ax1.set_xlabel('X position')\n",
"ax1.set_ylabel('Y position')\n",
"ax1.legend()\n",
"ax1.grid(True, alpha=0.3)\n",
"ax1.text(5, 0.5, 'Same queries regardless of text!', ha='center', fontsize=9, color='red', style='italic')\n",
"\n",
"# DINO: Mixed query selection (image-based)\n",
"ax2 = axes[1]\n",
"ax2.set_xlim(0, 10)\n",
"ax2.set_ylim(0, 10)\n",
"ax2.set_facecolor('#f8f8f8')\n",
"\n",
"# Show image feature scores\n",
"H, W = 20, 20\n",
"scores = np.random.rand(H, W)\n",
"# Add some high-score regions\n",
"scores[5:8, 3:6] = 0.9 # object 1\n",
"scores[12:16, 13:17] = 0.85 # object 2\n",
"\n",
"im = ax2.imshow(scores, cmap='hot', extent=[0, 10, 0, 10], origin='lower', alpha=0.6)\n",
"# Top-K selected\n",
"flat_idx = np.argsort(scores.flatten())[-10:]\n",
"sel_y = (flat_idx // W) / W * 10\n",
"sel_x = (flat_idx % W) / W * 10\n",
"ax2.scatter(sel_x, sel_y, c='cyan', s=150, zorder=5, marker='*', label='Top-K selected')\n",
"ax2.set_title('DINO: Mixed Query Selection\\n(From image features, text-agnostic)', fontsize=11, fontweight='bold')\n",
"ax2.legend()\n",
"ax2.text(5, 0.5, 'Selects salient regions, but ignores text', ha='center', fontsize=9, color='orange', style='italic')\n",
"\n",
"# GroundingDINO: Language-guided query selection\n",
"ax3 = axes[2]\n",
"ax3.set_xlim(0, 10)\n",
"ax3.set_ylim(0, 10)\n",
"ax3.set_facecolor('#f8f8f8')\n",
"\n",
"# Text: \"cat\"\n",
"# Scores are high only where cats are\n",
"scores_guided = np.random.rand(H, W) * 0.2\n",
"scores_guided[5:8, 3:6] = 0.95 # cat region\n",
"scores_guided[12:16, 13:17] = 0.1 # dog region (low score for \"cat\" query)\n",
"\n",
"im3 = ax3.imshow(scores_guided, cmap='hot', extent=[0, 10, 0, 10], origin='lower', alpha=0.6)\n",
"flat_idx3 = np.argsort(scores_guided.flatten())[-5:]\n",
"sel_y3 = (flat_idx3 // W) / W * 10\n",
"sel_x3 = (flat_idx3 % W) / W * 10\n",
"ax3.scatter(sel_x3, sel_y3, c='lime', s=200, zorder=5, marker='*', label='Top-K (cat regions)')\n",
"ax3.set_title('GroundingDINO: Language-Guided\\nQuery Selection (Text: \"cat\")', fontsize=11, fontweight='bold')\n",
"ax3.legend()\n",
"ax3.text(5, 0.5, '✓ Selects regions relevant to text!', ha='center', fontsize=9, color='green', style='italic')\n",
"\n",
"plt.suptitle('Query Selection Strategies Comparison', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('query_selection_comparison.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Language-Guided Query Selection \n",
"\n",
"### Algorithm\n",
"\n",
"Given:\n",
"- Enhanced image features $F_v \\in \\mathbb{R}^{B \\times N_v \\times 256}$ (from Feature Enhancer)\n",
"- Enhanced text features $F_t \\in \\mathbb{R}^{B \\times L \\times 256}$ (from Feature Enhancer)\n",
"\n",
"**Step 1**: Compute image-text similarity scores\n",
"$$S = F_v \\cdot F_t^T \\in \\mathbb{R}^{B \\times N_v \\times L}$$\n",
"\n",
"**Step 2**: Aggregate over text tokens (max or mean)\n",
"$$s_i = \\max_j S_{ij} \\in \\mathbb{R}^{B \\times N_v}$$\n",
"\n",
"**Step 3**: Select top-K image features\n",
"$$\\text{idx} = \\text{TopK}(s, K=900)$$\n",
"\n",
"**Step 4**: Use selected features as initial queries\n",
"$$Q_{init} = F_v[\\text{idx}] \\in \\mathbb{R}^{B \\times K \\times 256}$$\n",
"\n",
"**Step 5**: Use positions of selected features as anchor boxes\n",
"$$A_{init} = \\text{pos}[\\text{idx}] \\in \\mathbb{R}^{B \\times K \\times 4}$$\n",
"\n",
"### Key Parameters\n",
"- $K = 900$ queries (same as DINO)\n",
"- Positions are normalized $(x_c, y_c, w, h) \\in [0,1]^4$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LanguageGuidedQuerySelection(nn.Module):\n",
" \"\"\"Language-Guided Query Selection module.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, num_queries=900):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" self.num_queries = num_queries\n",
" \n",
" # Linear projection for scoring\n",
" self.score_proj = nn.Linear(d_model, d_model)\n",
" \n",
" def forward(self, img_feat, txt_feat, img_positions, txt_mask=None):\n",
" \"\"\"\n",
" img_feat: (B, Nv, D) - enhanced image features\n",
" txt_feat: (B, L, D) - enhanced text features\n",
" img_positions: (B, Nv, 4) - normalized positions (xc, yc, w, h)\n",
" txt_mask: (B, L) - 1 for real tokens\n",
" \n",
" Returns:\n",
" queries: (B, K, D) - selected image features as queries\n",
" anchors: (B, K, 4) - corresponding anchor boxes\n",
" topk_idx: (B, K) - indices of selected features\n",
" scores: (B, Nv) - relevance scores\n",
" \"\"\"\n",
" B, Nv, D = img_feat.shape\n",
" B, L, D = txt_feat.shape\n",
" K = self.num_queries\n",
" \n",
" # Project image features\n",
" img_proj = self.score_proj(img_feat) # B, Nv, D\n",
" \n",
" # Normalize for cosine similarity\n",
" img_norm = F.normalize(img_proj, dim=-1) # B, Nv, D\n",
" txt_norm = F.normalize(txt_feat, dim=-1) # B, L, D\n",
" \n",
" # Compute similarity: (B, Nv, D) @ (B, D, L) → (B, Nv, L)\n",
" similarity = torch.bmm(img_norm, txt_norm.transpose(1, 2)) # B, Nv, L\n",
" \n",
" # Mask padding tokens\n",
" if txt_mask is not None:\n",
" # txt_mask: B, L → B, 1, L\n",
" similarity = similarity.masked_fill(~txt_mask.bool().unsqueeze(1), float('-inf'))\n",
" \n",
" # Aggregate over text tokens (max pooling)\n",
" scores, _ = similarity.max(dim=-1) # B, Nv\n",
" scores = torch.nan_to_num(scores, nan=0.0)\n",
" \n",
" # Select top-K\n",
" K_actual = min(K, Nv)\n",
" topk_scores, topk_idx = scores.topk(K_actual, dim=-1) # B, K\n",
" \n",
" # Gather selected features\n",
" # topk_idx: B, K → expand to B, K, D\n",
" idx_expanded = topk_idx.unsqueeze(-1).expand(-1, -1, D) # B, K, D\n",
" queries = torch.gather(img_feat, 1, idx_expanded) # B, K, D\n",
" \n",
" # Gather anchor positions\n",
" idx_pos = topk_idx.unsqueeze(-1).expand(-1, -1, 4) # B, K, 4\n",
" anchors = torch.gather(img_positions, 1, idx_pos) # B, K, 4\n",
" \n",
" return queries, anchors, topk_idx, scores\n",
"\n",
"# Test\n",
"B = 1\n",
"Nv = 500 # simplified\n",
"L = 12\n",
"D = 256\n",
"K = 100 # simplified (normally 900)\n",
"\n",
"lgqs = LanguageGuidedQuerySelection(d_model=D, num_queries=K)\n",
"\n",
"img_feat = torch.randn(B, Nv, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"# Create positions: each visual token has a position\n",
"img_positions = torch.rand(B, Nv, 4) # normalized (xc, yc, w, h)\n",
"txt_mask = torch.ones(B, L)\n",
"txt_mask[0, 10:] = 0 # padding\n",
"\n",
"queries, anchors, topk_idx, scores = lgqs(img_feat, txt_feat, img_positions, txt_mask)\n",
"\n",
"print(\"Language-Guided Query Selection:\")\n",
"print(f\" Image features input: {img_feat.shape} (B, Nv, D)\")\n",
"print(f\" Text features input: {txt_feat.shape} (B, L, D)\")\n",
"print(f\" Image positions: {img_positions.shape} (B, Nv, 4)\")\n",
"print(f\" Similarity scores: {scores.shape} (B, Nv)\")\n",
"print(f\" Top-K indices: {topk_idx.shape} (B, K)\")\n",
"print(f\" Selected queries: {queries.shape} (B, K, D)\")\n",
"print(f\" Anchor boxes: {anchors.shape} (B, K, 4)\")\n",
"print(f\"\\nScore statistics:\")\n",
"print(f\" Min: {scores.min():.3f}, Max: {scores.max():.3f}, Mean: {scores.mean():.3f}\")\n",
"print(f\" Top-5 scores: {scores[0].topk(5).values.tolist()}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Visualize query selection process\n",
"fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n",
"\n",
"# Simulate a 2D feature map\n",
"H, W = 25, 42 # stride-32 feature map for 800×1333\n",
"Nv = H * W\n",
"D = 256\n",
"L = 8\n",
"K = 50\n",
"\n",
"torch.manual_seed(42)\n",
"img_feat_2d = torch.randn(1, Nv, D)\n",
"txt_feat_2d = torch.randn(1, L, D)\n",
"\n",
"# Create positions on a grid\n",
"ys = torch.linspace(0.02, 0.98, H)\n",
"xs = torch.linspace(0.02, 0.98, W)\n",
"grid_y, grid_x = torch.meshgrid(ys, xs, indexing='ij')\n",
"positions = torch.stack([\n",
" grid_x.flatten(),\n",
" grid_y.flatten(),\n",
" torch.full((Nv,), 1.0/W), # width\n",
" torch.full((Nv,), 1.0/H), # height\n",
"], dim=-1).unsqueeze(0) # 1, Nv, 4\n",
"\n",
"lgqs_2d = LanguageGuidedQuerySelection(d_model=D, num_queries=K)\n",
"queries_2d, anchors_2d, topk_idx_2d, scores_2d = lgqs_2d(img_feat_2d, txt_feat_2d, positions)\n",
"\n",
"# Plot 1: Score map\n",
"ax = axes[0, 0]\n",
"score_map = scores_2d[0].detach().numpy().reshape(H, W)\n",
"im = ax.imshow(score_map, cmap='hot', aspect='equal')\n",
"ax.set_title(f'Image-Text Similarity Scores\\n{H}×{W} feature map', fontsize=11, fontweight='bold')\n",
"plt.colorbar(im, ax=ax)\n",
"\n",
"# Plot 2: Top-K selected positions\n",
"ax = axes[0, 1]\n",
"ax.imshow(score_map, cmap='hot', aspect='equal', alpha=0.5)\n",
"# Mark selected positions\n",
"sel_idx = topk_idx_2d[0].numpy()\n",
"sel_y_2d = sel_idx // W\n",
"sel_x_2d = sel_idx % W\n",
"ax.scatter(sel_x_2d, sel_y_2d, c='cyan', s=50, zorder=5, marker='*', label=f'Top-{K} queries')\n",
"ax.set_title(f'Top-{K} Selected Queries\\n(Language-guided)', fontsize=11, fontweight='bold')\n",
"ax.legend()\n",
"\n",
"# Plot 3: Anchor boxes\n",
"ax = axes[0, 2]\n",
"ax.set_xlim(0, 1)\n",
"ax.set_ylim(0, 1)\n",
"ax.set_facecolor('#f0f0f0')\n",
"# Draw top-20 anchor boxes\n",
"for i in range(min(20, K)):\n",
" xc, yc, w, h = anchors_2d[0, i].detach().numpy()\n",
" rect = patches.Rectangle((xc - w/2, yc - h/2), w, h,\n",
" linewidth=1, edgecolor='blue', facecolor='none', alpha=0.7)\n",
" ax.add_patch(rect)\n",
"ax.set_title(f'Top-20 Anchor Boxes\\n(Initial positions for decoder)', fontsize=11, fontweight='bold')\n",
"ax.set_xlabel('X (normalized)')\n",
"ax.set_ylabel('Y (normalized)')\n",
"ax.invert_yaxis()\n",
"\n",
"# Plot 4: Score distribution\n",
"ax = axes[1, 0]\n",
"all_scores = scores_2d[0].detach().numpy()\n",
"ax.hist(all_scores, bins=50, color='blue', alpha=0.7, edgecolor='black')\n",
"threshold = np.sort(all_scores)[-K]\n",
"ax.axvline(x=threshold, color='red', linestyle='--', linewidth=2, label=f'Top-{K} threshold')\n",
"ax.set_xlabel('Similarity Score')\n",
"ax.set_ylabel('Count')\n",
"ax.set_title('Score Distribution\\n(Red line = selection threshold)', fontsize=11, fontweight='bold')\n",
"ax.legend()\n",
"ax.grid(True, alpha=0.3)\n",
"\n",
"# Plot 5: Query feature norms\n",
"ax = axes[1, 1]\n",
"query_norms = queries_2d[0].norm(dim=-1).detach().numpy()\n",
"ax.bar(range(K), np.sort(query_norms)[::-1], color='green', alpha=0.7)\n",
"ax.set_xlabel('Query index (sorted by norm)')\n",
"ax.set_ylabel('Feature norm')\n",
"ax.set_title(f'Selected Query Feature Norms\\n({K} queries)', fontsize=11, fontweight='bold')\n",
"ax.grid(True, alpha=0.3)\n",
"\n",
"# Plot 6: Anchor position distribution\n",
"ax = axes[1, 2]\n",
"anchor_x = anchors_2d[0, :, 0].detach().numpy()\n",
"anchor_y = anchors_2d[0, :, 1].detach().numpy()\n",
"ax.scatter(anchor_x, anchor_y, c=np.arange(K), cmap='viridis', s=30, alpha=0.7)\n",
"ax.set_xlim(0, 1)\n",
"ax.set_ylim(0, 1)\n",
"ax.set_xlabel('X center (normalized)')\n",
"ax.set_ylabel('Y center (normalized)')\n",
"ax.set_title(f'Anchor Box Centers\\n({K} selected queries)', fontsize=11, fontweight='bold')\n",
"ax.invert_yaxis()\n",
"ax.grid(True, alpha=0.3)\n",
"\n",
"plt.suptitle('Language-Guided Query Selection Visualization', fontsize=14, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('lgqs_visualization.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Mixed Query Selection \n",
"\n",
"### Two Types of Queries in GroundingDINO\n",
"\n",
"GroundingDINO uses **mixed query selection** (from DINO):\n",
"\n",
"1. **Content queries** (dynamic): Selected from image features\n",
" - $c_q = F_v[\\text{topk\\_idx}]$ — image features at selected positions\n",
" - These carry semantic content about what's at each location\n",
"\n",
"2. **Positional queries** (anchors): Positions of selected features\n",
" - $a_q = \\text{pos}[\\text{topk\\_idx}]$ — 4D box coordinates\n",
" - These tell the decoder where to look\n",
"\n",
"### Anchor Box Encoding\n",
"Anchor boxes are encoded as $(x_c, y_c, w, h)$ in normalized coordinates.\n",
"They are converted to positional embeddings using sinusoidal encoding:\n",
"\n",
"$$\\text{PE}(a) = \\text{MLP}(\\text{SinEmbed}(a))$$\n",
"\n",
"### Denoising Training\n",
"During training, DINO adds **noisy ground-truth boxes** as additional queries for denoising training:\n",
"- Adds Gaussian noise to GT box positions\n",
"- Adds random label noise\n",
"- Forces the model to denoise → better training signal"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class SinusoidalPositionEmbedding(nn.Module):\n",
" \"\"\"Sinusoidal position embedding for anchor boxes.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, temperature=10000):\n",
" super().__init__()\n",
" self.d_model = d_model\n",
" self.temperature = temperature\n",
" \n",
" def forward(self, x):\n",
" \"\"\"\n",
" x: (..., 4) - normalized box coordinates (xc, yc, w, h)\n",
" Returns: (..., d_model)\n",
" \"\"\"\n",
" assert x.shape[-1] == 4\n",
" d = self.d_model // 4 # 64 per coordinate\n",
" \n",
" dim_t = torch.arange(d, dtype=torch.float32, device=x.device)\n",
" dim_t = self.temperature ** (2 * (dim_t // 2) / d)\n",
" \n",
" # Encode each coordinate\n",
" pos_embeds = []\n",
" for i in range(4):\n",
" coord = x[..., i:i+1] # ..., 1\n",
" pos = coord / dim_t # ..., d\n",
" pos_embed = torch.stack([pos[..., 0::2].sin(), pos[..., 1::2].cos()], dim=-1)\n",
" pos_embed = pos_embed.flatten(-2) # ..., d\n",
" pos_embeds.append(pos_embed)\n",
" \n",
" return torch.cat(pos_embeds, dim=-1) # ..., d_model\n",
"\n",
"class MixedQuerySelection(nn.Module):\n",
" \"\"\"Mixed query selection combining content and positional queries.\"\"\"\n",
" \n",
" def __init__(self, d_model=256, num_queries=900):\n",
" super().__init__()\n",
" self.lgqs = LanguageGuidedQuerySelection(d_model, num_queries)\n",
" self.pos_embed = SinusoidalPositionEmbedding(d_model)\n",
" self.anchor_mlp = nn.Sequential(\n",
" nn.Linear(d_model, d_model),\n",
" nn.ReLU(),\n",
" nn.Linear(d_model, d_model)\n",
" )\n",
" \n",
" def forward(self, img_feat, txt_feat, img_positions, txt_mask=None):\n",
" # Language-guided selection\n",
" content_queries, anchors, topk_idx, scores = self.lgqs(\n",
" img_feat, txt_feat, img_positions, txt_mask\n",
" )\n",
" \n",
" # Positional queries from anchor boxes\n",
" pos_embed = self.pos_embed(anchors) # B, K, d_model\n",
" pos_queries = self.anchor_mlp(pos_embed) # B, K, d_model\n",
" \n",
" return content_queries, pos_queries, anchors, topk_idx, scores\n",
"\n",
"# Test\n",
"B, Nv, L, D, K = 1, 500, 12, 256, 100\n",
"mqs = MixedQuerySelection(d_model=D, num_queries=K)\n",
"\n",
"img_feat = torch.randn(B, Nv, D)\n",
"txt_feat = torch.randn(B, L, D)\n",
"img_positions = torch.rand(B, Nv, 4)\n",
"txt_mask = torch.ones(B, L)\n",
"\n",
"content_q, pos_q, anchors, idx, scores = mqs(img_feat, txt_feat, img_positions, txt_mask)\n",
"\n",
"print(\"Mixed Query Selection:\")\n",
"print(f\" Content queries: {content_q.shape} (B, K, D) - from image features\")\n",
"print(f\" Positional queries: {pos_q.shape} (B, K, D) - from anchor positions\")\n",
"print(f\" Anchor boxes: {anchors.shape} (B, K, 4) - (xc, yc, w, h)\")\n",
"print(f\" Selected indices: {idx.shape} (B, K)\")\n",
"print(f\" Relevance scores: {scores.shape} (B, Nv)\")\n",
"print(f\"\\nDecoder input:\")\n",
"print(f\" Query = content_q + pos_q: {content_q.shape}\")\n",
"print(f\" Initial anchor: {anchors.shape}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Tensor Shapes "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"Nv = 21950 # 800×1333 input\n",
"L = 20\n",
"D = 256\n",
"K = 900\n",
"\n",
"shapes = [\n",
" ('Enhanced Image Features', f'1 × {Nv} × 256', 'From Feature Enhancer'),\n",
" ('Enhanced Text Features', f'1 × {L} × 256', 'From Feature Enhancer'),\n",
" ('Image Projection', f'1 × {Nv} × 256', 'Linear projection for scoring'),\n",
" ('Normalized Image Features', f'1 × {Nv} × 256', 'L2 normalized'),\n",
" ('Normalized Text Features', f'1 × {L} × 256', 'L2 normalized'),\n",
" ('Similarity Matrix', f'1 × {Nv} × {L}', 'Image-text cosine similarity'),\n",
" ('Max-pooled Scores', f'1 × {Nv}', 'Max over text tokens'),\n",
" ('Top-K Indices', f'1 × {K}', 'Indices of top-900 features'),\n",
" ('Content Queries', f'1 × {K} × 256', 'Selected image features'),\n",
" ('Anchor Boxes', f'1 × {K} × 4', '(xc, yc, w, h) normalized'),\n",
" ('Sinusoidal Pos Embed', f'1 × {K} × 256', '64 dims per coordinate'),\n",
" ('Positional Queries', f'1 × {K} × 256', 'MLP(pos_embed)'),\n",
" ('Final Queries (content)', f'1 × {K} × 256', 'Input to decoder'),\n",
" ('Final Queries (pos)', f'1 × {K} × 256', 'Added to decoder queries'),\n",
"]\n",
"\n",
"df = pd.DataFrame(shapes, columns=['Component', 'Shape', 'Description'])\n",
"\n",
"fig, ax = plt.subplots(figsize=(16, 7))\n",
"ax.axis('off')\n",
"table = ax.table(cellText=df.values, colLabels=df.columns, cellLoc='left', loc='center')\n",
"table.auto_set_font_size(False)\n",
"table.set_fontsize(9)\n",
"table.scale(1, 2.2)\n",
"\n",
"for j in range(3):\n",
" table[0, j].set_facecolor('#1565C0')\n",
" table[0, j].set_text_props(color='white', fontweight='bold')\n",
"\n",
"colors = ['#E3F2FD', '#FFFDE7', '#E3F2FD', '#E3F2FD', '#FFFDE7',\n",
" '#F3E5F5', '#F3E5F5', '#E8F5E9', '#C8E6C9', '#C8E6C9',\n",
" '#FFF9C4', '#FFF9C4', '#FFCC80', '#FFCC80']\n",
"for i in range(1, len(shapes)+1):\n",
" for j in range(3):\n",
" table[i, j].set_facecolor(colors[i-1])\n",
"\n",
"plt.title('Language-Guided Query Selection: Tensor Shapes', fontsize=13, fontweight='bold', pad=20)\n",
"plt.tight_layout()\n",
"plt.savefig('lgqs_tensor_shapes.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Implementation & Visualization "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Demonstrate effect of different text queries on selection\n",
"fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"H, W = 20, 30\n",
"Nv = H * W\n",
"D = 64 # smaller for visualization\n",
"K = 30\n",
"\n",
"torch.manual_seed(42)\n",
"\n",
"# Create structured image features\n",
"# Region 1 (top-left): \"cat-like\" features\n",
"# Region 2 (bottom-right): \"dog-like\" features\n",
"img_feat_struct = torch.randn(1, Nv, D) * 0.1\n",
"cat_region = torch.arange(H*W).reshape(H, W)[:H//2, :W//2].flatten()\n",
"dog_region = torch.arange(H*W).reshape(H, W)[H//2:, W//2:].flatten()\n",
"\n",
"# Cat features: high in first D//2 dims\n",
"img_feat_struct[0, cat_region, :D//2] += 2.0\n",
"# Dog features: high in last D//2 dims\n",
"img_feat_struct[0, dog_region, D//2:] += 2.0\n",
"\n",
"# Text features for different queries\n",
"cat_txt = torch.zeros(1, 1, D)\n",
"cat_txt[0, 0, :D//2] = 2.0 # cat query\n",
"\n",
"dog_txt = torch.zeros(1, 1, D)\n",
"dog_txt[0, 0, D//2:] = 2.0 # dog query\n",
"\n",
"all_txt = torch.ones(1, 1, D) * 0.5 # all objects query\n",
"\n",
"# Create positions\n",
"ys = torch.linspace(0, 1, H)\n",
"xs = torch.linspace(0, 1, W)\n",
"gy, gx = torch.meshgrid(ys, xs, indexing='ij')\n",
"pos = torch.stack([gx.flatten(), gy.flatten(), \n",
" torch.full((Nv,), 1/W), torch.full((Nv,), 1/H)], dim=-1).unsqueeze(0)\n",
"\n",
"lgqs_small = LanguageGuidedQuerySelection(d_model=D, num_queries=K)\n",
"\n",
"for ax, txt_feat, title in zip(axes, [cat_txt, dog_txt, all_txt], \n",
" ['Query: \"cat\"', 'Query: \"dog\"', 'Query: \"all objects\"']):\n",
" _, _, topk_idx, scores = lgqs_small(img_feat_struct, txt_feat, pos)\n",
" \n",
" score_map = scores[0].detach().numpy().reshape(H, W)\n",
" ax.imshow(score_map, cmap='hot', aspect='equal')\n",
" \n",
" sel_idx = topk_idx[0].numpy()\n",
" sel_y = sel_idx // W\n",
" sel_x = sel_idx % W\n",
" ax.scatter(sel_x, sel_y, c='cyan', s=80, zorder=5, marker='*')\n",
" \n",
" # Draw region boundaries\n",
" ax.axhline(y=H//2 - 0.5, color='white', linewidth=2, linestyle='--')\n",
" ax.axvline(x=W//2 - 0.5, color='white', linewidth=2, linestyle='--')\n",
" ax.text(W//4, H//4, 'CAT\\nregion', ha='center', va='center', \n",
" fontsize=10, color='white', fontweight='bold')\n",
" ax.text(3*W//4, 3*H//4, 'DOG\\nregion', ha='center', va='center', \n",
" fontsize=10, color='white', fontweight='bold')\n",
" ax.set_title(f'{title}\\n(cyan stars = selected queries)', fontsize=11, fontweight='bold')\n",
" ax.axis('off')\n",
"\n",
"plt.suptitle('Language-Guided Query Selection: Different Text Queries → Different Selections', \n",
" fontsize=13, fontweight='bold')\n",
"plt.tight_layout()\n",
"plt.savefig('lgqs_different_queries.png', dpi=150, bbox_inches='tight')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Language-Guided Query Selection in GroundingDINO:\n",
"\n",
"1. **Computes** image-text similarity: `(B, Nv, 256) × (B, 256, L) → (B, Nv, L)`\n",
"2. **Aggregates** over text tokens: `(B, Nv, L) → (B, Nv)` via max-pooling\n",
"3. **Selects** top-K=900 image features by score\n",
"4. **Outputs** content queries `(B, 900, 256)` and anchor boxes `(B, 900, 4)`\n",
"\n",
"This is **Stage 2 of 3** fusion stages — the queries are now text-aware!\n",
"\n",
"**Next**: [Notebook 06 - Cross-Modal Decoder](06_cross_modal_decoder.ipynb)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}