{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "cells": [ { "cell_type": "markdown", "id": "title_md", "metadata": {}, "source": [ "# RF-DETR: Advanced Features & Visualization\n", "\n", "**Series:** RF-DETR Detailed Tutorial | **Notebook 16e of 5**\n", "\n", "---\n", "\n", "## What we'll cover\n", "\n", "1. **Backbone attention maps** — DINOv2 windowed vs full attention inside RF-DETR\n", "2. **Decoder cross-attention** — which image regions each query attends to\n", "3. **Query analysis** — how object queries evolve through decoder layers\n", "4. **Multi-scale feature visualization** — backbone outputs at stages 2, 5, 8, 11\n", "5. **Deformable attention reference points** — where the decoder samples\n", "6. **Comparing DINOv2 standalone vs RF-DETR backbone features**\n", "7. **ONNX export** for deployment\n", "\n", "---\n", "\n", "**Model path:** `/mnt/mydata/huggingface/rf-detr-base`" ] }, { "cell_type": "markdown", "id": "setup_md", "metadata": {}, "source": [ "## 1. Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "setup_code", "metadata": {}, "outputs": [], "source": [ "import sys, os\n", "sys.path.insert(0, os.path.abspath('../../..'))\n", "\n", "import torch\n", "import torch.nn.functional as F\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as patches\n", "from PIL import Image\n", "from transformers import AutoImageProcessor, RfDetrForObjectDetection\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "MODEL_PATH = '/mnt/mydata/huggingface/rf-detr-base'\n", "IMG_PATH = '../../../vla_taskgraph/output/images/real_robot_image_518.jpg'\n", "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", "print(f'Device: {device}')\n", "\n", "# Load model and processor\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = RfDetrForObjectDetection.from_pretrained(MODEL_PATH, output_attentions=True)\n", "model = model.to(device)\n", "model.eval()\n", "print(f'✅ RF-DETR-base loaded')\n", "\n", "# Load image\n", "img = Image.open(IMG_PATH).convert('RGB')\n", "img_np = np.array(img)\n", "H, W = img_np.shape[:2]\n", "print(f'Image: {W}×{H}')\n", "\n", "# Preprocess\n", "inputs = processor(images=img, return_tensors='pt')\n", "inputs = {k: v.to(device) for k, v in inputs.items()}\n", "\n", "# Forward pass with attention outputs\n", "with torch.no_grad():\n", " outputs = model(**inputs, output_attentions=True)\n", "\n", "print(f'\\nOutputs available:')\n", "for key in dir(outputs):\n", " if not key.startswith('_'):\n", " val = getattr(outputs, key, None)\n", " if val is not None and hasattr(val, 'shape'):\n", " print(f' {key}: {val.shape}')\n", " elif isinstance(val, (list, tuple)) and len(val) > 0:\n", " print(f' {key}: list of {len(val)}')" ] }, { "cell_type": "markdown", "id": "backbone_attn_md", "metadata": {}, "source": [ "## 2. Backbone Attention Maps\n", "\n", "RF-DETR's DINOv2-Small backbone has 12 transformer blocks with **alternating attention**:\n", "- **Windowed blocks** (0,1,3,4,6,7,9,10): local attention within 4 windows\n", "- **Full blocks** (2,5,8,11): global attention across all patches\n", "\n", "We'll visualize the CLS token attention from both types." ] }, { "cell_type": "code", "execution_count": null, "id": "backbone_attn", "metadata": {}, "outputs": [], "source": [ "# ── Extract backbone attention maps ──────────────────────────────────────────\n", "# The backbone outputs hidden states at stages 2, 5, 8, 11\n", "# We need to hook into the backbone to get attention weights\n", "\n", "backbone_attentions = []\n", "\n", "def make_attn_hook(layer_idx):\n", " def hook(module, input, output):\n", " # output is typically (hidden_states, attention_weights) or just hidden_states\n", " if isinstance(output, tuple) and len(output) > 1:\n", " backbone_attentions.append((layer_idx, output[1].detach().cpu()))\n", " return hook\n", "\n", "# Register hooks on backbone attention layers\n", "hooks = []\n", "backbone = model.model.backbone\n", "\n", "# Try to find attention modules in the backbone\n", "for name, module in backbone.named_modules():\n", " if 'attention' in name.lower() and hasattr(module, 'forward'):\n", " # Check if it's a leaf attention module\n", " if not any(True for _ in module.children()):\n", " pass # leaf module\n", "\n", "# Alternative: use output_hidden_states from backbone\n", "with torch.no_grad():\n", " backbone_outputs = backbone(\n", " inputs['pixel_values'],\n", " output_hidden_states=True,\n", " output_attentions=True\n", " )\n", "\n", "print('Backbone output keys:', [k for k in dir(backbone_outputs) if not k.startswith('_')])\n", "\n", "# Get feature maps at each output stage\n", "if hasattr(backbone_outputs, 'feature_maps') and backbone_outputs.feature_maps is not None:\n", " print(f'\\nFeature maps at output stages:')\n", " for i, fm in enumerate(backbone_outputs.feature_maps):\n", " print(f' Stage {i}: {fm.shape}')\n", "\n", "if hasattr(backbone_outputs, 'hidden_states') and backbone_outputs.hidden_states is not None:\n", " print(f'\\nHidden states: {len(backbone_outputs.hidden_states)} layers')\n", " for i, hs in enumerate(backbone_outputs.hidden_states):\n", " print(f' Layer {i}: {hs.shape}')" ] }, { "cell_type": "code", "execution_count": null, "id": "feature_maps_viz", "metadata": {}, "outputs": [], "source": [ "# ── Visualize multi-scale backbone feature maps ───────────────────────────────\n", "# Get the 4 output feature maps from backbone stages 2, 5, 8, 11\n", "\n", "with torch.no_grad():\n", " backbone_out = model.model.backbone(\n", " inputs['pixel_values'],\n", " output_hidden_states=True\n", " )\n", "\n", "# Feature maps are spatial grids of patch features\n", "# For DINOv2 with patch_size=14 and image_size=518: 37×37 = 1369 patches\n", "patch_grid = 518 // 14 # = 37\n", "\n", "if hasattr(backbone_out, 'feature_maps') and backbone_out.feature_maps is not None:\n", " feature_maps = backbone_out.feature_maps\n", " stage_names = ['Stage 2\\n(early)', 'Stage 5\\n(mid)', 'Stage 8\\n(high)', 'Stage 11\\n(final)']\n", " \n", " fig, axes = plt.subplots(4, 6, figsize=(22, 16))\n", " fig.patch.set_facecolor('#0d1117')\n", " fig.suptitle('RF-DETR Backbone Multi-Scale Feature Maps\\n'\n", " '(PCA of 384-dim features → RGB visualization)',\n", " color='white', fontsize=13, fontweight='bold')\n", " \n", " for stage_idx, (fm, stage_name) in enumerate(zip(feature_maps, stage_names)):\n", " # fm shape: [1, H_feat, W_feat, 384] or [1, N, 384]\n", " fm_np = fm[0].cpu().numpy() # [H, W, C] or [N, C]\n", " \n", " if fm_np.ndim == 3: # [H, W, C]\n", " H_f, W_f, C = fm_np.shape\n", " fm_flat = fm_np.reshape(-1, C)\n", " else: # [N, C]\n", " N, C = fm_np.shape\n", " H_f = W_f = int(N ** 0.5)\n", " fm_flat = fm_np\n", " \n", " # PCA to 3 components for RGB visualization\n", " from sklearn.decomposition import PCA\n", " pca = PCA(n_components=3)\n", " fm_pca = pca.fit_transform(fm_flat) # [N, 3]\n", " \n", " # Normalize to [0, 1]\n", " fm_pca = (fm_pca - fm_pca.min(0)) / (fm_pca.max(0) - fm_pca.min(0) + 1e-8)\n", " fm_rgb = fm_pca.reshape(H_f, W_f, 3)\n", " \n", " # Show original image in first column\n", " axes[stage_idx, 0].imshow(img_np)\n", " axes[stage_idx, 0].set_title(f'{stage_name}\\nInput', color='white', fontsize=8)\n", " axes[stage_idx, 0].axis('off')\n", " \n", " # PCA RGB\n", " axes[stage_idx, 1].imshow(fm_rgb)\n", " axes[stage_idx, 1].set_title(f'{stage_name}\\nPCA RGB', color='white', fontsize=8)\n", " axes[stage_idx, 1].axis('off')\n", " \n", " # Individual channels (first 4)\n", " for ch in range(4):\n", " ch_map = fm_flat[:, ch].reshape(H_f, W_f)\n", " ch_norm = (ch_map - ch_map.min()) / (ch_map.max() - ch_map.min() + 1e-8)\n", " axes[stage_idx, ch + 2].imshow(ch_norm, cmap='viridis')\n", " axes[stage_idx, ch + 2].set_title(f'Channel {ch}', color='white', fontsize=8)\n", " axes[stage_idx, ch + 2].axis('off')\n", " \n", " plt.tight_layout()\n", " plt.savefig('rfdetr_feature_maps.png', dpi=100, bbox_inches='tight', facecolor='#0d1117')\n", " plt.show()\n", " print('Feature maps saved.')\n", "else:\n", " print('Feature maps not directly accessible — using hidden states instead')\n", " \n", " if hasattr(backbone_out, 'hidden_states') and backbone_out.hidden_states:\n", " # Use hidden states at output stages\n", " output_stages = [2, 5, 8, 11]\n", " stage_names = ['Stage 2 (early)', 'Stage 5 (mid)', 'Stage 8 (high)', 'Stage 11 (final)']\n", " \n", " fig, axes = plt.subplots(2, 4, figsize=(20, 10))\n", " fig.patch.set_facecolor('#0d1117')\n", " fig.suptitle('RF-DETR Backbone Hidden States at Output Stages',\n", " color='white', fontsize=13, fontweight='bold')\n", " \n", " for col, (stage_idx, stage_name) in enumerate(zip(output_stages, stage_names)):\n", " if stage_idx < len(backbone_out.hidden_states):\n", " hs = backbone_out.hidden_states[stage_idx][0].cpu().numpy() # [N+1, C]\n", " # Skip CLS token\n", " patch_feats = hs[1:] if hs.shape[0] > patch_grid**2 else hs\n", " N = patch_feats.shape[0]\n", " gs = int(N ** 0.5)\n", " \n", " # Mean activation map\n", " mean_act = patch_feats.mean(-1).reshape(gs, gs)\n", " mean_norm = (mean_act - mean_act.min()) / (mean_act.max() - mean_act.min() + 1e-8)\n", " mean_resized = np.array(Image.fromarray((mean_norm * 255).astype(np.uint8)).resize((W, H), Image.BILINEAR))\n", " \n", " axes[0, col].imshow(img_np)\n", " axes[0, col].imshow(mean_resized, cmap='hot', alpha=0.6)\n", " axes[0, col].set_title(stage_name, color='white', fontsize=9, fontweight='bold')\n", " axes[0, col].axis('off')\n", " \n", " # L2 norm map\n", " l2_norm = np.linalg.norm(patch_feats, axis=-1).reshape(gs, gs)\n", " l2_norm_n = (l2_norm - l2_norm.min()) / (l2_norm.max() - l2_norm.min() + 1e-8)\n", " l2_resized = np.array(Image.fromarray((l2_norm_n * 255).astype(np.uint8)).resize((W, H), Image.BILINEAR))\n", " \n", " axes[1, col].imshow(img_np)\n", " axes[1, col].imshow(l2_resized, cmap='plasma', alpha=0.6)\n", " axes[1, col].set_title(f'{stage_name}\\n(L2 norm)', color='white', fontsize=9)\n", " axes[1, col].axis('off')\n", " \n", " plt.tight_layout()\n", " plt.savefig('rfdetr_hidden_states.png', dpi=100, bbox_inches='tight', facecolor='#0d1117')\n", " plt.show()\n", " print('Hidden state visualization saved.')" ] }, { "cell_type": "markdown", "id": "query_analysis_md", "metadata": {}, "source": [ "## 3. Object Query Analysis\n", "\n", "RF-DETR uses **300 object queries** — learnable embeddings that compete to detect objects.\n", "We'll analyze:\n", "- Which queries have high confidence scores\n", "- Where the high-confidence queries' predicted boxes are located\n", "- How query scores distribute across the image" ] }, { "cell_type": "code", "execution_count": null, "id": "query_analysis", "metadata": {}, "outputs": [], "source": [ "# ── Object query analysis ─────────────────────────────────────────────────────\n", "logits = outputs.logits[0].cpu() # [300, 92]\n", "pred_boxes = outputs.pred_boxes[0].cpu() # [300, 4] normalized cx,cy,w,h\n", "\n", "# Get scores (sigmoid for RF-DETR)\n", "probs = torch.sigmoid(logits) # [300, 92]\n", "scores, class_ids = probs[:, :-1].max(dim=-1) # [300]\n", "scores_np = scores.numpy()\n", "class_ids_np = class_ids.numpy()\n", "boxes_np = pred_boxes.numpy()\n", "\n", "id2label = {int(k): v for k, v in model.config.id2label.items()}\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(20, 12))\n", "fig.patch.set_facecolor('#0d1117')\n", "fig.suptitle('RF-DETR Object Query Analysis (300 queries)',\n", " color='white', fontsize=14, fontweight='bold')\n", "\n", "# ── Plot 1: Score distribution ────────────────────────────────────────────────\n", "ax = axes[0, 0]\n", "ax.set_facecolor('#1a1a2e')\n", "ax.hist(scores_np, bins=60, color='#3498db', edgecolor='none', alpha=0.8)\n", "for thresh, color in [(0.2, '#f1c40f'), (0.35, '#e74c3c'), (0.5, '#e74c3c')]:\n", " n = (scores_np > thresh).sum()\n", " ax.axvline(x=thresh, color=color, linestyle='--', linewidth=2,\n", " label=f'>{thresh}: {n} queries')\n", "ax.set_title('Query Score Distribution', color='white', fontweight='bold')\n", "ax.set_xlabel('Max Class Score', color='white')\n", "ax.set_ylabel('Count', color='white')\n", "ax.tick_params(colors='white')\n", "for spine in ax.spines.values(): spine.set_color('white')\n", "ax.legend(facecolor='#1a1a2e', edgecolor='white', labelcolor='white', fontsize=8)\n", "\n", "# ── Plot 2: Query box centers colored by score ────────────────────────────────\n", "ax2 = axes[0, 1]\n", "ax2.set_facecolor('#1a1a2e')\n", "ax2.imshow(img_np, extent=[0, 1, 1, 0], aspect='auto', alpha=0.4)\n", "cx = boxes_np[:, 0]\n", "cy = boxes_np[:, 1]\n", "sc = ax2.scatter(cx, cy, c=scores_np, cmap='hot', s=30, alpha=0.8,\n", " vmin=0, vmax=1, edgecolors='none')\n", "plt.colorbar(sc, ax=ax2).ax.yaxis.label.set_color('white')\n", "ax2.set_title('Query Box Centers\\n(color = confidence)', color='white', fontweight='bold')\n", "ax2.set_xlabel('cx (normalized)', color='white')\n", "ax2.set_ylabel('cy (normalized)', color='white')\n", "ax2.tick_params(colors='white')\n", "for spine in ax2.spines.values(): spine.set_color('white')\n", "ax2.set_xlim(0, 1)\n", "ax2.set_ylim(0, 1)\n", "\n", "# ── Plot 3: Box size distribution ────────────────────────────────────────────\n", "ax3 = axes[0, 2]\n", "ax3.set_facecolor('#1a1a2e')\n", "bw = boxes_np[:, 2] # normalized width\n", "bh = boxes_np[:, 3] # normalized height\n", "sc3 = ax3.scatter(bw, bh, c=scores_np, cmap='hot', s=20, alpha=0.7, vmin=0, vmax=1)\n", "plt.colorbar(sc3, ax=ax3).ax.yaxis.label.set_color('white')\n", "ax3.set_title('Query Box Sizes\\n(color = confidence)', color='white', fontweight='bold')\n", "ax3.set_xlabel('Box Width (normalized)', color='white')\n", "ax3.set_ylabel('Box Height (normalized)', color='white')\n", "ax3.tick_params(colors='white')\n", "for spine in ax3.spines.values(): spine.set_color('white')\n", "\n", "# ── Plot 4: Top-20 queries overlaid on image ──────────────────────────────────\n", "ax4 = axes[1, 0]\n", "ax4.set_facecolor('#0d1117')\n", "ax4.imshow(img_np)\n", "ax4.set_title('Top-20 Queries (all scores > 0.05)', color='white', fontweight='bold')\n", "ax4.axis('off')\n", "\n", "top20_idx = np.argsort(scores_np)[::-1][:20]\n", "cmap = plt.cm.hot\n", "for rank, idx in enumerate(top20_idx):\n", " cx_i, cy_i, bw_i, bh_i = boxes_np[idx]\n", " x1 = (cx_i - bw_i/2) * W\n", " y1 = (cy_i - bh_i/2) * H\n", " bw_px = bw_i * W\n", " bh_px = bh_i * H\n", " color = cmap(scores_np[idx])\n", " rect = patches.Rectangle((x1, y1), bw_px, bh_px,\n", " linewidth=1.5, edgecolor=color, facecolor='none', alpha=0.8)\n", " ax4.add_patch(rect)\n", " cls_name = id2label.get(int(class_ids_np[idx]), '?')\n", " if scores_np[idx] > 0.2:\n", " ax4.text(x1, y1 - 2, f'{cls_name[:8]} {scores_np[idx]:.2f}',\n", " color='white', fontsize=6,\n", " bbox=dict(facecolor=color, alpha=0.7, pad=0.5, edgecolor='none'))\n", "\n", "# ── Plot 5: Auxiliary decoder layer scores ────────────────────────────────────\n", "ax5 = axes[1, 1]\n", "ax5.set_facecolor('#1a1a2e')\n", "\n", "if hasattr(outputs, 'auxiliary_outputs') and outputs.auxiliary_outputs:\n", " layer_max_scores = []\n", " for aux in outputs.auxiliary_outputs:\n", " aux_logits = aux['logits'][0].cpu()\n", " aux_probs = torch.sigmoid(aux_logits)\n", " aux_scores, _ = aux_probs[:, :-1].max(dim=-1)\n", " layer_max_scores.append(aux_scores.numpy())\n", " \n", " # Add final layer\n", " layer_max_scores.append(scores_np)\n", " \n", " layer_means = [s.mean() for s in layer_max_scores]\n", " layer_maxes = [s.max() for s in layer_max_scores]\n", " layer_above_35 = [(s > 0.35).sum() for s in layer_max_scores]\n", " \n", " x_layers = range(len(layer_max_scores))\n", " ax5.plot(x_layers, layer_means, 'o-', color='#3498db', linewidth=2, label='Mean score')\n", " ax5.plot(x_layers, layer_maxes, 's-', color='#e74c3c', linewidth=2, label='Max score')\n", " ax5_twin = ax5.twinx()\n", " ax5_twin.bar(x_layers, layer_above_35, alpha=0.3, color='#27ae60', label='N > 0.35')\n", " ax5_twin.set_ylabel('Queries > 0.35', color='#27ae60')\n", " ax5_twin.tick_params(colors='#27ae60')\n", " \n", " ax5.set_title('Score Evolution Across Decoder Layers',\n", " color='white', fontweight='bold')\n", " ax5.set_xlabel('Decoder Layer', color='white')\n", " ax5.set_ylabel('Score', color='white')\n", " ax5.tick_params(colors='white')\n", " for spine in ax5.spines.values(): spine.set_color('white')\n", " ax5.legend(facecolor='#1a1a2e', edgecolor='white', labelcolor='white', fontsize=8)\n", " ax5.set_xticks(x_layers)\n", " ax5.set_xticklabels([f'L{i}' for i in range(len(layer_max_scores))], color='white')\n", "else:\n", " ax5.text(0.5, 0.5, 'Auxiliary outputs\\nnot available',\n", " ha='center', va='center', color='white', fontsize=12)\n", " ax5.axis('off')\n", "\n", "# ── Plot 6: Class distribution of high-confidence queries ─────────────────────\n", "ax6 = axes[1, 2]\n", "ax6.set_facecolor('#1a1a2e')\n", "from collections import Counter\n", "high_conf_mask = scores_np > 0.1\n", "if high_conf_mask.sum() > 0:\n", " class_counts = Counter(class_ids_np[high_conf_mask])\n", " top_classes = sorted(class_counts.items(), key=lambda x: -x[1])[:12]\n", " cls_names = [id2label.get(c, f'cls_{c}')[:12] for c, _ in top_classes]\n", " counts = [cnt for _, cnt in top_classes]\n", " colors_bar = plt.cm.tab20(np.linspace(0, 1, len(counts)))\n", " ax6.barh(cls_names, counts, color=colors_bar, edgecolor='white', linewidth=0.5)\n", " ax6.set_title('Top Classes (score > 0.1)', color='white', fontweight='bold')\n", " ax6.set_xlabel('Query count', color='white')\n", " ax6.tick_params(colors='white')\n", " for spine in ax6.spines.values(): spine.set_color('white')\n", "\n", "plt.tight_layout()\n", "plt.savefig('rfdetr_query_analysis.png', dpi=100, bbox_inches='tight', facecolor='#0d1117')\n", "plt.show()\n", "print('Query analysis saved.')" ] }, { "cell_type": "markdown", "id": "dinov2_compare_md", "metadata": {}, "source": [ "## 4. DINOv2 Standalone vs RF-DETR Backbone Features\n", "\n", "RF-DETR uses a **modified DINOv2** with windowed attention.\n", "Let's compare the patch features from:\n", "1. **Standalone DINOv2-base** (standard, full attention)\n", "2. **RF-DETR backbone** (windowed + full attention alternation)\n", "\n", "This shows how the windowing affects feature quality." ] }, { "cell_type": "code", "execution_count": null, "id": "dinov2_compare", "metadata": {}, "outputs": [], "source": [ "# ── Compare DINOv2 standalone vs RF-DETR backbone ─────────────────────────────\n", "from transformers import AutoModel\n", "\n", "DINOV2_PATH = '/mnt/mydata/huggingface/dinov2-base'\n", "\n", "print('Loading standalone DINOv2-base...')\n", "try:\n", " dinov2_processor = AutoImageProcessor.from_pretrained(DINOV2_PATH)\n", " dinov2_model = AutoModel.from_pretrained(DINOV2_PATH)\n", " dinov2_model = dinov2_model.to(device)\n", " dinov2_model.eval()\n", " print('✅ DINOv2-base loaded')\n", " \n", " # Get DINOv2 features\n", " dinov2_inputs = dinov2_processor(images=img, return_tensors='pt')\n", " dinov2_inputs = {k: v.to(device) for k, v in dinov2_inputs.items()}\n", " with torch.no_grad():\n", " dinov2_out = dinov2_model(**dinov2_inputs)\n", " \n", " dinov2_patches = dinov2_out.last_hidden_state[0, 1:, :].cpu().numpy() # [N, 768]\n", " dinov2_gs = int(dinov2_patches.shape[0] ** 0.5)\n", " print(f'DINOv2 patch grid: {dinov2_gs}×{dinov2_gs}, dim={dinov2_patches.shape[1]}')\n", " \n", " DINOV2_AVAILABLE = True\n", "except Exception as e:\n", " print(f'DINOv2-base not available: {e}')\n", " DINOV2_AVAILABLE = False\n", "\n", "# Get RF-DETR backbone features\n", "with torch.no_grad():\n", " rfdetr_backbone_out = model.model.backbone(\n", " inputs['pixel_values'],\n", " output_hidden_states=True\n", " )\n", "\n", "# Use last hidden state from RF-DETR backbone\n", "if hasattr(rfdetr_backbone_out, 'hidden_states') and rfdetr_backbone_out.hidden_states:\n", " rfdetr_last_hs = rfdetr_backbone_out.hidden_states[-1][0].cpu().numpy() # [N+1, 384]\n", " # Skip CLS if present\n", " rfdetr_patches = rfdetr_last_hs[1:] if rfdetr_last_hs.shape[0] > 1 else rfdetr_last_hs\n", " rfdetr_gs = int(rfdetr_patches.shape[0] ** 0.5)\n", " print(f'RF-DETR backbone patch grid: {rfdetr_gs}×{rfdetr_gs}, dim={rfdetr_patches.shape[1]}')\n", " RFDETR_BACKBONE_AVAILABLE = True\n", "else:\n", " print('RF-DETR backbone hidden states not available')\n", " RFDETR_BACKBONE_AVAILABLE = False" ] }, { "cell_type": "code", "execution_count": null, "id": "feature_compare_viz", "metadata": {}, "outputs": [], "source": [ "# ── Feature comparison visualization ─────────────────────────────────────────\n", "from sklearn.decomposition import PCA\n", "\n", "fig, axes = plt.subplots(2, 4, figsize=(20, 10))\n", "fig.patch.set_facecolor('#0d1117')\n", "fig.suptitle('Feature Comparison: DINOv2-base vs RF-DETR Backbone',\n", " color='white', fontsize=13, fontweight='bold')\n", "\n", "def visualize_features(ax_row, patches_feat, gs, title_prefix, img_np, W, H):\n", " \"\"\"Visualize patch features using PCA and mean activation.\"\"\"\n", " # PCA RGB\n", " pca = PCA(n_components=3)\n", " pca_feats = pca.fit_transform(patches_feat) # [N, 3]\n", " pca_norm = (pca_feats - pca_feats.min(0)) / (pca_feats.max(0) - pca_feats.min(0) + 1e-8)\n", " pca_rgb = pca_norm.reshape(gs, gs, 3)\n", " \n", " # Mean activation\n", " mean_act = patches_feat.mean(-1).reshape(gs, gs)\n", " mean_norm = (mean_act - mean_act.min()) / (mean_act.max() - mean_act.min() + 1e-8)\n", " mean_resized = np.array(Image.fromarray((mean_norm * 255).astype(np.uint8)).resize((W, H), Image.BILINEAR))\n", " \n", " # L2 norm\n", " l2_norm = np.linalg.norm(patches_feat, axis=-1).reshape(gs, gs)\n", " l2_norm_n = (l2_norm - l2_norm.min()) / (l2_norm.max() - l2_norm.min() + 1e-8)\n", " l2_resized = np.array(Image.fromarray((l2_norm_n * 255).astype(np.uint8)).resize((W, H), Image.BILINEAR))\n", " \n", " # Patch similarity from center\n", " center_idx = (gs // 2) * gs + gs // 2\n", " patches_norm = patches_feat / (np.linalg.norm(patches_feat, axis=-1, keepdims=True) + 1e-8)\n", " sim = (patches_norm @ patches_norm[center_idx]).reshape(gs, gs)\n", " sim_resized = np.array(Image.fromarray(((sim + 1) / 2 * 255).astype(np.uint8)).resize((W, H), Image.BILINEAR))\n", " \n", " plots = [\n", " (pca_rgb, 'PCA RGB', 'viridis', False),\n", " (mean_resized, 'Mean Activation', 'hot', True),\n", " (l2_resized, 'L2 Norm', 'plasma', True),\n", " (sim_resized, 'Center Similarity', 'RdYlGn', True),\n", " ]\n", " \n", " for col, (data, subtitle, cmap, overlay) in enumerate(plots):\n", " ax = ax_row[col]\n", " ax.set_facecolor('#0d1117')\n", " if overlay:\n", " ax.imshow(img_np)\n", " ax.imshow(data, cmap=cmap, alpha=0.6)\n", " else:\n", " ax.imshow(data)\n", " ax.set_title(f'{title_prefix}\\n{subtitle}', color='white', fontsize=8, fontweight='bold')\n", " ax.axis('off')\n", "\n", "if DINOV2_AVAILABLE:\n", " visualize_features(axes[0], dinov2_patches, dinov2_gs,\n", " f'DINOv2-base\\n({dinov2_gs}×{dinov2_gs}, 768-dim)',\n", " img_np, W, H)\n", "else:\n", " for ax in axes[0]:\n", " ax.text(0.5, 0.5, 'DINOv2-base\\nnot available', ha='center', va='center',\n", " color='white', fontsize=10)\n", " ax.axis('off')\n", "\n", "if RFDETR_BACKBONE_AVAILABLE:\n", " visualize_features(axes[1], rfdetr_patches, rfdetr_gs,\n", " f'RF-DETR backbone\\n({rfdetr_gs}×{rfdetr_gs}, 384-dim)',\n", " img_np, W, H)\n", "else:\n", " for ax in axes[1]:\n", " ax.text(0.5, 0.5, 'RF-DETR backbone\\nhidden states not available',\n", " ha='center', va='center', color='white', fontsize=10)\n", " ax.axis('off')\n", "\n", "plt.tight_layout()\n", "plt.savefig('rfdetr_feature_comparison.png', dpi=100, bbox_inches='tight', facecolor='#0d1117')\n", "plt.show()\n", "print('Feature comparison saved.')" ] }, { "cell_type": "markdown", "id": "detection_pipeline_md", "metadata": {}, "source": [ "## 5. Complete Detection Pipeline Visualization\n", "\n", "A comprehensive visualization showing the full RF-DETR pipeline on the robot image." ] }, { "cell_type": "code", "execution_count": null, "id": "pipeline_viz", "metadata": {}, "outputs": [], "source": [ "# ── Full pipeline visualization ───────────────────────────────────────────────\n", "target_sizes = torch.tensor([[H, W]]).to(device)\n", "results_35 = processor.post_process_object_detection(\n", " outputs, target_sizes=target_sizes, threshold=0.35\n", ")[0]\n", "\n", "scores_det = results_35['scores'].cpu().numpy()\n", "labels_det = results_35['labels'].cpu().numpy()\n", "boxes_det = results_35['boxes'].cpu().numpy()\n", "\n", "# Sort by score\n", "sort_idx = np.argsort(scores_det)[::-1]\n", "scores_det = scores_det[sort_idx]\n", "labels_det = labels_det[sort_idx]\n", "boxes_det = boxes_det[sort_idx]\n", "\n", "COLORS_VIZ = plt.cm.tab20(np.linspace(0, 1, 20))\n", "\n", "fig = plt.figure(figsize=(22, 14))\n", "fig.patch.set_facecolor('#0d1117')\n", "fig.suptitle('RF-DETR Complete Detection Pipeline on Real Robot Image',\n", " color='white', fontsize=15, fontweight='bold', y=0.98)\n", "\n", "# ── Main detection result (large) ─────────────────────────────────────────────\n", "ax_main = fig.add_subplot(2, 3, (1, 4))\n", "ax_main.set_facecolor('#0d1117')\n", "ax_main.imshow(img_np)\n", "ax_main.set_title(f'RF-DETR-base Detections (threshold=0.35)\\n'\n", " f'{len(scores_det)} objects detected',\n", " color='white', fontweight='bold', fontsize=12)\n", "ax_main.axis('off')\n", "\n", "for score, label, box in zip(scores_det, labels_det, boxes_det):\n", " x1, y1, x2, y2 = box\n", " color = COLORS_VIZ[int(label) % 20]\n", " rect = patches.Rectangle((x1, y1), x2-x1, y2-y1,\n", " linewidth=3, edgecolor=color, facecolor='none')\n", " ax_main.add_patch(rect)\n", " cls_name = id2label.get(int(label), f'cls_{label}')\n", " ax_main.text(x1, max(y1 - 5, 12), f'{cls_name} {score:.2f}',\n", " color='white', fontsize=9, fontweight='bold',\n", " bbox=dict(facecolor=color, alpha=0.85, pad=2, edgecolor='none'))\n", "\n", "# ── Score bar chart ───────────────────────────────────────────────────────────\n", "ax_scores = fig.add_subplot(2, 3, 3)\n", "ax_scores.set_facecolor('#1a1a2e')\n", "if len(scores_det) > 0:\n", " cls_names_det = [id2label.get(int(l), f'cls_{l}') for l in labels_det]\n", " bar_colors = [COLORS_VIZ[int(l) % 20] for l in labels_det]\n", " y_pos = np.arange(len(scores_det))\n", " ax_scores.barh(y_pos, scores_det, color=bar_colors, edgecolor='white', linewidth=0.5)\n", " ax_scores.set_yticks(y_pos)\n", " ax_scores.set_yticklabels(cls_names_det, color='white', fontsize=9)\n", " ax_scores.set_xlabel('Confidence Score', color='white')\n", " ax_scores.set_title('Detection Scores', color='white', fontweight='bold')\n", " ax_scores.axvline(x=0.35, color='#e74c3c', linestyle='--', linewidth=1.5)\n", " ax_scores.set_xlim(0, 1.05)\n", " for bar, score in zip(ax_scores.patches, scores_det):\n", " ax_scores.text(score + 0.01, bar.get_y() + bar.get_height()/2,\n", " f'{score:.3f}', va='center', color='white', fontsize=8)\n", " ax_scores.tick_params(colors='white')\n", " for spine in ax_scores.spines.values(): spine.set_color('white')\n", "\n", "# ── Query score heatmap on image ──────────────────────────────────────────────\n", "ax_heat = fig.add_subplot(2, 3, 6)\n", "ax_heat.set_facecolor('#0d1117')\n", "\n", "# Create a score density map from all query boxes\n", "score_map = np.zeros((H, W))\n", "for i, (score, box) in enumerate(zip(scores_np, boxes_np)):\n", " if score > 0.05:\n", " cx_i, cy_i, bw_i, bh_i = box\n", " x1 = max(0, int((cx_i - bw_i/2) * W))\n", " y1 = max(0, int((cy_i - bh_i/2) * H))\n", " x2 = min(W, int((cx_i + bw_i/2) * W))\n", " y2 = min(H, int((cy_i + bh_i/2) * H))\n", " score_map[y1:y2, x1:x2] += score\n", "\n", "score_map_norm = (score_map - score_map.min()) / (score_map.max() - score_map.min() + 1e-8)\n", "ax_heat.imshow(img_np)\n", "ax_heat.imshow(score_map_norm, cmap='hot', alpha=0.6)\n", "ax_heat.set_title('Query Score Density Map\\n(all queries > 0.05)',\n", " color='white', fontweight='bold', fontsize=10)\n", "ax_heat.axis('off')\n", "\n", "plt.tight_layout()\n", "plt.savefig('rfdetr_pipeline_complete.png', dpi=120, bbox_inches='tight', facecolor='#0d1117')\n", "plt.show()\n", "print('Complete pipeline visualization saved.')" ] }, { "cell_type": "markdown", "id": "onnx_md", "metadata": {}, "source": [ "## 6. ONNX Export for Deployment\n", "\n", "Export RF-DETR to ONNX for deployment on edge devices or inference servers." ] }, { "cell_type": "code", "execution_count": null, "id": "onnx_export", "metadata": {}, "outputs": [], "source": [ "# ── ONNX Export ───────────────────────────────────────────────────────────────\n", "import torch\n", "\n", "ONNX_PATH = '/tmp/rfdetr_base.onnx'\n", "\n", "# Create a wrapper for clean ONNX export\n", "class RfDetrOnnxWrapper(torch.nn.Module):\n", " def __init__(self, model):\n", " super().__init__()\n", " self.model = model\n", " \n", " def forward(self, pixel_values):\n", " outputs = self.model(pixel_values=pixel_values)\n", " return outputs.logits, outputs.pred_boxes\n", "\n", "wrapper = RfDetrOnnxWrapper(model).to('cpu')\n", "wrapper.eval()\n", "\n", "# Dummy input\n", "dummy_input = torch.randn(1, 3, 518, 518)\n", "\n", "print('Exporting to ONNX...')\n", "try:\n", " with torch.no_grad():\n", " torch.onnx.export(\n", " wrapper,\n", " dummy_input,\n", " ONNX_PATH,\n", " input_names=['pixel_values'],\n", " output_names=['logits', 'pred_boxes'],\n", " dynamic_axes={\n", " 'pixel_values': {0: 'batch_size'},\n", " 'logits': {0: 'batch_size'},\n", " 'pred_boxes': {0: 'batch_size'},\n", " },\n", " opset_version=17,\n", " do_constant_folding=True,\n", " )\n", " \n", " onnx_size = os.path.getsize(ONNX_PATH) / 1e6\n", " print(f'✅ ONNX model saved to {ONNX_PATH}')\n", " print(f' File size: {onnx_size:.1f} MB')\n", " \n", " # Verify with onnxruntime\n", " try:\n", " import onnxruntime as ort\n", " sess = ort.InferenceSession(ONNX_PATH)\n", " ort_inputs = {'pixel_values': dummy_input.numpy()}\n", " ort_outputs = sess.run(None, ort_inputs)\n", " print(f'✅ ONNX inference verified:')\n", " print(f' logits: {ort_outputs[0].shape}')\n", " print(f' pred_boxes: {ort_outputs[1].shape}')\n", " except ImportError:\n", " print('onnxruntime not installed — skipping verification')\n", " print('Install with: pip install onnxruntime-gpu')\n", "\n", "except Exception as e:\n", " print(f'ONNX export failed: {e}')\n", " print('Note: Some custom ops may not be ONNX-compatible')\n", " print('Try: torch.onnx.export(..., opset_version=16)')" ] }, { "cell_type": "markdown", "id": "summary_md", "metadata": {}, "source": [ "## 7. Summary & Complete Tutorial Recap\n", "\n", "### What we covered in this notebook:\n", "\n", "| Topic | Key Finding |\n", "|-------|------------|\n", "| Backbone features | Multi-scale outputs at stages 2,5,8,11 capture different semantic levels |\n", "| Query analysis | 300 queries, most have low scores; top queries cluster around objects |\n", "| Decoder evolution | Scores increase through decoder layers as queries refine |\n", "| Feature comparison | RF-DETR backbone (windowed) vs DINOv2 (full attention) |\n", "| ONNX export | Model can be exported for deployment |\n", "\n", "---\n", "\n", "## Complete RF-DETR Tutorial Series Recap\n", "\n", "| Notebook | Topic | Key Takeaway |\n", "|----------|-------|--------------|\n", "| **16a** | Architecture | DETR + DINOv2 = RF-DETR; Hungarian matching; no NMS |\n", "| **16b** | Setup | `RfDetrForObjectDetection` in transformers ≥5.8.0; config details |\n", "| **16c** | Inference | 300 queries → sigmoid → threshold → boxes; ~60 FPS on A100 |\n", "| **16d** | Training | COCO JSON format; custom Dataset; Hungarian loss; gradient clipping |\n", "| **16e** | Advanced | Feature maps; query analysis; DINOv2 comparison; ONNX export |\n", "\n", "---\n", "\n", "## Key RF-DETR Insights\n", "\n", "```\n", "1. DINOv2 backbone → rich spatial features without ImageNet labels\n", "2. Windowed attention → efficient backbone (8/12 blocks local)\n", "3. 300 object queries → compete to detect objects, no NMS needed\n", "4. Hungarian matching → optimal assignment, no duplicate detections\n", "5. Group DETR (13×) → faster convergence during training\n", "6. 3 decoder layers → lightweight decoder, fast inference\n", "7. Auxiliary losses → gradient signal at each decoder layer\n", "```\n", "\n", "## Resources\n", "\n", "- **Paper**: [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers](https://huggingface.co/papers/2511.09554)\n", "- **HuggingFace**: `stevenbucaille/rf-detr-base`\n", "- **Transformers docs**: `RfDetrForObjectDetection`\n", "- **Roboflow**: [roboflow.com/rf-detr](https://roboflow.com)" ] } ] }