{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "cells": [ { "cell_type": "markdown", "id": "title_md", "metadata": {}, "source": [ "# RF-DETR: Environment Setup & Installation\n", "\n", "**Series:** RF-DETR Detailed Tutorial | **Notebook 16b of 5**\n", "\n", "---\n", "\n", "## What we'll cover\n", "\n", "1. Verify Python & CUDA environment\n", "2. Install `transformers` (ā‰„5.8.0 for RF-DETR support)\n", "3. Inspect the local RF-DETR model checkpoint at `/mnt/mydata/huggingface/rf-detr-base`\n", "4. Understand the model config: backbone, decoder, queries\n", "5. Verify the full dependency stack\n", "\n", "---\n", "\n", "## Prerequisites\n", "\n", "- Python 3.9+\n", "- PyTorch 2.0+ (with CUDA recommended)\n", "- `transformers` ā‰„ 5.8.0 (RF-DETR was merged in PR #36895)\n", "\n", "**Local model path:** `/mnt/mydata/huggingface/rf-detr-base`" ] }, { "cell_type": "markdown", "id": "install_md", "metadata": {}, "source": [ "## 1. Installation\n", "\n", "RF-DETR is available natively in šŸ¤— Transformers ā‰„ 5.8.0.\n", "Run the cell below to install/upgrade if needed." ] }, { "cell_type": "code", "execution_count": null, "id": "install_code", "metadata": {}, "outputs": [], "source": [ "# Check current transformers version\n", "import importlib.metadata\n", "try:\n", " version = importlib.metadata.version('transformers')\n", " print(f'transformers version: {version}')\n", " major, minor, *_ = version.split('.')\n", " if int(major) < 5 or (int(major) == 5 and int(minor) < 8):\n", " print('āš ļø Need transformers >= 5.8.0 for RF-DETR support')\n", " print('Run: pip install --upgrade transformers')\n", " else:\n", " print('āœ… transformers version is compatible with RF-DETR')\n", "except Exception as e:\n", " print(f'transformers not found: {e}')\n", " print('Run: pip install transformers>=5.8.0')" ] }, { "cell_type": "code", "execution_count": null, "id": "deps_check", "metadata": {}, "outputs": [], "source": [ "# ā”€ā”€ Full dependency check ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "import sys\n", "import importlib\n", "\n", "required = [\n", " ('torch', '2.0.0', 'PyTorch'),\n", " ('transformers', '5.8.0', 'HuggingFace Transformers'),\n", " ('PIL', '9.0.0', 'Pillow'),\n", " ('numpy', '1.21.0', 'NumPy'),\n", " ('matplotlib', '3.5.0', 'Matplotlib'),\n", " ('torchvision', '0.15.0', 'TorchVision'),\n", " ('scipy', '1.7.0', 'SciPy'),\n", " ('cv2', '4.5.0', 'OpenCV (optional)'),\n", "]\n", "\n", "print(f'Python: {sys.version}\\n')\n", "print(f'{\"Package\":<30} {\"Status\":<12} {\"Version\"}')\n", "print('-' * 60)\n", "\n", "for pkg, min_ver, display_name in required:\n", " try:\n", " mod = importlib.import_module(pkg)\n", " ver = getattr(mod, '__version__', 'unknown')\n", " print(f'{display_name:<30} āœ… OK {ver}')\n", " except ImportError:\n", " optional = '(optional)' in display_name\n", " status = 'āš ļø MISSING' if optional else 'āŒ MISSING'\n", " print(f'{display_name:<30} {status}')" ] }, { "cell_type": "markdown", "id": "cuda_md", "metadata": {}, "source": [ "## 2. CUDA / GPU Environment" ] }, { "cell_type": "code", "execution_count": null, "id": "cuda_check", "metadata": {}, "outputs": [], "source": [ "import torch\n", "\n", "print('=' * 50)\n", "print('PyTorch & CUDA Environment')\n", "print('=' * 50)\n", "print(f'PyTorch version: {torch.__version__}')\n", "print(f'CUDA available: {torch.cuda.is_available()}')\n", "\n", "if torch.cuda.is_available():\n", " print(f'CUDA version: {torch.version.cuda}')\n", " print(f'GPU count: {torch.cuda.device_count()}')\n", " for i in range(torch.cuda.device_count()):\n", " props = torch.cuda.get_device_properties(i)\n", " mem_gb = props.total_memory / 1e9\n", " print(f' GPU {i}: {props.name} ({mem_gb:.1f} GB VRAM)')\n", " print(f'Current device: {torch.cuda.current_device()}')\n", " print(f'cuDNN version: {torch.backends.cudnn.version()}')\n", " print(f'cuDNN enabled: {torch.backends.cudnn.enabled}')\n", "else:\n", " print('āš ļø No GPU detected ā€” inference will run on CPU (slower)')\n", " print(' RF-DETR-base requires ~2GB VRAM for inference')\n", "\n", "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", "print(f'\\nUsing device: {device}')" ] }, { "cell_type": "markdown", "id": "model_files_md", "metadata": {}, "source": [ "## 3. Inspect Local Model Checkpoint\n", "\n", "The RF-DETR-base checkpoint is stored locally at `/mnt/mydata/huggingface/rf-detr-base`.\n", "Let's inspect its files and configuration." ] }, { "cell_type": "code", "execution_count": null, "id": "inspect_files", "metadata": {}, "outputs": [], "source": [ "import os\n", "import json\n", "\n", "MODEL_PATH = '/mnt/mydata/huggingface/rf-detr-base'\n", "\n", "print(f'Model directory: {MODEL_PATH}')\n", "print('\\nFiles:')\n", "for fname in sorted(os.listdir(MODEL_PATH)):\n", " fpath = os.path.join(MODEL_PATH, fname)\n", " size_mb = os.path.getsize(fpath) / 1e6\n", " print(f' {fname:<35} {size_mb:>8.1f} MB')\n", "\n", "# Load and display key config fields\n", "config_path = os.path.join(MODEL_PATH, 'config.json')\n", "with open(config_path) as f:\n", " cfg = json.load(f)\n", "\n", "print('\\n' + '=' * 50)\n", "print('Key Configuration')\n", "print('=' * 50)\n", "print(f'Architecture: {cfg[\"architectures\"][0]}')\n", "print(f'Model type: {cfg[\"model_type\"]}')\n", "print(f'Num object queries: {cfg[\"num_queries\"]}')\n", "print(f'Decoder layers: {cfg[\"decoder_layers\"]}')\n", "print(f'd_model: {cfg[\"d_model\"]}')\n", "print(f'Decoder FFN dim: {cfg[\"decoder_ffn_dim\"]}')\n", "print(f'Decoder self-attn heads: {cfg[\"decoder_self_attention_heads\"]}')\n", "print(f'Decoder cross-attn heads: {cfg[\"decoder_cross_attention_heads\"]}')\n", "print(f'Group DETR: {cfg[\"group_detr\"]}')\n", "print(f'Auxiliary loss: {cfg[\"auxiliary_loss\"]}')\n", "print(f'Num feature levels: {cfg[\"num_feature_levels\"]}')\n", "\n", "bb = cfg['backbone_config']\n", "print(f'\\nBackbone type: {bb[\"model_type\"]}')\n", "print(f'Backbone hidden size: {bb[\"hidden_size\"]} (DINOv2-Small)')\n", "print(f'Backbone layers: {bb[\"num_hidden_layers\"]}')\n", "print(f'Backbone attn heads: {bb[\"num_attention_heads\"]}')\n", "print(f'Patch size: {bb[\"patch_size\"]}px')\n", "print(f'Image size: {bb[\"image_size\"]}px')\n", "print(f'Window blocks: {bb[\"window_block_indexes\"]}')\n", "print(f'Output stages: {bb[\"out_features\"]}')\n", "\n", "print(f'\\nCOCO classes: {len([v for v in cfg[\"id2label\"].values() if v != \"N/A\"])} (80 categories)')" ] }, { "cell_type": "markdown", "id": "config_deep_md", "metadata": {}, "source": [ "## 4. Understanding the Configuration\n", "\n", "### 4.1 Backbone: DINOv2-Small with Windowed Attention\n", "\n", "```\n", "hidden_size = 384 ā†’ DINOv2-Small (vs 768 for Base, 1024 for Large)\n", "num_hidden_layers = 12 ā†’ 12 transformer blocks\n", "num_attention_heads = 6 ā†’ 6 attention heads per block\n", "patch_size = 14 ā†’ 14Ɨ14 pixel patches\n", "image_size = 518 ā†’ 518Ɨ518 input (37Ɨ37 = 1369 patches)\n", "window_block_indexes = [0,1,3,4,6,7,9,10] ā†’ 8 windowed blocks\n", " 4 full-attention blocks\n", "```\n", "\n", "**Windowed attention** (RF-DETR innovation):\n", "- 8 of 12 blocks use **local window attention** (4 windows, efficient)\n", "- 4 blocks use **full global attention** (captures long-range context)\n", "- This alternation gives efficiency without sacrificing global context\n", "\n", "**Multi-scale outputs** from stages 2, 5, 8, 11:\n", "```\n", "stage2 ā†’ early features (edges, textures)\n", "stage5 ā†’ mid-level features (parts)\n", "stage8 ā†’ high-level features (objects)\n", "stage11 ā†’ final features (semantics)\n", "```\n", "\n", "### 4.2 Decoder Configuration\n", "\n", "```\n", "num_queries = 300 ā†’ 300 object query slots\n", "decoder_layers = 3 ā†’ 3 decoder transformer layers (lightweight!)\n", "d_model = 256 ā†’ query/key/value dimension\n", "decoder_ffn_dim = 2048 ā†’ FFN hidden dimension\n", "decoder_cross_attention_heads = 16 ā†’ deformable cross-attention\n", "decoder_n_points = 2 ā†’ 2 reference points per head (deformable)\n", "group_detr = 13 ā†’ 13 parallel decoder groups during training\n", "```\n", "\n", "**Group DETR**: During training, 13 copies of the decoder run in parallel\n", "with different query initializations, providing 13Ɨ more positive samples\n", "for the Hungarian matching loss ā†’ faster convergence.\n", "\n", "### 4.3 Loss Configuration\n", "\n", "```\n", "class_cost = 2 ā†’ weight for classification in matching cost\n", "bbox_cost = 5 ā†’ weight for L1 bbox in matching cost\n", "giou_cost = 2 ā†’ weight for GIoU in matching cost\n", "\n", "class_loss_coefficient = 1\n", "bbox_loss_coefficient = 5\n", "giou_loss_coefficient = 2\n", "focal_alpha = 0.25 ā†’ focal loss alpha (class imbalance)\n", "eos_coefficient = 0.1 ā†’ weight for 'no object' class\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "config_viz", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import matplotlib.patches as mpatches\n", "import json\n", "\n", "with open('/mnt/mydata/huggingface/rf-detr-base/config.json') as f:\n", " cfg = json.load(f)\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(18, 7))\n", "fig.patch.set_facecolor('#0d1117')\n", "\n", "# ā”€ā”€ Plot 1: Backbone windowed vs full attention ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "ax = axes[0]\n", "ax.set_facecolor('#1a1a2e')\n", "ax.set_title('Backbone: Windowed vs Full Attention\\n(12 transformer blocks)',\n", " color='white', fontsize=10, fontweight='bold')\n", "\n", "window_blocks = set(cfg['backbone_config']['window_block_indexes'])\n", "for i in range(12):\n", " color = '#3498db' if i in window_blocks else '#e74c3c'\n", " label = 'Window' if i in window_blocks else 'Full'\n", " rect = mpatches.FancyBboxPatch((0.1, 11 - i), 0.8, 0.7,\n", " boxstyle='round,pad=0.05',\n", " facecolor=color, edgecolor='white',\n", " linewidth=1, alpha=0.85)\n", " ax.add_patch(rect)\n", " ax.text(0.5, 11.35 - i, f'Block {i}: {label} Attention',\n", " ha='center', va='center', color='white', fontsize=8)\n", "\n", "# Output stage markers\n", "for stage_idx, stage_name in zip([2, 5, 8, 11], ['stage2', 'stage5', 'stage8', 'stage11']):\n", " ax.annotate(f'ā†’ {stage_name}', xy=(0.9, 11.35 - stage_idx),\n", " xytext=(1.05, 11.35 - stage_idx),\n", " color='#f1c40f', fontsize=7.5,\n", " arrowprops=dict(arrowstyle='->', color='#f1c40f', lw=1))\n", "\n", "legend_items = [\n", " mpatches.Patch(color='#3498db', label='Windowed (local)'),\n", " mpatches.Patch(color='#e74c3c', label='Full (global)'),\n", "]\n", "ax.legend(handles=legend_items, loc='lower right', facecolor='#1a1a2e',\n", " edgecolor='white', labelcolor='white', fontsize=8)\n", "ax.set_xlim(0, 1.5)\n", "ax.set_ylim(-0.5, 12.5)\n", "ax.axis('off')\n", "\n", "# ā”€ā”€ Plot 2: Loss weights ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "ax2 = axes[1]\n", "ax2.set_facecolor('#1a1a2e')\n", "ax2.set_title('Loss Coefficients', color='white', fontsize=10, fontweight='bold')\n", "\n", "loss_names = ['Classification\\n(Focal)', 'BBox L1', 'GIoU']\n", "loss_vals = [cfg['class_loss_coefficient'], cfg['bbox_loss_coefficient'], cfg['giou_loss_coefficient']]\n", "colors_loss = ['#e74c3c', '#3498db', '#2ecc71']\n", "bars = ax2.bar(loss_names, loss_vals, color=colors_loss, edgecolor='white', linewidth=1.5)\n", "for bar, val in zip(bars, loss_vals):\n", " ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.05,\n", " f'Ī»={val}', ha='center', va='bottom', color='white', fontsize=11, fontweight='bold')\n", "ax2.set_ylabel('Coefficient', color='white')\n", "ax2.tick_params(colors='white')\n", "for spine in ax2.spines.values():\n", " spine.set_color('white')\n", "ax2.set_ylim(0, 7)\n", "\n", "# ā”€ā”€ Plot 3: Decoder structure ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "ax3 = axes[2]\n", "ax3.set_facecolor('#1a1a2e')\n", "ax3.set_title('Decoder: 3 Layers Ɨ 300 Queries\\n(Group DETR: 13 parallel groups in training)',\n", " color='white', fontsize=10, fontweight='bold')\n", "\n", "decoder_blocks = [\n", " (0.5, 5.5, 'Object Queries\\n(300 Ɨ 256)', '#8e44ad'),\n", " (0.5, 4.0, 'Decoder Layer 1\\n(Self-Attn + Cross-Attn + FFN)', '#2980b9'),\n", " (0.5, 2.5, 'Decoder Layer 2\\n(Self-Attn + Cross-Attn + FFN)', '#2980b9'),\n", " (0.5, 1.0, 'Decoder Layer 3\\n(Self-Attn + Cross-Attn + FFN)', '#2980b9'),\n", " (0.5, -0.5, 'Detection Head\\n(FFN ā†’ class + bbox)', '#e67e22'),\n", "]\n", "for xc, yb, label, color in decoder_blocks:\n", " rect = mpatches.FancyBboxPatch((xc - 0.45, yb), 0.9, 1.0,\n", " boxstyle='round,pad=0.05',\n", " facecolor=color, edgecolor='white',\n", " linewidth=1, alpha=0.85)\n", " ax3.add_patch(rect)\n", " ax3.text(xc, yb + 0.5, label, ha='center', va='center',\n", " color='white', fontsize=8)\n", "\n", "# Arrows between blocks\n", "for y_from, y_to in [(5.5, 5.0), (5.0, 4.0), (4.0, 3.5), (3.5, 2.5), (2.5, 2.0), (2.0, 1.0), (1.0, 0.5), (0.5, -0.5)]:\n", " ax3.annotate('', xy=(0.5, y_to), xytext=(0.5, y_from),\n", " arrowprops=dict(arrowstyle='->', color='white', lw=1.5))\n", "\n", "ax3.set_xlim(0, 1)\n", "ax3.set_ylim(-1.5, 7)\n", "ax3.axis('off')\n", "\n", "plt.tight_layout()\n", "plt.savefig('rfdetr_config_viz.png', dpi=120, bbox_inches='tight', facecolor='#0d1117')\n", "plt.show()\n", "print('Config visualization saved.')" ] }, { "cell_type": "markdown", "id": "verify_md", "metadata": {}, "source": [ "## 5. Verify Model Can Be Loaded" ] }, { "cell_type": "code", "execution_count": null, "id": "verify_load", "metadata": {}, "outputs": [], "source": [ "from transformers import AutoImageProcessor, RfDetrForObjectDetection\n", "import torch\n", "\n", "MODEL_PATH = '/mnt/mydata/huggingface/rf-detr-base'\n", "\n", "print('Loading processor...')\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "print(f' āœ… Processor loaded: {type(processor).__name__}')\n", "print(f' Image size: {processor.size}')\n", "\n", "print('\\nLoading model...')\n", "model = RfDetrForObjectDetection.from_pretrained(MODEL_PATH)\n", "model.eval()\n", "print(f' āœ… Model loaded: {type(model).__name__}')\n", "\n", "# Count parameters\n", "total_params = sum(p.numel() for p in model.parameters())\n", "trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "backbone_params = sum(p.numel() for p in model.model.backbone.parameters())\n", "decoder_params = sum(p.numel() for p in model.model.decoder.parameters())\n", "\n", "print(f'\\nParameter counts:')\n", "print(f' Total: {total_params/1e6:.1f}M')\n", "print(f' Backbone: {backbone_params/1e6:.1f}M (DINOv2-Small)')\n", "print(f' Decoder: {decoder_params/1e6:.1f}M')\n", "print(f' Other: {(total_params - backbone_params - decoder_params)/1e6:.1f}M')\n", "\n", "# Move to GPU if available\n", "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", "model = model.to(device)\n", "print(f'\\nāœ… Model on device: {device}')\n", "\n", "if torch.cuda.is_available():\n", " mem_used = torch.cuda.memory_allocated() / 1e9\n", " print(f' GPU memory used: {mem_used:.2f} GB')" ] }, { "cell_type": "code", "execution_count": null, "id": "model_summary", "metadata": {}, "outputs": [], "source": [ "# Print model architecture summary\n", "print('RF-DETR-base Architecture Summary')\n", "print('=' * 60)\n", "\n", "def count_params(module):\n", " return sum(p.numel() for p in module.parameters()) / 1e6\n", "\n", "# Top-level components\n", "for name, module in model.named_children():\n", " params = count_params(module)\n", " print(f'{name:<30} {params:>8.2f}M params')\n", " # One level deeper for model\n", " if name == 'model':\n", " for subname, submodule in module.named_children():\n", " sub_params = count_params(submodule)\n", " print(f' ā””ā”€ {subname:<26} {sub_params:>8.2f}M params')\n", "\n", "print('\\nPreprocessor config:')\n", "print(f' do_resize: {processor.do_resize}')\n", "print(f' size: {processor.size}')\n", "print(f' do_normalize: {processor.do_normalize}')\n", "print(f' image_mean: {processor.image_mean}')\n", "print(f' image_std: {processor.image_std}')" ] }, { "cell_type": "markdown", "id": "summary_md", "metadata": {}, "source": [ "## 6. Summary\n", "\n", "### What we verified:\n", "\n", "| Component | Details |\n", "|-----------|--------|\n", "| Model | `RfDetrForObjectDetection` (transformers ā‰„ 5.8.0) |\n", "| Checkpoint | `/mnt/mydata/huggingface/rf-detr-base` |\n", "| Backbone | DINOv2-Small (384-dim, 12 layers, 6 heads) |\n", "| Decoder | 3 layers, 300 queries, d_model=256 |\n", "| Classes | 80 COCO categories |\n", "| Input size | 518Ɨ518 pixels |\n", "| Patch size | 14Ɨ14 pixels ā†’ 37Ɨ37 = 1369 patches |\n", "\n", "### Key insight: Windowed attention\n", "\n", "RF-DETR uses **alternating windowed/full attention** in the backbone:\n", "- 8 blocks: local window attention (4 windows) ā†’ efficient\n", "- 4 blocks: full global attention ā†’ captures long-range context\n", "\n", "This is the key efficiency innovation over standard DINOv2.\n", "\n", "### Next: Notebook 16c ā€” Model Loading & Inference Basics\n", "\n", "In the next notebook, we'll run RF-DETR inference on real images,\n", "visualize detections, and understand the output format." ] } ] }