{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DINOv2 Tutorial — Notebook 14: Registers & Artifact Removal\n", "\n", "**Series:** DINOv2 Deep Dive | **Prev:** `13_model_variants.ipynb`\n", "\n", "---\n", "\n", "## What You Will Learn\n", "- The artifact problem in ViT attention maps\n", "- What are \"high-norm\" outlier tokens and why they appear\n", "- DINOv2 with Registers: the solution (Darcet et al., 2023)\n", "- How register tokens absorb global information\n", "- Before/after comparison of attention maps\n", "- How to load DINOv2 with registers from Hugging Face\n", "\n", "**Paper:** *Vision Transformers Need Registers* (Darcet et al., 2023) \n", "arXiv: 2309.16588\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.1 The Artifact Problem\n", "\n", "When visualizing DINOv2 attention maps, researchers noticed **artifacts**: some patches have extremely high feature norms and attract disproportionate attention from all other tokens.\n", "\n", "### What are these artifacts?\n", "- Certain patches (often in **smooth, uninformative regions** like backgrounds) develop **very high L2 norms**\n", "- These \"outlier tokens\" attract attention from all other tokens\n", "- They disrupt the semantic structure of attention maps\n", "- They appear in **all large ViT models** trained with self-supervised methods\n", "\n", "### Why do they appear?\n", "The hypothesis: ViTs need to **aggregate global information** somewhere. \n", "When there's no dedicated place for this, some patch tokens are \"hijacked\" to store global context. \n", "These tokens then have high norms and attract broad attention.\n", "\n", "```\n", "Without registers: With registers:\n", "\n", "[CLS] [p1] [p2] ... [p256] [CLS] [R1] [R2] [R3] [R4] [p1] [p2] ... [p256]\n", " ↑ ↑ ↑ ↑ ↑\n", " Some patches become Register tokens absorb\n", " \"global info sinks\" global information\n", " (artifacts) (patches stay local)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "from PIL import Image, ImageDraw\n", "from transformers import AutoImageProcessor, AutoModel\n", "\n", "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n", "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n", "cfg = model.config\n", "\n", "n_grid = cfg.image_size // cfg.patch_size\n", "\n", "def make_test_image(size=224):\n", " img = Image.new('RGB', (size, size), (135, 206, 235))\n", " draw = ImageDraw.Draw(img)\n", " draw.rectangle([0, size*2//3, size, size], fill=(80, 160, 80))\n", " draw.rectangle([20, size//3, size//2-10, size*2//3], fill=(180, 120, 80))\n", " draw.polygon([(0, size//3), (size//2, size//3), (size//4, size//6)], fill=(160, 60, 60))\n", " draw.rectangle([size//4-15, size//2, size//4+15, size*2//3], fill=(100, 60, 30))\n", " draw.rectangle([size*3//4-15, size//2, size*3//4+15, size*2//3], fill=(100, 70, 40))\n", " draw.ellipse([size*3//4-45, size//4, size*3//4+45, size//2+10], fill=(50, 140, 50))\n", " draw.ellipse([size*3//4+5, 15, size*3//4+45, 55], fill=(255, 220, 50))\n", " return img\n", "\n", "test_image = make_test_image()\n", "inputs = processor(images=test_image, return_tensors='pt')\n", "inputs = {k: v.to(DEVICE) for k, v in inputs.items()}\n", "\n", "with torch.no_grad():\n", " outputs = model(**inputs, output_attentions=True, output_hidden_states=True)\n", "\n", "print(\"Model loaded and forward pass complete.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.2 Detecting Outlier Tokens (Artifacts)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Detect high-norm outlier tokens ───────────────────────────────────────────\n", "# Get patch features from last hidden state\n", "last_hidden = outputs.last_hidden_state[0].cpu().numpy() # [257, 768]\n", "patch_feats = last_hidden[1:] # [256, 768]\n", "cls_feat = last_hidden[0] # [768]\n", "\n", "# Compute L2 norms\n", "patch_norms = np.linalg.norm(patch_feats, axis=1) # [256]\n", "norm_grid = patch_norms.reshape(n_grid, n_grid)\n", "\n", "# Identify outliers (norms > mean + 2*std)\n", "mean_norm = patch_norms.mean()\n", "std_norm = patch_norms.std()\n", "threshold = mean_norm + 2 * std_norm\n", "outlier_mask = patch_norms > threshold\n", "outlier_grid = outlier_mask.reshape(n_grid, n_grid)\n", "\n", "print(f\"Patch norm statistics:\")\n", "print(f\" Mean: {mean_norm:.2f}\")\n", "print(f\" Std: {std_norm:.2f}\")\n", "print(f\" Max: {patch_norms.max():.2f}\")\n", "print(f\" Min: {patch_norms.min():.2f}\")\n", "print(f\" Outlier threshold (mean + 2*std): {threshold:.2f}\")\n", "print(f\" Number of outlier patches: {outlier_mask.sum()} / {len(patch_norms)}\")\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n", "\n", "# Original image\n", "axes[0, 0].imshow(test_image)\n", "axes[0, 0].set_title('Original Image', fontweight='bold')\n", "axes[0, 0].axis('off')\n", "\n", "# Patch norm heatmap\n", "im1 = axes[0, 1].imshow(norm_grid, cmap='hot', interpolation='nearest')\n", "axes[0, 1].set_title(f'Patch Feature L2 Norm\\n(mean={mean_norm:.1f}, std={std_norm:.1f})',\n", " fontweight='bold')\n", "axes[0, 1].axis('off')\n", "plt.colorbar(im1, ax=axes[0, 1], fraction=0.046)\n", "\n", "# Outlier mask\n", "axes[0, 2].imshow(outlier_grid.astype(float), cmap='RdYlGn_r', vmin=0, vmax=1,\n", " interpolation='nearest')\n", "axes[0, 2].set_title(f'Outlier Patches (norm > {threshold:.1f})\\n'\n", " f'{outlier_mask.sum()} outliers detected',\n", " fontweight='bold')\n", "axes[0, 2].axis('off')\n", "\n", "# Norm distribution\n", "axes[1, 0].hist(patch_norms, bins=30, color='steelblue', alpha=0.8, edgecolor='black')\n", "axes[1, 0].axvline(mean_norm, color='green', linestyle='--', linewidth=2,\n", " label=f'Mean = {mean_norm:.1f}')\n", "axes[1, 0].axvline(threshold, color='red', linestyle='--', linewidth=2,\n", " label=f'Threshold = {threshold:.1f}')\n", "axes[1, 0].set_title('Patch Norm Distribution', fontweight='bold')\n", "axes[1, 0].set_xlabel('L2 Norm'); axes[1, 0].set_ylabel('Count')\n", "axes[1, 0].legend(); axes[1, 0].grid(True, alpha=0.3)\n", "\n", "# Attention to outlier patches\n", "last_attn = outputs.attentions[-1][0].cpu().numpy() # [12, 257, 257]\n", "# How much attention do outlier patches receive from CLS?\n", "cls_attn_to_patches = last_attn[:, 0, 1:].mean(axis=0) # [256] mean over heads\n", "outlier_attn = cls_attn_to_patches[outlier_mask]\n", "normal_attn = cls_attn_to_patches[~outlier_mask]\n", "\n", "axes[1, 1].hist(normal_attn, bins=20, alpha=0.7, color='green',\n", " label=f'Normal patches (n={(~outlier_mask).sum()})\\nmean={normal_attn.mean():.4f}')\n", "axes[1, 1].hist(outlier_attn, bins=10, alpha=0.7, color='red',\n", " label=f'Outlier patches (n={outlier_mask.sum()})\\nmean={outlier_attn.mean():.4f}')\n", "axes[1, 1].set_title('CLS Attention to Normal vs Outlier Patches\\n'\n", " '(do outliers attract more attention?)',\n", " fontweight='bold')\n", "axes[1, 1].set_xlabel('Attention Weight'); axes[1, 1].set_ylabel('Count')\n", "axes[1, 1].legend(fontsize=9); axes[1, 1].grid(True, alpha=0.3)\n", "\n", "# Overlay outliers on image\n", "img_np = np.array(test_image).astype(float) / 255.0\n", "outlier_up = np.array(Image.fromarray(outlier_grid.astype(np.uint8) * 255).resize(\n", " (224, 224), Image.NEAREST)) / 255.0\n", "overlay = img_np.copy()\n", "overlay[outlier_up > 0.5] = [1.0, 0.0, 0.0] # red for outliers\n", "axes[1, 2].imshow(overlay.clip(0, 1))\n", "axes[1, 2].set_title('Outlier Patches Highlighted\\n(red = high-norm artifact patches)',\n", " fontweight='bold')\n", "axes[1, 2].axis('off')\n", "\n", "plt.suptitle('Detecting High-Norm Outlier Tokens (Artifacts) in DINOv2',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.3 The Register Solution\n", "\n", "**DINOv2 with Registers** (Darcet et al., 2023) adds **register tokens** to the sequence:\n", "\n", "```\n", "Standard DINOv2:\n", "[CLS] [p1] [p2] ... [p256]\n", "\n", "DINOv2 with Registers (n_reg=4):\n", "[CLS] [R1] [R2] [R3] [R4] [p1] [p2] ... [p256]\n", " ↑ ↑ ↑ ↑\n", " Register tokens (learnable, discarded at output)\n", "```\n", "\n", "### How registers work:\n", "1. **Register tokens** are learnable vectors inserted after CLS\n", "2. They can attend to all patches and accumulate global information\n", "3. Patch tokens no longer need to be \"hijacked\" as global info sinks\n", "4. Register tokens are **discarded** at the output (only CLS and patch tokens used)\n", "5. Result: **cleaner attention maps** with no artifacts\n", "\n", "### Available models:\n", "- `facebook/dinov2-base-imagenet1k-1-layer` — with registers\n", "- `facebook/dinov2-with-registers-base`\n", "- `facebook/dinov2-with-registers-large`\n", "- `facebook/dinov2-with-registers-giant`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Simulate the effect of registers ──────────────────────────────────────────\n", "# We simulate what registers do by showing the concept\n", "# (actual register model requires facebook/dinov2-with-registers-base)\n", "\n", "# Simulate: without registers, some patches have very high norms\n", "np.random.seed(42)\n", "n_patches = n_grid * n_grid # 256\n", "\n", "# Without registers: some patches become outliers\n", "norms_without_reg = np.random.exponential(scale=50, size=n_patches)\n", "# Add some outliers (the artifact patches)\n", "outlier_indices = np.random.choice(n_patches, size=8, replace=False)\n", "norms_without_reg[outlier_indices] *= 4 # 4x higher norm\n", "\n", "# With registers: norms are more uniform (registers absorb global info)\n", "norms_with_reg = np.random.exponential(scale=50, size=n_patches)\n", "# No extreme outliers\n", "\n", "# Attention maps: without registers, outliers attract too much attention\n", "attn_without_reg = np.random.dirichlet(np.ones(n_patches) * 0.5)\n", "# Outliers attract more attention\n", "attn_without_reg[outlier_indices] *= 5\n", "attn_without_reg /= attn_without_reg.sum()\n", "\n", "# With registers: attention is more semantically meaningful\n", "attn_with_reg = np.random.dirichlet(np.ones(n_patches) * 0.5)\n", "# Semantic regions get more attention\n", "semantic_patches = np.arange(n_patches // 3, 2 * n_patches // 3)\n", "attn_with_reg[semantic_patches] *= 2\n", "attn_with_reg /= attn_with_reg.sum()\n", "\n", "fig, axes = plt.subplots(2, 4, figsize=(20, 10))\n", "\n", "# Without registers\n", "im0 = axes[0, 0].imshow(norms_without_reg.reshape(n_grid, n_grid),\n", " cmap='hot', interpolation='nearest')\n", "axes[0, 0].set_title('Patch Norms WITHOUT Registers\\n(outliers visible)', fontweight='bold')\n", "axes[0, 0].axis('off')\n", "plt.colorbar(im0, ax=axes[0, 0], fraction=0.046)\n", "\n", "im1 = axes[0, 1].imshow(attn_without_reg.reshape(n_grid, n_grid),\n", " cmap='hot', interpolation='bilinear')\n", "axes[0, 1].set_title('CLS Attention WITHOUT Registers\\n(outliers attract too much attention)',\n", " fontweight='bold')\n", "axes[0, 1].axis('off')\n", "plt.colorbar(im1, ax=axes[0, 1], fraction=0.046)\n", "\n", "# With registers\n", "im2 = axes[0, 2].imshow(norms_with_reg.reshape(n_grid, n_grid),\n", " cmap='hot', interpolation='nearest')\n", "axes[0, 2].set_title('Patch Norms WITH Registers\\n(uniform, no outliers)', fontweight='bold')\n", "axes[0, 2].axis('off')\n", "plt.colorbar(im2, ax=axes[0, 2], fraction=0.046)\n", "\n", "im3 = axes[0, 3].imshow(attn_with_reg.reshape(n_grid, n_grid),\n", " cmap='hot', interpolation='bilinear')\n", "axes[0, 3].set_title('CLS Attention WITH Registers\\n(semantic, no artifacts)',\n", " fontweight='bold')\n", "axes[0, 3].axis('off')\n", "plt.colorbar(im3, ax=axes[0, 3], fraction=0.046)\n", "\n", "# Norm distributions\n", "axes[1, 0].hist(norms_without_reg, bins=30, color='red', alpha=0.7, label='Without registers')\n", "axes[1, 0].hist(norms_with_reg, bins=30, color='green', alpha=0.7, label='With registers')\n", "axes[1, 0].set_title('Patch Norm Distribution\\n(simulated)', fontweight='bold')\n", "axes[1, 0].set_xlabel('L2 Norm'); axes[1, 0].set_ylabel('Count')\n", "axes[1, 0].legend(); axes[1, 0].grid(True, alpha=0.3)\n", "\n", "# Attention distributions\n", "axes[1, 1].hist(attn_without_reg, bins=30, color='red', alpha=0.7, label='Without registers')\n", "axes[1, 1].hist(attn_with_reg, bins=30, color='green', alpha=0.7, label='With registers')\n", "axes[1, 1].set_title('CLS Attention Distribution\\n(simulated)', fontweight='bold')\n", "axes[1, 1].set_xlabel('Attention Weight'); axes[1, 1].set_ylabel('Count')\n", "axes[1, 1].legend(); axes[1, 1].grid(True, alpha=0.3)\n", "\n", "# Architecture diagram\n", "ax = axes[1, 2]\n", "ax.set_xlim(0, 10); ax.set_ylim(0, 10)\n", "ax.axis('off')\n", "ax.set_title('Architecture: Without vs With Registers', fontweight='bold')\n", "\n", "# Without registers\n", "ax.text(2.5, 9.5, 'Without Registers', ha='center', fontsize=10, fontweight='bold', color='red')\n", "tokens_no_reg = ['CLS'] + [f'p{i}' for i in range(1, 9)]\n", "for i, tok in enumerate(tokens_no_reg):\n", " color = '#e74c3c' if tok == 'CLS' else ('#ff9999' if i in [3, 6] else '#ffcccc')\n", " ax.add_patch(plt.FancyBboxPatch((0.2 + i*0.5, 7.5), 0.4, 0.8,\n", " boxstyle='round,pad=0.05', facecolor=color,\n", " edgecolor='black', linewidth=1))\n", " ax.text(0.4 + i*0.5, 7.9, tok, ha='center', va='center', fontsize=7, fontweight='bold')\n", "# Mark outliers\n", "for i in [3, 6]:\n", " ax.text(0.4 + i*0.5, 7.2, '!', ha='center', fontsize=12, color='red', fontweight='bold')\n", "ax.text(2.5, 6.8, 'Some patches become outliers (artifacts)', ha='center', fontsize=8, color='red')\n", "\n", "# With registers\n", "ax.text(7.5, 9.5, 'With Registers (n=4)', ha='center', fontsize=10, fontweight='bold', color='green')\n", "tokens_reg = ['CLS', 'R1', 'R2', 'R3', 'R4'] + [f'p{i}' for i in range(1, 9)]\n", "for i, tok in enumerate(tokens_reg[:9]):\n", " if tok == 'CLS':\n", " color = '#e74c3c'\n", " elif tok.startswith('R'):\n", " color = '#f39c12'\n", " else:\n", " color = '#aed6f1'\n", " ax.add_patch(plt.FancyBboxPatch((5.2 + i*0.5, 7.5), 0.4, 0.8,\n", " boxstyle='round,pad=0.05', facecolor=color,\n", " edgecolor='black', linewidth=1))\n", " ax.text(5.4 + i*0.5, 7.9, tok, ha='center', va='center', fontsize=6, fontweight='bold')\n", "ax.text(7.5, 6.8, 'Registers absorb global info\\nPatches stay local', ha='center',\n", " fontsize=8, color='green')\n", "\n", "# Legend\n", "ax.add_patch(plt.FancyBboxPatch((5.2, 5.5), 0.4, 0.5, boxstyle='round,pad=0.05',\n", " facecolor='#f39c12', edgecolor='black', linewidth=1))\n", "ax.text(5.8, 5.75, '= Register token (discarded at output)', va='center', fontsize=8)\n", "\n", "# Performance comparison\n", "axes[1, 3].axis('off')\n", "axes[1, 3].text(0.5, 0.95, 'Registers: Performance Impact', ha='center',\n", " fontsize=11, fontweight='bold', transform=axes[1,3].transAxes)\n", "perf_text = [\n", " 'Task Without With',\n", " '─────────────────────────────',\n", " 'ImageNet k-NN 82.1% 82.3%',\n", " 'ImageNet Linear 84.5% 84.8%',\n", " 'ADE20K Seg 53.0 54.0',\n", " 'Attention quality ★★★ ★★★★★',\n", " 'Artifact-free No Yes',\n", " '',\n", " 'Key benefit:',\n", " ' Cleaner attention maps',\n", " ' Better dense prediction',\n", " ' No accuracy loss',\n", " '',\n", " 'n_registers = 4 (default)',\n", " 'Registers discarded at output',\n", "]\n", "for i, line in enumerate(perf_text):\n", " axes[1, 3].text(0.05, 0.85 - i*0.065, line, transform=axes[1,3].transAxes,\n", " fontsize=8, va='top', family='monospace')\n", "\n", "plt.suptitle('DINOv2 with Registers: Eliminating Attention Artifacts',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.4 Loading DINOv2 with Registers" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Load DINOv2 with registers (if available) ──────────────────────────────────\n", "print(\"DINOv2 with Registers — Available Models:\")\n", "print()\n", "register_models = [\n", " ('facebook/dinov2-with-registers-small', 'ViT-S/14 + 4 registers', '21M'),\n", " ('facebook/dinov2-with-registers-base', 'ViT-B/14 + 4 registers', '86M'),\n", " ('facebook/dinov2-with-registers-large', 'ViT-L/14 + 4 registers', '307M'),\n", " ('facebook/dinov2-with-registers-giant', 'ViT-g/14 + 4 registers', '1.1B'),\n", "]\n", "for hf_name, desc, params in register_models:\n", " print(f\" {desc} ({params})\")\n", " print(f\" HF: {hf_name}\")\n", " print()\n", "\n", "print(\"Loading code:\")\n", "print(\"\"\"\n", "from transformers import AutoImageProcessor, AutoModel\n", "\n", "# Load DINOv2 with registers\n", "processor = AutoImageProcessor.from_pretrained('facebook/dinov2-with-registers-base')\n", "model = AutoModel.from_pretrained('facebook/dinov2-with-registers-base')\n", "\n", "# The model has 4 extra register tokens\n", "# Sequence: [CLS, R1, R2, R3, R4, p1, p2, ..., p256]\n", "# Total seq_len = 1 + 4 + 256 = 261\n", "\n", "# Extract features (skip register tokens)\n", "outputs = model(**inputs)\n", "cls_token = outputs.last_hidden_state[:, 0] # CLS token\n", "patch_feats = outputs.last_hidden_state[:, 5:] # Skip CLS + 4 registers\n", "\"\"\")\n", "\n", "# Try to load if available\n", "try:\n", " reg_processor = AutoImageProcessor.from_pretrained('facebook/dinov2-with-registers-base')\n", " reg_model = AutoModel.from_pretrained('facebook/dinov2-with-registers-base').to(DEVICE).eval()\n", " print(\"\\nSuccessfully loaded DINOv2 with registers!\")\n", " print(f\"Config: {reg_model.config}\")\n", " n_registers = getattr(reg_model.config, 'num_register_tokens', 4)\n", " print(f\"Number of register tokens: {n_registers}\")\n", " print(f\"Sequence length: 1 (CLS) + {n_registers} (registers) + 256 (patches) = {1+n_registers+256}\")\n", "except Exception as e:\n", " print(f\"\\nNote: Could not load register model from HF: {e}\")\n", " print(\"Using standard DINOv2 for demonstration.\")\n", " print(\"To use registers, download: facebook/dinov2-with-registers-base\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.5 Analyzing Artifacts in the Current Model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Analyze artifacts across all layers ───────────────────────────────────────\n", "# Track which patches become outliers at each layer\n", "hidden_states = [hs[0].cpu().numpy() for hs in outputs.hidden_states]\n", "\n", "fig, axes = plt.subplots(3, 5, figsize=(22, 14))\n", "\n", "# Show norm maps at layers 1, 3, 6, 9, 12\n", "layers_to_show = [0, 2, 5, 8, 11, 12] # embedding + blocks 3,6,9,12 + final\n", "layer_names = ['Embed', 'Block 3', 'Block 6', 'Block 9', 'Block 12', 'Final LN']\n", "\n", "for col, (layer_idx, layer_name) in enumerate(zip(layers_to_show, layer_names)):\n", " hs = hidden_states[layer_idx]\n", " patch_norms_l = np.linalg.norm(hs[1:], axis=1).reshape(n_grid, n_grid)\n", " \n", " # Norm map\n", " im = axes[0, col].imshow(patch_norms_l, cmap='hot', interpolation='nearest')\n", " axes[0, col].set_title(f'{layer_name}\\nNorm map', fontweight='bold', fontsize=9)\n", " axes[0, col].axis('off')\n", " plt.colorbar(im, ax=axes[0, col], fraction=0.046)\n", " \n", " # Outlier detection\n", " norms_flat = np.linalg.norm(hs[1:], axis=1)\n", " thresh = norms_flat.mean() + 2 * norms_flat.std()\n", " outliers = (norms_flat > thresh).reshape(n_grid, n_grid)\n", " axes[1, col].imshow(outliers.astype(float), cmap='RdYlGn_r', vmin=0, vmax=1,\n", " interpolation='nearest')\n", " axes[1, col].set_title(f'{layer_name}\\nOutliers ({outliers.sum()})',\n", " fontweight='bold', fontsize=9)\n", " axes[1, col].axis('off')\n", "\n", "# Track outlier count across all layers\n", "outlier_counts = []\n", "for layer_idx, hs in enumerate(hidden_states):\n", " norms_flat = np.linalg.norm(hs[1:], axis=1)\n", " thresh = norms_flat.mean() + 2 * norms_flat.std()\n", " outlier_counts.append((norms_flat > thresh).sum())\n", "\n", "axes[2, 0].plot(range(13), outlier_counts, 'r-o', linewidth=2, markersize=8)\n", "axes[2, 0].set_title('Outlier Count per Layer\\n(patches with norm > mean+2std)',\n", " fontweight='bold')\n", "axes[2, 0].set_xlabel('Layer (0=embed, 1-12=blocks)')\n", "axes[2, 0].set_ylabel('Number of outlier patches')\n", "axes[2, 0].set_xticks(range(13))\n", "axes[2, 0].grid(True, alpha=0.3)\n", "\n", "# Max norm per layer\n", "max_norms = [np.linalg.norm(hs[1:], axis=1).max() for hs in hidden_states]\n", "axes[2, 1].plot(range(13), max_norms, 'b-o', linewidth=2, markersize=8)\n", "axes[2, 1].set_title('Max Patch Norm per Layer', fontweight='bold')\n", "axes[2, 1].set_xlabel('Layer'); axes[2, 1].set_ylabel('Max L2 Norm')\n", "axes[2, 1].set_xticks(range(13))\n", "axes[2, 1].grid(True, alpha=0.3)\n", "\n", "# Norm std per layer (measure of outlier-ness)\n", "norm_stds = [np.linalg.norm(hs[1:], axis=1).std() for hs in hidden_states]\n", "axes[2, 2].plot(range(13), norm_stds, 'g-o', linewidth=2, markersize=8)\n", "axes[2, 2].set_title('Patch Norm Std per Layer\\n(higher = more outliers)', fontweight='bold')\n", "axes[2, 2].set_xlabel('Layer'); axes[2, 2].set_ylabel('Std of L2 Norms')\n", "axes[2, 2].set_xticks(range(13))\n", "axes[2, 2].grid(True, alpha=0.3)\n", "\n", "# Summary text\n", "axes[2, 3].axis('off')\n", "axes[2, 3].text(0.5, 0.95, 'Artifact Summary', ha='center', fontsize=12,\n", " fontweight='bold', transform=axes[2,3].transAxes)\n", "summary = [\n", " f'Total patches: {n_grid*n_grid}',\n", " f'Outliers (final layer): {outlier_counts[-1]}',\n", " f'Max norm (final): {max_norms[-1]:.1f}',\n", " f'Mean norm (final): {np.linalg.norm(hidden_states[-1][1:], axis=1).mean():.1f}',\n", " '',\n", " 'Artifacts appear in:',\n", " ' - Smooth background regions',\n", " ' - Low-texture areas',\n", " ' - Uninformative patches',\n", " '',\n", " 'Solution: Add register tokens',\n", " ' facebook/dinov2-with-registers-*',\n", "]\n", "for i, line in enumerate(summary):\n", " axes[2, 3].text(0.05, 0.85 - i*0.075, line, transform=axes[2,3].transAxes,\n", " fontsize=9, va='top')\n", "\n", "axes[2, 4].axis('off')\n", "\n", "plt.suptitle('Artifact Analysis Across All Layers', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14.6 Summary\n", "\n", "| Aspect | Without Registers | With Registers |\n", "|--------|-------------------|----------------|\n", "| Artifact patches | Present (high-norm outliers) | Eliminated |\n", "| Attention maps | Noisy, artifacts visible | Clean, semantic |\n", "| Dense prediction | Slightly degraded | Improved |\n", "| Classification | Same | Same |\n", "| Sequence length | 257 (CLS + 256 patches) | 261 (CLS + 4 reg + 256 patches) |\n", "| Extra parameters | — | ~4 × 768 = 3K (negligible) |\n", "\n", "### Key Takeaways\n", "1. **Artifacts are universal** in large SSL-trained ViTs — not just DINOv2\n", "2. **Registers are a simple fix**: add 4 learnable tokens, discard at output\n", "3. **No accuracy loss**: registers don't hurt classification performance\n", "4. **Better dense features**: patch features are cleaner without artifacts\n", "5. **Recommended**: always use DINOv2 with registers for dense prediction tasks\n", "\n", "---\n", "\n", "## Series Complete!\n", "\n", "You have completed the full DINOv2 tutorial series. Here's what you covered:\n", "\n", "| # | Notebook | Topic |\n", "|---|----------|-------|\n", "| 1 | Setup & Environment | Model loading, first forward pass |\n", "| 2 | SSL Theory | Self-supervised learning concepts |\n", "| 3 | Training Procedure | DINO + iBOT + KoLeo losses |\n", "| 4 | Dataset Preparation | LVD-142M curation pipeline |\n", "| 5 | ViT Architecture | Patch embedding, positional encoding, transformer blocks |\n", "| 6 | Attention Layer I/O | Q, K, V matrices, attention maps, rollout |\n", "| 7 | Intermediate Layers | Feature evolution, residual stream |\n", "| 8 | Patch Embeddings | PCA, k-means, spatial feature maps |\n", "| 9 | CLS Token Analysis | Global representation, t-SNE, k-NN |\n", "| 10 | Head Specialization | All 12 heads × 12 layers, entropy analysis |\n", "| 11 | Similarity & Retrieval | Cosine similarity, k-NN retrieval |\n", "| 12 | Downstream Tasks | Linear probing, segmentation, depth |\n", "| 13 | Model Variants | ViT-S/B/L/g comparison, scaling laws |\n", "| 14 | Registers | Artifact removal, register tokens |\n", "\n", "**Return to:** [`README.md`](README.md)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }