{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DINOv2 Tutorial — Notebook 2: Self-Supervised Learning Theory\n", "\n", "**Series:** DINOv2 Deep Dive | **Prev:** `01_setup_and_environment.ipynb` | **Next:** `03_training_procedure.ipynb`\n", "\n", "---\n", "\n", "## What You Will Learn\n", "- What self-supervised learning (SSL) is and why it matters\n", "- The landscape of SSL methods: contrastive, non-contrastive, masked, distillation\n", "- How DINO (v1) works: self-distillation with no labels\n", "- What DINOv2 adds on top of DINO\n", "- Visualizations of augmentation views, collapse prevention, and knowledge distillation\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.1 What is Self-Supervised Learning?\n", "\n", "**Supervised learning** requires human-annotated labels (expensive, slow, limited scale). \n", "**Self-supervised learning** creates *pretext tasks* from the data itself — no labels needed.\n", "\n", "```\n", "Supervised: Image → [human labels] → Train classifier\n", "Self-Supervised: Image → [automatic signal from data] → Learn representations\n", "```\n", "\n", "### Why SSL for Vision?\n", "- The internet has **billions of images** but very few are labeled\n", "- SSL can leverage this massive unlabeled data\n", "- Good SSL representations transfer to many downstream tasks\n", "- DINOv2 achieves **state-of-the-art** on ImageNet with a *frozen* backbone + linear probe\n", "\n", "### Core Idea: Invariance\n", "Two augmented views of the same image should produce **similar representations**. \n", "Two views of *different* images should produce **different representations**.\n", "\n", "```\n", "Image X ──┬── augment_1 ──→ view_1 ──→ encoder ──→ z_1 ─┐\n", " │ ├── similar\n", " └── augment_2 ──→ view_2 ──→ encoder ──→ z_2 ─┘\n", "\n", "Image Y ──── augment_1 ──→ view_3 ──→ encoder ──→ z_3 ── different from z_1, z_2\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as mpatches\n", "from PIL import Image, ImageDraw, ImageFilter\n", "import random\n", "\n", "# ── Visualize the SSL invariance concept ──────────────────────────────────────\n", "def make_base_image(size=224):\n", " img = Image.new('RGB', (size, size), (240, 240, 240))\n", " draw = ImageDraw.Draw(img)\n", " draw.ellipse([40, 40, 184, 184], fill=(70, 130, 180), outline=(30, 80, 130), width=4)\n", " draw.rectangle([80, 80, 144, 144], fill=(255, 165, 0), outline=(200, 120, 0), width=3)\n", " return img\n", "\n", "def augment_view(img, crop_box, flip=False, brightness=1.0):\n", " \"\"\"Simulate a random augmented crop.\"\"\"\n", " view = img.crop(crop_box).resize((112, 112), Image.BILINEAR)\n", " if flip:\n", " view = view.transpose(Image.FLIP_LEFT_RIGHT)\n", " # Adjust brightness\n", " arr = np.array(view).astype(float)\n", " arr = np.clip(arr * brightness, 0, 255).astype(np.uint8)\n", " return Image.fromarray(arr)\n", "\n", "base = make_base_image()\n", "view1 = augment_view(base, (20, 20, 180, 180), flip=False, brightness=1.1)\n", "view2 = augment_view(base, (40, 40, 200, 200), flip=True, brightness=0.9)\n", "other = make_base_image(224)\n", "draw2 = ImageDraw.Draw(other)\n", "draw2.rectangle([20, 20, 200, 200], fill=(180, 60, 60))\n", "draw2.ellipse([60, 60, 160, 160], fill=(60, 180, 60))\n", "view3 = augment_view(other, (30, 30, 190, 190), flip=False, brightness=1.0)\n", "\n", "fig, axes = plt.subplots(1, 5, figsize=(18, 4))\n", "axes[0].imshow(base); axes[0].set_title('Image X\\n(original)', fontweight='bold'); axes[0].axis('off')\n", "axes[1].imshow(view1); axes[1].set_title('View 1 of X\\n(crop + flip)', fontweight='bold'); axes[1].axis('off')\n", "axes[2].imshow(view2); axes[2].set_title('View 2 of X\\n(crop + brightness)', fontweight='bold'); axes[2].axis('off')\n", "axes[3].imshow(other); axes[3].set_title('Image Y\\n(different image)', fontweight='bold'); axes[3].axis('off')\n", "axes[4].imshow(view3); axes[4].set_title('View of Y\\n(crop)', fontweight='bold'); axes[4].axis('off')\n", "\n", "# Annotations\n", "fig.text(0.38, -0.05, '← These should have SIMILAR representations →',\n", " ha='center', fontsize=12, color='green', fontweight='bold')\n", "fig.text(0.72, -0.05, '← DIFFERENT from above →',\n", " ha='center', fontsize=12, color='red', fontweight='bold')\n", "\n", "plt.suptitle('SSL Core Idea: Invariance to Augmentations', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 SSL Method Landscape\n", "\n", "| Method | Year | Category | Key Idea | Collapse Prevention |\n", "|--------|------|----------|----------|--------------------|\n", "| SimCLR | 2020 | Contrastive | Maximize agreement between views, push apart negatives | Negative pairs |\n", "| MoCo v2 | 2020 | Contrastive | Momentum encoder + memory bank | Negative queue |\n", "| BYOL | 2020 | Non-contrastive | Bootstrap: online→target, no negatives | Momentum + predictor |\n", "| SimSiam | 2021 | Non-contrastive | Stop-gradient prevents collapse | Stop-gradient |\n", "| SwAV | 2020 | Clustering | Online clustering + swapped prediction | Prototype assignment |\n", "| BarlowTwins | 2021 | Redundancy reduction | Cross-correlation matrix → identity | Decorrelation loss |\n", "| **DINO** | **2021** | **Self-distillation** | **Student/teacher with centering+sharpening** | **Centering + EMA** |\n", "| MAE | 2021 | Masked autoencoding | Reconstruct masked patches | Reconstruction task |\n", "| iBOT | 2022 | Masked distillation | DINO + masked patch prediction | Centering + EMA |\n", "| **DINOv2** | **2023** | **Combined** | **DINO + iBOT + KoLeo + curated data** | **Multiple mechanisms** |\n", "\n", "### The Collapse Problem\n", "Without careful design, the encoder can **collapse** — outputting the same representation for all inputs. \n", "This is a trivial solution that minimizes the loss but learns nothing useful." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize the collapse problem ────────────────────────────────────────────\n", "fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n", "\n", "np.random.seed(42)\n", "n = 200\n", "\n", "# Good representations: well-separated clusters\n", "ax = axes[0]\n", "for i, (color, label) in enumerate(zip(['#e74c3c','#3498db','#2ecc71','#f39c12'],\n", " ['Cat','Dog','Car','Bird'])):\n", " cx, cy = np.cos(i*np.pi/2)*2, np.sin(i*np.pi/2)*2\n", " pts = np.random.randn(50, 2)*0.4 + [cx, cy]\n", " ax.scatter(pts[:,0], pts[:,1], c=color, label=label, s=30, alpha=0.8)\n", "ax.set_title('Good Representations\\n(well-separated clusters)', fontweight='bold', color='green')\n", "ax.legend(loc='upper right', fontsize=9)\n", "ax.set_xlim(-4, 4); ax.set_ylim(-4, 4)\n", "ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Dim 1'); ax.set_ylabel('Dim 2')\n", "\n", "# Collapsed: all same point\n", "ax = axes[1]\n", "for color, label in zip(['#e74c3c','#3498db','#2ecc71','#f39c12'],\n", " ['Cat','Dog','Car','Bird']):\n", " pts = np.random.randn(50, 2)*0.05\n", " ax.scatter(pts[:,0], pts[:,1], c=color, label=label, s=30, alpha=0.8)\n", "ax.set_title('Collapsed Representations\\n(all same point — useless!)', fontweight='bold', color='red')\n", "ax.legend(loc='upper right', fontsize=9)\n", "ax.set_xlim(-4, 4); ax.set_ylim(-4, 4)\n", "ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Dim 1'); ax.set_ylabel('Dim 2')\n", "\n", "# Dimensional collapse: all on a line\n", "ax = axes[2]\n", "for i, (color, label) in enumerate(zip(['#e74c3c','#3498db','#2ecc71','#f39c12'],\n", " ['Cat','Dog','Car','Bird'])):\n", " x = np.random.randn(50)*1.5 + i*0.5\n", " y = x * 0.1 + np.random.randn(50)*0.05\n", " ax.scatter(x, y, c=color, label=label, s=30, alpha=0.8)\n", "ax.set_title('Dimensional Collapse\\n(all on a line — partial collapse)', fontweight='bold', color='orange')\n", "ax.legend(loc='upper right', fontsize=9)\n", "ax.set_xlim(-4, 4); ax.set_ylim(-4, 4)\n", "ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Dim 1'); ax.set_ylabel('Dim 2')\n", "\n", "plt.suptitle('The Representation Collapse Problem in SSL', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 DINO (v1): Self-Distillation with No Labels\n", "\n", "**DINO** (Self-**DI**stillation with **NO** labels) was introduced by Caron et al. (2021). \n", "It is the direct predecessor of DINOv2.\n", "\n", "### Core Architecture\n", "\n", "```\n", " ┌─────────────────────────────────────────┐\n", " │ DINO Framework │\n", " │ │\n", " Image ──┬─ aug1 ──┤──→ Student ViT ──→ Student Head ──→ p_s│\n", " │ │ ↓ │\n", " │ │ Cross-Entropy Loss\n", " │ │ ↑ │\n", " └─ aug2 ──┤──→ Teacher ViT ──→ Teacher Head ──→ p_t│\n", " │ ↑ │\n", " │ EMA update │\n", " │ θ_t ← λθ_t + (1-λ)θ_s │\n", " └─────────────────────────────────────────┘\n", "```\n", "\n", "### Key Mechanisms\n", "1. **Student** processes all crops (global + local)\n", "2. **Teacher** processes only global crops (224×224)\n", "3. **Teacher weights** = exponential moving average (EMA) of student weights\n", "4. **Centering**: subtract running mean from teacher output → prevents one-dim collapse\n", "5. **Sharpening**: low temperature on teacher output → sharp probability distribution\n", "6. **No negative pairs needed!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize centering and sharpening ────────────────────────────────────────\n", "import torch\n", "import torch.nn.functional as F\n", "\n", "np.random.seed(0)\n", "K = 10 # number of output dimensions (prototype classes)\n", "\n", "# Raw logits from teacher head\n", "raw_logits = np.array([2.1, 0.3, -0.5, 1.8, 0.1, -1.2, 0.9, 0.4, -0.3, 1.1])\n", "\n", "# Running center (mean over batch)\n", "center = np.array([0.8, 0.2, -0.1, 0.6, 0.0, -0.4, 0.3, 0.1, -0.1, 0.4])\n", "\n", "# After centering\n", "centered = raw_logits - center\n", "\n", "# Softmax at different temperatures\n", "def softmax_temp(x, T):\n", " e = np.exp((x - x.max()) / T)\n", " return e / e.sum()\n", "\n", "p_raw = softmax_temp(raw_logits, T=1.0)\n", "p_centered = softmax_temp(centered, T=1.0)\n", "p_sharp = softmax_temp(centered, T=0.04) # teacher temperature\n", "p_student = softmax_temp(centered, T=0.1) # student temperature\n", "\n", "fig, axes = plt.subplots(2, 2, figsize=(14, 8))\n", "x = np.arange(K)\n", "width = 0.6\n", "\n", "axes[0,0].bar(x, raw_logits, color='steelblue', width=width)\n", "axes[0,0].set_title('Raw Teacher Logits', fontweight='bold')\n", "axes[0,0].set_xlabel('Prototype Index'); axes[0,0].set_ylabel('Logit Value')\n", "axes[0,0].grid(True, alpha=0.3)\n", "\n", "axes[0,1].bar(x, centered, color='darkorange', width=width)\n", "axes[0,1].axhline(0, color='black', linewidth=1)\n", "axes[0,1].set_title('After Centering (subtract running mean)\\n→ prevents one-dim collapse', fontweight='bold')\n", "axes[0,1].set_xlabel('Prototype Index'); axes[0,1].set_ylabel('Centered Logit')\n", "axes[0,1].grid(True, alpha=0.3)\n", "\n", "axes[1,0].bar(x - 0.2, p_centered, width=0.35, label='T=1.0 (soft)', color='steelblue', alpha=0.8)\n", "axes[1,0].bar(x + 0.2, p_sharp, width=0.35, label='T=0.04 (sharp)', color='crimson', alpha=0.8)\n", "axes[1,0].set_title('Sharpening Effect (Teacher Temperature)\\nLow T → sharp distribution → clear signal', fontweight='bold')\n", "axes[1,0].set_xlabel('Prototype Index'); axes[1,0].set_ylabel('Probability')\n", "axes[1,0].legend(); axes[1,0].grid(True, alpha=0.3)\n", "\n", "# Cross-entropy loss visualization\n", "axes[1,1].bar(x - 0.2, p_sharp, width=0.35, label='Teacher (T=0.04)', color='crimson', alpha=0.8)\n", "axes[1,1].bar(x + 0.2, p_student, width=0.35, label='Student (T=0.1)', color='steelblue', alpha=0.8)\n", "ce_loss = -np.sum(p_sharp * np.log(p_student + 1e-8))\n", "axes[1,1].set_title(f'Student vs Teacher Distributions\\nCross-Entropy Loss = {ce_loss:.3f}', fontweight='bold')\n", "axes[1,1].set_xlabel('Prototype Index'); axes[1,1].set_ylabel('Probability')\n", "axes[1,1].legend(); axes[1,1].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('DINO: Centering and Sharpening Mechanisms', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print(f\"Entropy of soft distribution (T=1.0) : {-np.sum(p_centered * np.log(p_centered+1e-8)):.3f} bits\")\n", "print(f\"Entropy of sharp distribution (T=0.04): {-np.sum(p_sharp * np.log(p_sharp+1e-8)):.3f} bits\")\n", "print(\"→ Lower entropy = sharper = more informative training signal\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 EMA (Exponential Moving Average) Teacher Update\n", "\n", "The teacher is **never trained directly** — it is updated as an EMA of the student:\n", "\n", "$$\\theta_t \\leftarrow \\lambda \\theta_t + (1 - \\lambda) \\theta_s$$\n", "\n", "where $\\lambda$ is the **momentum** (typically 0.996 → 0.9999 via cosine schedule).\n", "\n", "**Why EMA?**\n", "- The teacher is a **temporally averaged** version of the student\n", "- It provides a **stable, slowly-changing target** — prevents oscillation\n", "- Similar to Polyak averaging in optimization\n", "- The teacher is always \"ahead\" of the student in terms of stability" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Simulate EMA update dynamics ──────────────────────────────────────────────\n", "np.random.seed(42)\n", "T_steps = 200\n", "\n", "# Simulate student weight (one scalar for illustration)\n", "student_weight = np.zeros(T_steps)\n", "student_weight[0] = 0.0\n", "# Student makes noisy gradient steps toward target=1.0\n", "for t in range(1, T_steps):\n", " grad = -(student_weight[t-1] - 1.0) + np.random.randn() * 0.3\n", " student_weight[t] = student_weight[t-1] + 0.05 * grad\n", "\n", "# EMA teacher at different momentum values\n", "def compute_ema(student, momentum):\n", " teacher = np.zeros_like(student)\n", " teacher[0] = student[0]\n", " for t in range(1, len(student)):\n", " teacher[t] = momentum * teacher[t-1] + (1 - momentum) * student[t]\n", " return teacher\n", "\n", "teacher_996 = compute_ema(student_weight, 0.996)\n", "teacher_999 = compute_ema(student_weight, 0.999)\n", "teacher_9999 = compute_ema(student_weight, 0.9999)\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", "\n", "t = np.arange(T_steps)\n", "axes[0].plot(t, student_weight, 'b-', alpha=0.6, linewidth=1, label='Student (noisy)')\n", "axes[0].plot(t, teacher_996, 'r-', linewidth=2, label='Teacher (λ=0.996)')\n", "axes[0].plot(t, teacher_999, 'g-', linewidth=2, label='Teacher (λ=0.999)')\n", "axes[0].plot(t, teacher_9999, 'm-', linewidth=2, label='Teacher (λ=0.9999)')\n", "axes[0].axhline(1.0, color='black', linestyle='--', alpha=0.5, label='True target')\n", "axes[0].set_title('EMA Teacher vs Student Weight Dynamics', fontweight='bold')\n", "axes[0].set_xlabel('Training Step'); axes[0].set_ylabel('Weight Value')\n", "axes[0].legend(); axes[0].grid(True, alpha=0.3)\n", "\n", "# Momentum schedule (cosine)\n", "steps = np.linspace(0, 1, 200)\n", "lambda_start, lambda_end = 0.996, 0.9999\n", "momentum_schedule = lambda_end - (lambda_end - lambda_start) * (np.cos(np.pi * steps) + 1) / 2\n", "axes[1].plot(steps * 100, momentum_schedule, 'purple', linewidth=2)\n", "axes[1].fill_between(steps * 100, lambda_start, momentum_schedule, alpha=0.2, color='purple')\n", "axes[1].set_title('Momentum Schedule (Cosine)\\nλ: 0.996 → 0.9999 over training', fontweight='bold')\n", "axes[1].set_xlabel('Training Progress (%)'); axes[1].set_ylabel('Momentum λ')\n", "axes[1].set_ylim(0.994, 1.0001)\n", "axes[1].grid(True, alpha=0.3)\n", "axes[1].annotate('Early: lower momentum\\n(faster adaptation)', xy=(10, 0.9965),\n", " xytext=(20, 0.9975), arrowprops=dict(arrowstyle='->', color='black'), fontsize=10)\n", "axes[1].annotate('Late: higher momentum\\n(more stable target)', xy=(90, 0.9998),\n", " xytext=(60, 0.9985), arrowprops=dict(arrowstyle='->', color='black'), fontsize=10)\n", "\n", "plt.suptitle('EMA Teacher Update in DINO/DINOv2', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.5 From DINO to DINOv2: What Changed?\n", "\n", "| Aspect | DINO (2021) | DINOv2 (2023) |\n", "|--------|-------------|---------------|\n", "| **Loss** | DINO (CLS only) | DINO + iBOT + KoLeo |\n", "| **Data** | ImageNet-1K (1.2M) | LVD-142M (142M curated) |\n", "| **Data curation** | None | Retrieval-based deduplication |\n", "| **Patch prediction** | No | Yes (iBOT masked patches) |\n", "| **Regularization** | Centering only | Centering + KoLeo entropy |\n", "| **Model scale** | ViT-S/B | ViT-S/B/L/g |\n", "| **Training** | Standard | Fully sharded, mixed precision |\n", "| **Downstream** | Good | SOTA on many benchmarks |\n", "\n", "### DINOv2 Key Innovations\n", "1. **Curated dataset (LVD-142M)**: retrieval-based curation from 1.2B web images\n", "2. **iBOT loss**: masked image modeling at the patch level (like BERT for images)\n", "3. **KoLeo regularizer**: maximizes entropy of batch representations\n", "4. **Larger models**: ViT-g with 1.1B parameters\n", "5. **Frozen features**: works directly without fine-tuning for many tasks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Compare SSL methods conceptually ──────────────────────────────────────────\n", "fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n", "\n", "def draw_method_diagram(ax, title, description, color):\n", " ax.set_xlim(0, 10); ax.set_ylim(0, 10)\n", " ax.axis('off')\n", " ax.set_facecolor(color + '22')\n", " ax.add_patch(plt.Rectangle((0,0), 10, 10, fill=True, facecolor=color+'11',\n", " edgecolor=color, linewidth=2))\n", " ax.text(5, 9, title, ha='center', va='top', fontsize=13, fontweight='bold', color=color)\n", " for i, line in enumerate(description):\n", " ax.text(0.5, 7.5 - i*1.2, line, ha='left', va='top', fontsize=10,\n", " wrap=True, color='black')\n", "\n", "methods = [\n", " ('SimCLR / MoCo\\n(Contrastive)',\n", " ['• Positive pairs: same image, diff aug',\n", " '• Negative pairs: different images',\n", " '• Loss: InfoNCE / NT-Xent',\n", " '• Needs large batch or memory bank',\n", " '• Collapse: prevented by negatives'],\n", " '#e74c3c'),\n", " ('BYOL\\n(Non-Contrastive)',\n", " ['• No negative pairs needed',\n", " '• Online → Target (EMA) network',\n", " '• Predictor MLP on online side',\n", " '• Stop-gradient on target',\n", " '• Collapse: prevented by asymmetry'],\n", " '#3498db'),\n", " ('MAE\\n(Masked Autoencoding)',\n", " ['• Mask 75% of patches randomly',\n", " '• Encoder sees only visible patches',\n", " '• Decoder reconstructs masked pixels',\n", " '• Loss: MSE on pixel values',\n", " '• Collapse: prevented by reconstruction'],\n", " '#2ecc71'),\n", " ('DINO v1\\n(Self-Distillation)',\n", " ['• Student/Teacher with EMA',\n", " '• Multi-crop: global + local views',\n", " '• CLS token cross-entropy loss',\n", " '• Centering + sharpening',\n", " '• Collapse: centering prevents it'],\n", " '#9b59b6'),\n", " ('iBOT\\n(Masked Distillation)',\n", " ['• DINO + masked patch prediction',\n", " '• Mask random patches in student',\n", " '• Teacher predicts masked patches',\n", " '• Both CLS and patch-level loss',\n", " '• Better dense features'],\n", " '#e67e22'),\n", " ('DINOv2\\n(Combined)',\n", " ['• DINO loss (CLS token)',\n", " '• iBOT loss (masked patches)',\n", " '• KoLeo regularizer',\n", " '• Curated LVD-142M dataset',\n", " '• SOTA frozen features'],\n", " '#1abc9c'),\n", "]\n", "\n", "for ax, (title, desc, color) in zip(axes.flat, methods):\n", " draw_method_diagram(ax, title, desc, color)\n", "\n", "plt.suptitle('SSL Method Landscape: From SimCLR to DINOv2', fontsize=16, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.6 Knowledge Distillation in DINO\n", "\n", "DINO is a form of **self-distillation** — the teacher and student have the same architecture, \n", "but the teacher is a temporally averaged (EMA) version of the student.\n", "\n", "This is different from traditional knowledge distillation where:\n", "- Teacher is a **larger, pre-trained** model\n", "- Student is a **smaller** model being trained\n", "\n", "In DINO/DINOv2:\n", "- Teacher = EMA of student (same architecture)\n", "- Teacher provides **pseudo-labels** (soft probability distributions)\n", "- Student learns to match teacher's output on different views\n", "- This creates a **bootstrapping** effect: better student → better teacher → better student" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize the bootstrapping effect ────────────────────────────────────────\n", "fig, ax = plt.subplots(figsize=(12, 6))\n", "ax.set_xlim(0, 12); ax.set_ylim(0, 8)\n", "ax.axis('off')\n", "\n", "# Draw boxes\n", "def box(ax, x, y, w, h, color, text, fontsize=11):\n", " ax.add_patch(plt.FancyBboxPatch((x, y), w, h, boxstyle='round,pad=0.1',\n", " facecolor=color+'33', edgecolor=color, linewidth=2))\n", " ax.text(x+w/2, y+h/2, text, ha='center', va='center', fontsize=fontsize,\n", " fontweight='bold', color=color, multialignment='center')\n", "\n", "def arrow(ax, x1, y1, x2, y2, label='', color='black'):\n", " ax.annotate('', xy=(x2, y2), xytext=(x1, y1),\n", " arrowprops=dict(arrowstyle='->', color=color, lw=2))\n", " if label:\n", " mx, my = (x1+x2)/2, (y1+y2)/2\n", " ax.text(mx, my+0.2, label, ha='center', fontsize=9, color=color)\n", "\n", "# Image\n", "box(ax, 0.2, 3.5, 1.5, 1.5, '#555555', 'Image\\nX', 10)\n", "\n", "# Augmentations\n", "box(ax, 2.5, 5.0, 1.5, 1.0, '#3498db', 'Global\\nCrop 1', 9)\n", "box(ax, 2.5, 3.5, 1.5, 1.0, '#3498db', 'Global\\nCrop 2', 9)\n", "box(ax, 2.5, 2.0, 1.5, 1.0, '#e67e22', 'Local\\nCrop', 9)\n", "\n", "# Student\n", "box(ax, 5.0, 2.5, 2.0, 2.5, '#e74c3c', 'Student\\nViT\\n(trained)', 10)\n", "\n", "# Teacher\n", "box(ax, 5.0, 5.5, 2.0, 1.5, '#2ecc71', 'Teacher\\nViT\\n(EMA)', 10)\n", "\n", "# Heads\n", "box(ax, 7.5, 2.5, 1.5, 1.0, '#e74c3c', 'Student\\nHead', 9)\n", "box(ax, 7.5, 5.5, 1.5, 1.0, '#2ecc71', 'Teacher\\nHead', 9)\n", "\n", "# Loss\n", "box(ax, 9.5, 3.5, 2.0, 1.5, '#9b59b6', 'Cross-\\nEntropy\\nLoss', 10)\n", "\n", "# Arrows\n", "arrow(ax, 1.7, 4.25, 2.5, 5.5, '', '#3498db')\n", "arrow(ax, 1.7, 4.25, 2.5, 4.0, '', '#3498db')\n", "arrow(ax, 1.7, 4.25, 2.5, 2.5, '', '#e67e22')\n", "\n", "arrow(ax, 4.0, 5.5, 5.0, 6.2, 'global only', '#2ecc71')\n", "arrow(ax, 4.0, 4.0, 5.0, 3.5, 'all crops', '#e74c3c')\n", "arrow(ax, 4.0, 2.5, 5.0, 3.0, '', '#e74c3c')\n", "\n", "arrow(ax, 7.0, 3.5, 7.5, 3.0, '', '#e74c3c')\n", "arrow(ax, 7.0, 6.2, 7.5, 6.0, '', '#2ecc71')\n", "\n", "arrow(ax, 9.0, 3.0, 9.5, 4.0, '', '#e74c3c')\n", "arrow(ax, 9.0, 6.0, 9.5, 5.0, '', '#2ecc71')\n", "\n", "# EMA arrow (curved)\n", "ax.annotate('', xy=(6.0, 5.5), xytext=(6.0, 5.0),\n", " arrowprops=dict(arrowstyle='->', color='purple', lw=2,\n", " connectionstyle='arc3,rad=0.5'))\n", "ax.text(4.2, 5.2, 'EMA update\\nθ_t ← λθ_t + (1-λ)θ_s', fontsize=9, color='purple',\n", " ha='center', style='italic')\n", "\n", "ax.text(6, 0.5, 'Backprop through student only (teacher has no gradient)',\n", " ha='center', fontsize=10, color='gray', style='italic')\n", "\n", "ax.set_title('DINO Self-Distillation Framework', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.7 Why DINOv2 Features Are Special\n", "\n", "DINOv2 produces features with remarkable properties:\n", "\n", "1. **Semantic segmentation emerges** from patch features without any segmentation supervision\n", "2. **Depth estimation** correlates with patch feature distances\n", "3. **Object parts** are consistently represented across instances\n", "4. **Linear probing** achieves 86.5% on ImageNet-1K (ViT-g)\n", "5. **k-NN classification** achieves 83.5% on ImageNet-1K (ViT-g)\n", "\n", "The key insight: **the model learns to separate semantic content from style/texture**, \n", "because the augmentations change style but not semantics." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Demonstrate feature quality with real model ────────────────────────────────\n", "import torch\n", "from transformers import AutoImageProcessor, AutoModel\n", "\n", "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n", "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n", "\n", "# Create two semantically similar and one different image\n", "def make_image(color1, color2, size=224):\n", " img = Image.new('RGB', (size, size), color1)\n", " draw = ImageDraw.Draw(img)\n", " draw.ellipse([40, 40, 184, 184], fill=color2)\n", " return img\n", "\n", "img_a1 = make_image((200, 230, 200), (50, 150, 50)) # green circle on light green\n", "img_a2 = make_image((230, 200, 200), (150, 50, 50)) # red circle on light red (same structure)\n", "img_b = make_image((200, 200, 230), (50, 50, 200)) # blue circle on light blue\n", "img_c = make_image((240, 240, 200), (200, 200, 50)) # yellow square-ish\n", "\n", "images = [img_a1, img_a2, img_b, img_c]\n", "labels = ['Green Circle', 'Red Circle', 'Blue Circle', 'Yellow BG']\n", "\n", "# Extract CLS features\n", "feats = []\n", "for img in images:\n", " inp = processor(images=img, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " feats.append(out.last_hidden_state[0, 0].cpu().numpy())\n", "\n", "feats = np.array(feats) # [4, 768]\n", "\n", "# Cosine similarity matrix\n", "norms = np.linalg.norm(feats, axis=1, keepdims=True)\n", "feats_norm = feats / norms\n", "sim_matrix = feats_norm @ feats_norm.T\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n", "\n", "# Show images\n", "for i, (img, label) in enumerate(zip(images, labels)):\n", " ax_inset = fig.add_axes([0.02 + i*0.12, 0.55, 0.10, 0.35])\n", " ax_inset.imshow(img)\n", " ax_inset.set_title(label, fontsize=8)\n", " ax_inset.axis('off')\n", "\n", "# Similarity matrix\n", "im = axes[0].imshow(sim_matrix, cmap='RdYlGn', vmin=0.5, vmax=1.0)\n", "axes[0].set_xticks(range(4)); axes[0].set_xticklabels(labels, rotation=30, ha='right')\n", "axes[0].set_yticks(range(4)); axes[0].set_yticklabels(labels)\n", "for i in range(4):\n", " for j in range(4):\n", " axes[0].text(j, i, f'{sim_matrix[i,j]:.3f}', ha='center', va='center', fontsize=10)\n", "plt.colorbar(im, ax=axes[0])\n", "axes[0].set_title('CLS Token Cosine Similarity\\n(DINOv2 features)', fontweight='bold')\n", "\n", "# Feature norms\n", "axes[1].bar(labels, np.linalg.norm(feats, axis=1), color=['green','red','blue','gold'])\n", "axes[1].set_title('CLS Token L2 Norm\\nper image', fontweight='bold')\n", "axes[1].set_ylabel('L2 Norm')\n", "axes[1].tick_params(axis='x', rotation=20)\n", "axes[1].grid(True, alpha=0.3)\n", "\n", "# Feature values (first 50 dims)\n", "for i, (feat, label, color) in enumerate(zip(feats, labels, ['green','red','blue','gold'])):\n", " axes[2].plot(feat[:50], alpha=0.7, label=label, color=color)\n", "axes[2].set_title('CLS Token Features\\n(first 50 of 768 dims)', fontweight='bold')\n", "axes[2].set_xlabel('Feature Dimension')\n", "axes[2].set_ylabel('Value')\n", "axes[2].legend(fontsize=9)\n", "axes[2].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('DINOv2 Feature Quality Demonstration', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.8 Summary\n", "\n", "| Concept | Key Takeaway |\n", "|---------|-------------|\n", "| SSL | Learn from data structure, no labels needed |\n", "| Invariance | Same image → same representation despite augmentations |\n", "| Collapse | Trivial solution: all same output — must be prevented |\n", "| DINO | Student/teacher EMA + centering + sharpening |\n", "| EMA | Teacher = temporally averaged student → stable target |\n", "| Centering | Subtract running mean → prevents one-dim collapse |\n", "| Sharpening | Low temperature → sharp teacher distribution → clear signal |\n", "| DINOv2 | DINO + iBOT + KoLeo + curated data → SOTA features |\n", "\n", "---\n", "**Next:** [`03_training_procedure.ipynb`](03_training_procedure.ipynb) — Deep dive into the full DINOv2 training procedure, loss functions, and multi-crop strategy." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }