{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DINOv2 Tutorial — Notebook 12: Downstream Tasks\n", "\n", "**Series:** DINOv2 Deep Dive | **Prev:** `11_image_similarity_retrieval.ipynb` | **Next:** `13_model_variants.ipynb`\n", "\n", "---\n", "\n", "## What You Will Learn\n", "- Linear probing: train a linear classifier on frozen DINOv2 features\n", "- Semantic segmentation using patch features + k-means\n", "- Dense feature matching between images\n", "- Depth estimation concept using patch feature distances\n", "- How DINOv2 compares to supervised models on downstream tasks\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "from PIL import Image, ImageDraw\n", "from transformers import AutoImageProcessor, AutoModel\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.svm import LinearSVC\n", "from sklearn.cluster import KMeans\n", "from sklearn.decomposition import PCA\n", "from sklearn.metrics import classification_report, confusion_matrix\n", "from sklearn.model_selection import cross_val_score\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n", "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n", "cfg = model.config\n", "n_grid = cfg.image_size // cfg.patch_size\n", "\n", "def extract_cls(image):\n", " inp = processor(images=image, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " return out.last_hidden_state[0, 0].cpu().numpy()\n", "\n", "def extract_patches(image):\n", " inp = processor(images=image, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " return out.last_hidden_state[0, 1:].cpu().numpy() # [256, 768]\n", "\n", "print(\"Model loaded.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.1 Linear Probing\n", "\n", "**Linear probing** evaluates the quality of frozen features by training only a linear classifier on top.\n", "\n", "- **Frozen backbone**: DINOv2 weights are NOT updated\n", "- **Linear head**: only a single linear layer is trained\n", "- **Why it matters**: if features are linearly separable, the backbone has learned good representations\n", "\n", "DINOv2 achieves **86.5% on ImageNet** with linear probing (ViT-g) — comparable to supervised models!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Create a synthetic classification dataset ──────────────────────────────────\n", "def make_class_image(class_id, variant=0, size=224):\n", " \"\"\"Create images for 5 classes with 10 variants each.\"\"\"\n", " np.random.seed(class_id * 100 + variant)\n", " img = Image.new('RGB', (size, size), (240, 240, 240))\n", " draw = ImageDraw.Draw(img)\n", " \n", " # Base color per class\n", " base_colors = [\n", " (200 + variant*3, 50, 50), # class 0: red\n", " (50, 50, 200 + variant*3), # class 1: blue\n", " (50, 180 + variant*3, 50), # class 2: green\n", " (200 + variant*3, 180, 50), # class 3: yellow\n", " (150 + variant*3, 50, 180), # class 4: purple\n", " ]\n", " color = base_colors[class_id]\n", " \n", " # Shape per class\n", " shapes = ['circle', 'square', 'triangle', 'diamond', 'cross']\n", " shape = shapes[class_id]\n", " \n", " # Random position offset\n", " dx = np.random.randint(-20, 20)\n", " dy = np.random.randint(-20, 20)\n", " cx, cy = size//2 + dx, size//2 + dy\n", " r = 70 + np.random.randint(-10, 10)\n", " \n", " if shape == 'circle':\n", " draw.ellipse([cx-r, cy-r, cx+r, cy+r], fill=color)\n", " elif shape == 'square':\n", " draw.rectangle([cx-r, cy-r, cx+r, cy+r], fill=color)\n", " elif shape == 'triangle':\n", " draw.polygon([(cx, cy-r), (cx-r, cy+r), (cx+r, cy+r)], fill=color)\n", " elif shape == 'diamond':\n", " draw.polygon([(cx, cy-r), (cx+r, cy), (cx, cy+r), (cx-r, cy)], fill=color)\n", " elif shape == 'cross':\n", " w = r // 3\n", " draw.rectangle([cx-w, cy-r, cx+w, cy+r], fill=color)\n", " draw.rectangle([cx-r, cy-w, cx+r, cy+w], fill=color)\n", " \n", " return img\n", "\n", "# Create dataset: 5 classes x 10 images = 50 total\n", "n_classes = 5\n", "n_per_class = 10\n", "class_names = ['Red Circle', 'Blue Square', 'Green Triangle', 'Yellow Diamond', 'Purple Cross']\n", "\n", "print(\"Extracting features for linear probing dataset...\")\n", "X_all = []\n", "y_all = []\n", "images_all = []\n", "\n", "for class_id in range(n_classes):\n", " for variant in range(n_per_class):\n", " img = make_class_image(class_id, variant)\n", " feat = extract_cls(img)\n", " X_all.append(feat)\n", " y_all.append(class_id)\n", " images_all.append(img)\n", "\n", "X_all = np.array(X_all) # [50, 768]\n", "y_all = np.array(y_all) # [50]\n", "print(f\"Dataset: {X_all.shape[0]} images, {X_all.shape[1]} features, {n_classes} classes\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Train and evaluate linear classifiers ─────────────────────────────────────\n", "classifiers = {\n", " 'Logistic Regression': LogisticRegression(max_iter=1000, C=1.0),\n", " 'Linear SVM': LinearSVC(max_iter=2000, C=1.0),\n", "}\n", "\n", "results = {}\n", "for name, clf in classifiers.items():\n", " scores = cross_val_score(clf, X_all, y_all, cv=5, scoring='accuracy')\n", " results[name] = scores\n", " print(f\"{name}: {scores.mean()*100:.1f}% +/- {scores.std()*100:.1f}%\")\n", "\n", "# Fit on all data for confusion matrix\n", "lr = LogisticRegression(max_iter=1000, C=1.0)\n", "lr.fit(X_all, y_all)\n", "y_pred = lr.predict(X_all)\n", "cm_matrix = confusion_matrix(y_all, y_pred)\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n", "\n", "# Cross-validation scores\n", "for i, (name, scores) in enumerate(results.items()):\n", " axes[0].bar(i, scores.mean()*100, yerr=scores.std()*100,\n", " color=['#3498db', '#e74c3c'][i], alpha=0.8, capsize=5,\n", " label=f'{name}\\n{scores.mean()*100:.1f}%')\n", "axes[0].set_title('Linear Probing Accuracy\\n(5-fold cross-validation)', fontweight='bold')\n", "axes[0].set_ylabel('Accuracy (%)')\n", "axes[0].set_xticks(range(len(results)))\n", "axes[0].set_xticklabels(list(results.keys()), rotation=15, ha='right')\n", "axes[0].set_ylim(0, 110)\n", "axes[0].legend(fontsize=9)\n", "axes[0].grid(True, alpha=0.3, axis='y')\n", "\n", "# Confusion matrix\n", "im = axes[1].imshow(cm_matrix, cmap='Blues')\n", "axes[1].set_xticks(range(n_classes))\n", "axes[1].set_xticklabels([c[:8] for c in class_names], rotation=30, ha='right')\n", "axes[1].set_yticks(range(n_classes))\n", "axes[1].set_yticklabels([c[:8] for c in class_names])\n", "for i in range(n_classes):\n", " for j in range(n_classes):\n", " axes[1].text(j, i, str(cm_matrix[i,j]), ha='center', va='center',\n", " fontsize=12, fontweight='bold',\n", " color='white' if cm_matrix[i,j] > cm_matrix.max()/2 else 'black')\n", "plt.colorbar(im, ax=axes[1])\n", "axes[1].set_title('Confusion Matrix\\n(Logistic Regression on DINOv2 features)', fontweight='bold')\n", "axes[1].set_xlabel('Predicted'); axes[1].set_ylabel('True')\n", "\n", "# PCA visualization of features\n", "pca = PCA(n_components=2)\n", "X_pca = pca.fit_transform(X_all)\n", "colors_cls = ['#e74c3c', '#3498db', '#2ecc71', '#f39c12', '#9b59b6']\n", "for class_id, (name, color) in enumerate(zip(class_names, colors_cls)):\n", " mask = y_all == class_id\n", " axes[2].scatter(X_pca[mask, 0], X_pca[mask, 1], c=color, s=80,\n", " label=name, zorder=3, edgecolors='black', linewidths=0.5)\n", "axes[2].set_title(f'PCA of DINOv2 CLS Features\\n({pca.explained_variance_ratio_.sum()*100:.1f}% variance)',\n", " fontweight='bold')\n", "axes[2].set_xlabel('PC1'); axes[2].set_ylabel('PC2')\n", "axes[2].legend(fontsize=8); axes[2].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('Linear Probing with DINOv2 Features', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.2 Semantic Segmentation via Patch Features\n", "\n", "DINOv2 patch features can be used for **unsupervised semantic segmentation** without any training:\n", "1. Extract patch features from the image\n", "2. Apply k-means clustering to group patches by semantic content\n", "3. Each cluster corresponds to a semantic region" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Semantic segmentation via patch features ───────────────────────────────────\n", "def make_scene(size=224):\n", " img = Image.new('RGB', (size, size), (135, 206, 235))\n", " draw = ImageDraw.Draw(img)\n", " draw.rectangle([0, size*2//3, size, size], fill=(80, 160, 80))\n", " draw.rectangle([20, size//3, size//2-10, size*2//3], fill=(180, 120, 80))\n", " draw.polygon([(0, size//3), (size//2, size//3), (size//4, size//6)], fill=(160, 60, 60))\n", " draw.rectangle([size//4-15, size//2, size//4+15, size*2//3], fill=(100, 60, 30))\n", " draw.rectangle([35, size//3+15, 75, size//3+55], fill=(200, 230, 255))\n", " draw.rectangle([size*3//4-15, size//2, size*3//4+15, size*2//3], fill=(100, 70, 40))\n", " draw.ellipse([size*3//4-45, size//4, size*3//4+45, size//2+10], fill=(50, 140, 50))\n", " draw.ellipse([size*3//4+5, 15, size*3//4+45, 55], fill=(255, 220, 50))\n", " return img\n", "\n", "scene = make_scene()\n", "patch_feats = extract_patches(scene) # [256, 768]\n", "\n", "k_values = [2, 3, 4, 5, 6]\n", "cluster_palettes = [\n", " ['#e74c3c', '#3498db'],\n", " ['#e74c3c', '#3498db', '#2ecc71'],\n", " ['#e74c3c', '#3498db', '#2ecc71', '#f39c12'],\n", " ['#e74c3c', '#3498db', '#2ecc71', '#f39c12', '#9b59b6'],\n", " ['#e74c3c', '#3498db', '#2ecc71', '#f39c12', '#9b59b6', '#1abc9c'],\n", "]\n", "\n", "fig, axes = plt.subplots(2, 6, figsize=(24, 8))\n", "\n", "# Original\n", "axes[0, 0].imshow(scene)\n", "axes[0, 0].set_title('Original Scene', fontweight='bold')\n", "axes[0, 0].axis('off')\n", "\n", "# PCA RGB\n", "pca3 = PCA(n_components=3)\n", "pca_feats = pca3.fit_transform(patch_feats)\n", "pca_rgb = pca_feats.copy()\n", "for i in range(3):\n", " pca_rgb[:, i] = (pca_rgb[:, i] - pca_rgb[:, i].min()) / (pca_rgb[:, i].max() - pca_rgb[:, i].min() + 1e-8)\n", "axes[1, 0].imshow(pca_rgb.reshape(n_grid, n_grid, 3))\n", "axes[1, 0].set_title('PCA RGB\\n(semantic map)', fontweight='bold')\n", "axes[1, 0].axis('off')\n", "\n", "for col, (k, palette) in enumerate(zip(k_values, cluster_palettes)):\n", " kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)\n", " labels = kmeans.fit_predict(patch_feats)\n", " label_grid = labels.reshape(n_grid, n_grid)\n", " \n", " # Patch-level segmentation\n", " seg_img = np.zeros((n_grid, n_grid, 3))\n", " for cluster_id, color_hex in enumerate(palette):\n", " r = int(color_hex[1:3], 16) / 255\n", " g = int(color_hex[3:5], 16) / 255\n", " b = int(color_hex[5:7], 16) / 255\n", " seg_img[label_grid == cluster_id] = [r, g, b]\n", " axes[0, col+1].imshow(seg_img, interpolation='nearest')\n", " axes[0, col+1].set_title(f'K-Means k={k}\\n(patch level)', fontweight='bold', fontsize=9)\n", " axes[0, col+1].axis('off')\n", " \n", " # Upsampled overlay\n", " label_up = np.array(Image.fromarray(label_grid.astype(np.uint8)).resize(\n", " (224, 224), Image.NEAREST))\n", " overlay = np.array(scene).astype(float) / 255.0\n", " color_layer = np.zeros_like(overlay)\n", " for cluster_id, color_hex in enumerate(palette):\n", " r = int(color_hex[1:3], 16) / 255\n", " g = int(color_hex[3:5], 16) / 255\n", " b = int(color_hex[5:7], 16) / 255\n", " color_layer[label_up == cluster_id] = [r, g, b]\n", " blended = (0.5 * overlay + 0.5 * color_layer).clip(0, 1)\n", " axes[1, col+1].imshow(blended)\n", " axes[1, col+1].set_title(f'K-Means k={k}\\n(overlay)', fontweight='bold', fontsize=9)\n", " axes[1, col+1].axis('off')\n", "\n", "plt.suptitle('Unsupervised Semantic Segmentation via DINOv2 Patch Features',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.3 Dense Feature Matching\n", "\n", "DINOv2 patch features enable **dense matching** between images — finding corresponding patches across different images." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Dense feature matching ─────────────────────────────────────────────────────\n", "def make_variant(base_img, brightness=1.0, flip=False):\n", " arr = np.array(base_img).astype(float)\n", " arr = np.clip(arr * brightness, 0, 255).astype(np.uint8)\n", " img = Image.fromarray(arr)\n", " if flip:\n", " img = img.transpose(Image.FLIP_LEFT_RIGHT)\n", " return img\n", "\n", "img_a = scene\n", "img_b = make_variant(scene, brightness=0.85, flip=False) # darker version\n", "\n", "feats_a = extract_patches(img_a) # [256, 768]\n", "feats_b = extract_patches(img_b) # [256, 768]\n", "\n", "# Normalize\n", "fa_norm = feats_a / (np.linalg.norm(feats_a, axis=1, keepdims=True) + 1e-8)\n", "fb_norm = feats_b / (np.linalg.norm(feats_b, axis=1, keepdims=True) + 1e-8)\n", "\n", "# Cross-image similarity\n", "cross_sim = fa_norm @ fb_norm.T # [256, 256]\n", "\n", "# Best match for each patch in A\n", "best_match = cross_sim.argmax(axis=1) # [256]\n", "best_sim = cross_sim.max(axis=1) # [256]\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n", "\n", "axes[0, 0].imshow(img_a)\n", "axes[0, 0].set_title('Image A (original)', fontweight='bold')\n", "axes[0, 0].axis('off')\n", "\n", "axes[0, 1].imshow(img_b)\n", "axes[0, 1].set_title('Image B (darker variant)', fontweight='bold')\n", "axes[0, 1].axis('off')\n", "\n", "# Best match similarity map\n", "best_sim_grid = best_sim.reshape(n_grid, n_grid)\n", "im2 = axes[0, 2].imshow(best_sim_grid, cmap='RdYlGn', vmin=0.8, vmax=1.0, interpolation='nearest')\n", "axes[0, 2].set_title('Best Match Similarity\\n(A->B, per patch)', fontweight='bold')\n", "axes[0, 2].axis('off')\n", "plt.colorbar(im2, ax=axes[0, 2], fraction=0.046)\n", "\n", "# Visualize correspondences for a few patches\n", "query_patches = [(4, 4), (4, 12), (12, 4), (12, 12), (8, 8)]\n", "colors_match = ['red', 'blue', 'green', 'orange', 'purple']\n", "\n", "img_a_arr = np.array(img_a)\n", "img_b_arr = np.array(img_b)\n", "combined = np.zeros((224, 224*2+20, 3), dtype=np.uint8)\n", "combined[:, :224] = img_a_arr\n", "combined[:, 224+20:] = img_b_arr\n", "combined_img = Image.fromarray(combined)\n", "draw = ImageDraw.Draw(combined_img)\n", "\n", "for (qr, qc), color in zip(query_patches, colors_match):\n", " q_idx = qr * n_grid + qc\n", " m_idx = best_match[q_idx]\n", " mr, mc = m_idx // n_grid, m_idx % n_grid\n", " ps = cfg.patch_size\n", " # Draw query box in A\n", " draw.rectangle([qc*ps, qr*ps, (qc+1)*ps, (qr+1)*ps], outline=color, width=3)\n", " # Draw match box in B\n", " draw.rectangle([224+20+mc*ps, mr*ps, 224+20+(mc+1)*ps, (mr+1)*ps], outline=color, width=3)\n", " # Draw line\n", " draw.line([(qc*ps+ps//2, qr*ps+ps//2),\n", " (224+20+mc*ps+ps//2, mr*ps+ps//2)], fill=color, width=2)\n", "\n", "axes[1, 0].imshow(combined_img)\n", "axes[1, 0].set_title('Dense Matching: A -> B\\n(colored boxes = matched patches)',\n", " fontweight='bold')\n", "axes[1, 0].axis('off')\n", "\n", "# Match displacement map\n", "match_rows = best_match // n_grid\n", "match_cols = best_match % n_grid\n", "query_rows = np.arange(256) // n_grid\n", "query_cols = np.arange(256) % n_grid\n", "dr = (match_rows - query_rows).reshape(n_grid, n_grid)\n", "dc = (match_cols - query_cols).reshape(n_grid, n_grid)\n", "displacement = np.sqrt(dr**2 + dc**2)\n", "\n", "im3 = axes[1, 1].imshow(displacement, cmap='hot', interpolation='nearest')\n", "axes[1, 1].set_title('Match Displacement\\n(lower = better match)', fontweight='bold')\n", "axes[1, 1].axis('off')\n", "plt.colorbar(im3, ax=axes[1, 1], fraction=0.046)\n", "\n", "# Cycle consistency\n", "# A->B->A: for each patch in A, find match in B, then find match back in A\n", "best_match_ba = (fb_norm @ fa_norm.T).argmax(axis=1) # B->A\n", "cycle_match = best_match_ba[best_match] # A->B->A\n", "cycle_correct = (cycle_match == np.arange(256)).reshape(n_grid, n_grid)\n", "\n", "axes[1, 2].imshow(cycle_correct.astype(float), cmap='RdYlGn', vmin=0, vmax=1,\n", " interpolation='nearest')\n", "axes[1, 2].set_title(f'Cycle Consistency (A->B->A)\\n'\n", " f'{cycle_correct.sum()}/{256} patches consistent '\n", " f'({cycle_correct.mean()*100:.1f}%)',\n", " fontweight='bold')\n", "axes[1, 2].axis('off')\n", "\n", "plt.suptitle('Dense Feature Matching with DINOv2 Patch Features',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.4 Depth Estimation Concept\n", "\n", "DINOv2 patch features correlate with depth — patches at similar depths tend to have similar features. \n", "This is used in **DPT (Dense Prediction Transformer)** for monocular depth estimation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Depth estimation concept ───────────────────────────────────────────────────\n", "# Create a synthetic depth-like image (gradient from near to far)\n", "def make_depth_scene(size=224):\n", " img = Image.new('RGB', (size, size), (135, 206, 235))\n", " draw = ImageDraw.Draw(img)\n", " # Ground plane (perspective: darker = farther)\n", " for y in range(size//2, size):\n", " brightness = int(80 + (y - size//2) / (size//2) * 80)\n", " draw.line([(0, y), (size, y)], fill=(brightness//2, brightness, brightness//2))\n", " # Objects at different depths\n", " # Far object (small, high up)\n", " draw.ellipse([80, 60, 120, 100], fill=(180, 80, 80))\n", " # Medium object\n", " draw.rectangle([60, 120, 120, 180], fill=(80, 80, 180))\n", " # Near object (large, low)\n", " draw.ellipse([130, 150, 210, 220], fill=(80, 180, 80))\n", " return img\n", "\n", "depth_scene = make_depth_scene()\n", "depth_patches = extract_patches(depth_scene) # [256, 768]\n", "\n", "# PCA of patch features\n", "pca3 = PCA(n_components=3)\n", "depth_pca = pca3.fit_transform(depth_patches)\n", "depth_pca_norm = depth_pca.copy()\n", "for i in range(3):\n", " depth_pca_norm[:, i] = (depth_pca_norm[:, i] - depth_pca_norm[:, i].min()) / \\\n", " (depth_pca_norm[:, i].max() - depth_pca_norm[:, i].min() + 1e-8)\n", "\n", "# Simulated depth map (ground truth for illustration)\n", "depth_gt = np.zeros((n_grid, n_grid))\n", "for r in range(n_grid):\n", " for c in range(n_grid):\n", " # Simple heuristic: lower in image = closer (larger depth value)\n", " depth_gt[r, c] = r / n_grid\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n", "\n", "axes[0, 0].imshow(depth_scene)\n", "axes[0, 0].set_title('Scene with Depth Cues', fontweight='bold')\n", "axes[0, 0].axis('off')\n", "\n", "im1 = axes[0, 1].imshow(depth_gt, cmap='plasma', interpolation='bilinear')\n", "axes[0, 1].set_title('Simulated Depth Map\\n(darker=far, brighter=near)', fontweight='bold')\n", "axes[0, 1].axis('off')\n", "plt.colorbar(im1, ax=axes[0, 1], fraction=0.046)\n", "\n", "axes[0, 2].imshow(depth_pca_norm.reshape(n_grid, n_grid, 3))\n", "axes[0, 2].set_title('PCA RGB of Patch Features\\n(colors encode semantic regions)', fontweight='bold')\n", "axes[0, 2].axis('off')\n", "\n", "# Correlation between PC1 and depth\n", "pc1 = depth_pca[:, 0].reshape(n_grid, n_grid)\n", "corr = np.corrcoef(pc1.flatten(), depth_gt.flatten())[0, 1]\n", "im3 = axes[1, 0].imshow(pc1, cmap='plasma', interpolation='bilinear')\n", "axes[1, 0].set_title(f'PC1 of Patch Features\\n(corr with depth = {corr:.3f})', fontweight='bold')\n", "axes[1, 0].axis('off')\n", "plt.colorbar(im3, ax=axes[1, 0], fraction=0.046)\n", "\n", "# Patch norm as depth proxy\n", "patch_norms = np.linalg.norm(depth_patches, axis=1).reshape(n_grid, n_grid)\n", "corr_norm = np.corrcoef(patch_norms.flatten(), depth_gt.flatten())[0, 1]\n", "im4 = axes[1, 1].imshow(patch_norms, cmap='plasma', interpolation='bilinear')\n", "axes[1, 1].set_title(f'Patch Feature Norm\\n(corr with depth = {corr_norm:.3f})', fontweight='bold')\n", "axes[1, 1].axis('off')\n", "plt.colorbar(im4, ax=axes[1, 1], fraction=0.046)\n", "\n", "# DPT architecture concept\n", "axes[1, 2].axis('off')\n", "axes[1, 2].text(0.5, 0.95, 'DPT: Dense Prediction Transformer', ha='center',\n", " fontsize=12, fontweight='bold', transform=axes[1,2].transAxes)\n", "dpt_text = [\n", " 'Architecture:',\n", " ' Image -> DINOv2 ViT backbone',\n", " ' Extract features from layers:',\n", " ' L3, L6, L9, L12',\n", " ' Reassemble + Fusion decoder',\n", " ' -> Dense depth prediction',\n", " '',\n", " 'Key insight:',\n", " ' Multi-scale features from',\n", " ' different ViT layers capture',\n", " ' both local and global depth cues',\n", " '',\n", " 'DINOv2 + DPT achieves SOTA',\n", " 'on NYU Depth v2 dataset',\n", "]\n", "for i, line in enumerate(dpt_text):\n", " axes[1, 2].text(0.05, 0.85 - i*0.065, line, transform=axes[1,2].transAxes,\n", " fontsize=9, va='top', family='monospace')\n", "\n", "plt.suptitle('Depth Estimation with DINOv2 Features (Concept)',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.5 DINOv2 Benchmark Results (from Paper)\n", "\n", "| Task | Dataset | DINOv2 ViT-g | Supervised ViT-g | Notes |\n", "|------|---------|--------------|------------------|-------|\n", "| Classification (linear) | ImageNet-1K | 86.5% | 87.0% | Near-supervised! |\n", "| Classification (k-NN) | ImageNet-1K | 83.5% | — | No training needed |\n", "| Segmentation | ADE20K | 53.0 mIoU | 54.0 mIoU | Frozen backbone |\n", "| Depth estimation | NYU Depth v2 | 0.279 RMSE | 0.271 RMSE | Frozen backbone |\n", "| Instance seg. | COCO | 58.4 AP | 59.1 AP | Frozen backbone |" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize benchmark comparison ────────────────────────────────────────────\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "tasks = ['ImageNet\\nLinear', 'ImageNet\\nk-NN', 'ADE20K\\nSeg', 'NYUd\\nDepth', 'COCO\\nInst Seg']\n", "dinov2_scores = [86.5, 83.5, 53.0, 72.1, 58.4] # normalized for comparison\n", "supervised_scores = [87.0, None, 54.0, 73.5, 59.1]\n", "prev_ssl_scores = [81.1, 78.2, 47.2, 68.0, 54.0] # previous SSL methods\n", "\n", "x = np.arange(len(tasks))\n", "width = 0.25\n", "\n", "bars1 = axes[0].bar(x - width, dinov2_scores, width, label='DINOv2 (SSL)',\n", " color='#2ecc71', alpha=0.9)\n", "bars2 = axes[0].bar(x, [s if s else 0 for s in supervised_scores], width,\n", " label='Supervised', color='#3498db', alpha=0.9)\n", "bars3 = axes[0].bar(x + width, prev_ssl_scores, width, label='Previous SSL',\n", " color='#e74c3c', alpha=0.9)\n", "\n", "for bar in bars1:\n", " axes[0].text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.3,\n", " f'{bar.get_height():.1f}', ha='center', fontsize=8, fontweight='bold')\n", "\n", "axes[0].set_xticks(x); axes[0].set_xticklabels(tasks)\n", "axes[0].set_ylabel('Score')\n", "axes[0].set_title('DINOv2 vs Supervised vs Previous SSL\\n(frozen backbone)', fontweight='bold')\n", "axes[0].legend(fontsize=9)\n", "axes[0].grid(True, alpha=0.3, axis='y')\n", "\n", "# Improvement over previous SSL\n", "improvements = [d - p for d, p in zip(dinov2_scores, prev_ssl_scores)]\n", "colors_imp = ['green' if i > 0 else 'red' for i in improvements]\n", "axes[1].bar(tasks, improvements, color=colors_imp, alpha=0.8)\n", "axes[1].axhline(0, color='black', linewidth=1)\n", "for i, (task, imp) in enumerate(zip(tasks, improvements)):\n", " axes[1].text(i, imp + 0.1 if imp > 0 else imp - 0.3,\n", " f'+{imp:.1f}' if imp > 0 else f'{imp:.1f}',\n", " ha='center', fontsize=10, fontweight='bold',\n", " color='green' if imp > 0 else 'red')\n", "axes[1].set_title('DINOv2 Improvement over Previous SSL Methods',\n", " fontweight='bold')\n", "axes[1].set_ylabel('Score Improvement')\n", "axes[1].grid(True, alpha=0.3, axis='y')\n", "\n", "plt.suptitle('DINOv2 Downstream Task Performance (from paper)',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12.6 Summary\n", "\n", "| Task | Method | DINOv2 Advantage |\n", "|------|--------|------------------|\n", "| Classification | Linear probe on CLS | Near-supervised accuracy |\n", "| Segmentation | K-means on patches | Zero-shot, no labels |\n", "| Dense matching | Patch cosine similarity | High cycle consistency |\n", "| Depth estimation | DPT decoder on patches | SOTA with frozen backbone |\n", "| Retrieval | k-NN on CLS features | Zero-shot, no training |\n", "\n", "**Key insight:** DINOv2 features are **universal** — the same frozen backbone works for classification, segmentation, depth, and retrieval without task-specific fine-tuning.\n", "\n", "---\n", "**Next:** [`13_model_variants.ipynb`](13_model_variants.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }