{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DINOv2 Tutorial — Notebook 9: CLS Token Analysis\n", "\n", "**Series:** DINOv2 Deep Dive | **Prev:** `08_patch_embeddings_spatial_features.ipynb` | **Next:** `10_attention_head_specialization.ipynb`\n", "\n", "---\n", "\n", "## What You Will Learn\n", "- The CLS token as a global image representation\n", "- Feature vector visualization and statistics\n", "- Cosine similarity between CLS tokens of different images\n", "- t-SNE and UMAP visualization of CLS features across image categories\n", "- How CLS features cluster by semantic content\n", "- Nearest-neighbor classification using CLS features\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "from PIL import Image, ImageDraw\n", "from transformers import AutoImageProcessor, AutoModel\n", "from sklearn.decomposition import PCA\n", "from sklearn.manifold import TSNE\n", "\n", "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n", "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n", "cfg = model.config\n", "\n", "def extract_cls(image):\n", " inp = processor(images=image, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " return out.last_hidden_state[0, 0].cpu().numpy() # [768]\n", "\n", "print(f\"CLS token dimension: {cfg.hidden_size}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.1 CLS Token: What Is It?\n", "\n", "The **CLS token** (Classification token) is:\n", "- A **learnable vector** prepended to the patch sequence\n", "- Position 0 in the sequence: `[CLS, patch_1, patch_2, ..., patch_256]`\n", "- After 12 transformer layers, it has **attended to all patches** via self-attention\n", "- It aggregates **global information** about the entire image\n", "- Used as the **image-level representation** for classification, retrieval, etc.\n", "\n", "**Why CLS and not average pooling?**\n", "- CLS can learn to selectively attend to important patches\n", "- Average pooling treats all patches equally\n", "- CLS is more flexible and powerful for global representation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Create a diverse set of synthetic images ───────────────────────────────────\n", "def make_image(bg_color, shape='circle', shape_color=(255,255,255), size=224):\n", " img = Image.new('RGB', (size, size), bg_color)\n", " draw = ImageDraw.Draw(img)\n", " if shape == 'circle':\n", " draw.ellipse([40, 40, 184, 184], fill=shape_color)\n", " elif shape == 'square':\n", " draw.rectangle([40, 40, 184, 184], fill=shape_color)\n", " elif shape == 'triangle':\n", " draw.polygon([(112, 30), (30, 194), (194, 194)], fill=shape_color)\n", " elif shape == 'cross':\n", " draw.rectangle([90, 30, 134, 194], fill=shape_color)\n", " draw.rectangle([30, 90, 194, 134], fill=shape_color)\n", " return img\n", "\n", "# Create image gallery: 4 categories × 5 images each\n", "categories = {\n", " 'Red Circle': [(make_image((200+i*5, 50, 50), 'circle', (255,255,255)), f'RC{i}') for i in range(5)],\n", " 'Blue Square': [(make_image((50, 50, 200+i*5), 'square', (255,255,255)), f'BS{i}') for i in range(5)],\n", " 'Green Triangle': [(make_image((50, 150+i*10, 50), 'triangle', (255,255,255)), f'GT{i}') for i in range(5)],\n", " 'Yellow Cross': [(make_image((200+i*5, 200+i*5, 50), 'cross', (255,255,255)), f'YC{i}') for i in range(5)],\n", "}\n", "\n", "# Extract CLS features\n", "all_features = []\n", "all_labels = []\n", "all_images = []\n", "label_to_idx = {label: i for i, label in enumerate(categories.keys())}\n", "\n", "for label, images in categories.items():\n", " for img, name in images:\n", " feat = extract_cls(img)\n", " all_features.append(feat)\n", " all_labels.append(label)\n", " all_images.append(img)\n", "\n", "all_features = np.array(all_features) # [20, 768]\n", "all_label_ids = np.array([label_to_idx[l] for l in all_labels])\n", "\n", "print(f\"Total images: {len(all_features)}\")\n", "print(f\"Feature shape: {all_features.shape}\")\n", "print(f\"Categories: {list(categories.keys())}\")\n", "\n", "# Show sample images\n", "fig, axes = plt.subplots(4, 5, figsize=(15, 12))\n", "for row, (label, images) in enumerate(categories.items()):\n", " for col, (img, name) in enumerate(images):\n", " axes[row, col].imshow(img)\n", " axes[row, col].set_title(f'{label}\\n{name}', fontsize=8)\n", " axes[row, col].axis('off')\n", "plt.suptitle('Image Gallery: 4 Categories × 5 Images', fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.2 CLS Token Feature Vector Visualization" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize CLS feature vectors ─────────────────────────────────────────────\n", "colors_cat = ['#e74c3c', '#3498db', '#2ecc71', '#f39c12']\n", "labels_cat = list(categories.keys())\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n", "\n", "# Feature values (first 100 dims) for one image per category\n", "for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " feat = all_features[i*5] # first image of each category\n", " axes[0, 0].plot(feat[:100], color=color, alpha=0.8, linewidth=1.5, label=label)\n", "axes[0, 0].set_title('CLS Token Feature Values\\n(first 100 of 768 dims)', fontweight='bold')\n", "axes[0, 0].set_xlabel('Feature Dimension')\n", "axes[0, 0].set_ylabel('Value')\n", "axes[0, 0].legend(fontsize=8)\n", "axes[0, 0].grid(True, alpha=0.3)\n", "\n", "# Feature distribution per category\n", "for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " feats_cat = all_features[i*5:(i+1)*5].flatten()\n", " axes[0, 1].hist(feats_cat, bins=40, alpha=0.5, color=color, label=label, density=True)\n", "axes[0, 1].set_title('CLS Feature Value Distribution\\nper Category', fontweight='bold')\n", "axes[0, 1].set_xlabel('Feature Value')\n", "axes[0, 1].set_ylabel('Density')\n", "axes[0, 1].legend(fontsize=8)\n", "axes[0, 1].grid(True, alpha=0.3)\n", "\n", "# L2 norms\n", "norms = np.linalg.norm(all_features, axis=1) # [20]\n", "for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " cat_norms = norms[i*5:(i+1)*5]\n", " axes[0, 2].scatter(range(i*5, (i+1)*5), cat_norms, c=color, s=80, label=label, zorder=3)\n", "axes[0, 2].set_title('CLS Token L2 Norm per Image', fontweight='bold')\n", "axes[0, 2].set_xlabel('Image index')\n", "axes[0, 2].set_ylabel('L2 Norm')\n", "axes[0, 2].legend(fontsize=8)\n", "axes[0, 2].grid(True, alpha=0.3)\n", "\n", "# Cosine similarity matrix\n", "feats_norm = all_features / (np.linalg.norm(all_features, axis=1, keepdims=True) + 1e-8)\n", "sim_matrix = feats_norm @ feats_norm.T # [20, 20]\n", "\n", "im = axes[1, 0].imshow(sim_matrix, cmap='RdYlGn', vmin=0.5, vmax=1.0)\n", "# Draw category boundaries\n", "for boundary in [4.5, 9.5, 14.5]:\n", " axes[1, 0].axhline(boundary, color='black', linewidth=2)\n", " axes[1, 0].axvline(boundary, color='black', linewidth=2)\n", "# Category labels\n", "for i, label in enumerate(labels_cat):\n", " axes[1, 0].text(i*5+2, -0.8, label[:8], ha='center', fontsize=7, rotation=30)\n", " axes[1, 0].text(-0.8, i*5+2, label[:8], ha='right', fontsize=7)\n", "plt.colorbar(im, ax=axes[1, 0])\n", "axes[1, 0].set_title('CLS Token Cosine Similarity Matrix\\n(20 images, 4 categories)',\n", " fontweight='bold')\n", "\n", "# Within vs between category similarity\n", "within_sims = []\n", "between_sims = []\n", "for i in range(20):\n", " cat_i = i // 5\n", " for j in range(i+1, 20):\n", " cat_j = j // 5\n", " if cat_i == cat_j:\n", " within_sims.append(sim_matrix[i, j])\n", " else:\n", " between_sims.append(sim_matrix[i, j])\n", "\n", "axes[1, 1].hist(within_sims, bins=20, alpha=0.7, color='green', label=f'Within category\\n(mean={np.mean(within_sims):.3f})')\n", "axes[1, 1].hist(between_sims, bins=20, alpha=0.7, color='red', label=f'Between category\\n(mean={np.mean(between_sims):.3f})')\n", "axes[1, 1].set_title('Within vs Between Category\\nCosine Similarity', fontweight='bold')\n", "axes[1, 1].set_xlabel('Cosine Similarity')\n", "axes[1, 1].set_ylabel('Count')\n", "axes[1, 1].legend(fontsize=9)\n", "axes[1, 1].grid(True, alpha=0.3)\n", "\n", "# Mean similarity per category pair\n", "mean_sim_matrix = np.zeros((4, 4))\n", "for i in range(4):\n", " for j in range(4):\n", " block = sim_matrix[i*5:(i+1)*5, j*5:(j+1)*5]\n", " if i == j:\n", " # Within category: exclude diagonal\n", " mask = ~np.eye(5, dtype=bool)\n", " mean_sim_matrix[i, j] = block[mask].mean()\n", " else:\n", " mean_sim_matrix[i, j] = block.mean()\n", "\n", "im2 = axes[1, 2].imshow(mean_sim_matrix, cmap='RdYlGn', vmin=0.5, vmax=1.0)\n", "short_labels = ['RedCirc', 'BlueSq', 'GreenTri', 'YellCross']\n", "axes[1, 2].set_xticks(range(4)); axes[1, 2].set_xticklabels(short_labels, rotation=30, ha='right', fontsize=8)\n", "axes[1, 2].set_yticks(range(4)); axes[1, 2].set_yticklabels(short_labels, fontsize=8)\n", "for i in range(4):\n", " for j in range(4):\n", " axes[1, 2].text(j, i, f'{mean_sim_matrix[i,j]:.3f}', ha='center', va='center', fontsize=10)\n", "plt.colorbar(im2, ax=axes[1, 2])\n", "axes[1, 2].set_title('Mean Similarity per Category Pair', fontweight='bold')\n", "\n", "plt.suptitle('CLS Token Feature Analysis', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.3 t-SNE Visualization of CLS Features" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── t-SNE visualization ────────────────────────────────────────────────────────\n", "# PCA first to reduce to 50 dims (speeds up t-SNE)\n", "pca_50 = PCA(n_components=min(50, len(all_features)-1))\n", "feats_pca = pca_50.fit_transform(all_features)\n", "\n", "# t-SNE\n", "tsne = TSNE(n_components=2, perplexity=5, random_state=42, n_iter=1000)\n", "feats_tsne = tsne.fit_transform(feats_pca) # [20, 2]\n", "\n", "# PCA 2D\n", "pca_2 = PCA(n_components=2)\n", "feats_pca2 = pca_2.fit_transform(all_features) # [20, 2]\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(16, 7))\n", "\n", "for ax, (feats_2d, title) in zip(axes, [\n", " (feats_pca2, f'PCA (2D)\\n({pca_2.explained_variance_ratio_.sum()*100:.1f}% variance)'),\n", " (feats_tsne, 't-SNE (2D)\\n(perplexity=5)'),\n", "]):\n", " for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " mask = all_label_ids == i\n", " ax.scatter(feats_2d[mask, 0], feats_2d[mask, 1],\n", " c=color, s=150, label=label, zorder=3,\n", " edgecolors='black', linewidths=1)\n", " # Add image index labels\n", " for j, (x, y) in enumerate(feats_2d[mask]):\n", " ax.annotate(f'{j+1}', (x, y), ha='center', va='center',\n", " fontsize=7, fontweight='bold', color='white')\n", " ax.set_title(title, fontweight='bold', fontsize=12)\n", " ax.legend(fontsize=9)\n", " ax.grid(True, alpha=0.3)\n", " ax.set_xlabel('Component 1')\n", " ax.set_ylabel('Component 2')\n", "\n", "plt.suptitle('CLS Token Feature Space Visualization\\n'\n", " 'Same-category images cluster together',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.4 UMAP Visualization (if available)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── UMAP visualization ────────────────────────────────────────────────────────\n", "try:\n", " import umap\n", " reducer = umap.UMAP(n_components=2, n_neighbors=5, min_dist=0.1, random_state=42)\n", " feats_umap = reducer.fit_transform(all_features)\n", " \n", " fig, ax = plt.subplots(figsize=(10, 8))\n", " for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " mask = all_label_ids == i\n", " ax.scatter(feats_umap[mask, 0], feats_umap[mask, 1],\n", " c=color, s=150, label=label, zorder=3,\n", " edgecolors='black', linewidths=1)\n", " for j, (x, y) in enumerate(feats_umap[mask]):\n", " ax.annotate(f'{j+1}', (x, y), ha='center', va='center',\n", " fontsize=7, fontweight='bold', color='white')\n", " ax.set_title('UMAP of CLS Token Features\\n(n_neighbors=5, min_dist=0.1)',\n", " fontweight='bold', fontsize=13)\n", " ax.legend(fontsize=10)\n", " ax.grid(True, alpha=0.3)\n", " ax.set_xlabel('UMAP 1'); ax.set_ylabel('UMAP 2')\n", " plt.tight_layout()\n", " plt.show()\n", " print(\"UMAP visualization complete.\")\n", "except ImportError:\n", " print(\"umap-learn not installed. Install with: pip install umap-learn\")\n", " print(\"Showing t-SNE instead (already computed above).\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.5 Nearest-Neighbor Classification with CLS Features" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── k-NN classification ────────────────────────────────────────────────────────\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.model_selection import cross_val_score\n", "\n", "# Use leave-one-out cross-validation\n", "knn = KNeighborsClassifier(n_neighbors=3, metric='cosine')\n", "scores = cross_val_score(knn, all_features, all_label_ids, cv=5)\n", "print(f\"k-NN (k=3) cross-validation accuracy: {scores.mean()*100:.1f}% ± {scores.std()*100:.1f}%\")\n", "\n", "# Fit on all data and show confusion matrix\n", "knn.fit(all_features, all_label_ids)\n", "predictions = knn.predict(all_features)\n", "\n", "# Confusion matrix\n", "from sklearn.metrics import confusion_matrix\n", "cm_matrix = confusion_matrix(all_label_ids, predictions)\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "im = axes[0].imshow(cm_matrix, cmap='Blues')\n", "axes[0].set_xticks(range(4)); axes[0].set_xticklabels(short_labels, rotation=30, ha='right')\n", "axes[0].set_yticks(range(4)); axes[0].set_yticklabels(short_labels)\n", "for i in range(4):\n", " for j in range(4):\n", " axes[0].text(j, i, str(cm_matrix[i,j]), ha='center', va='center',\n", " fontsize=14, fontweight='bold',\n", " color='white' if cm_matrix[i,j] > cm_matrix.max()/2 else 'black')\n", "plt.colorbar(im, ax=axes[0])\n", "axes[0].set_title('k-NN Confusion Matrix\\n(k=3, cosine distance)', fontweight='bold')\n", "axes[0].set_xlabel('Predicted'); axes[0].set_ylabel('True')\n", "\n", "# Distance to nearest neighbors\n", "distances, indices = knn.kneighbors(all_features, n_neighbors=4) # 4 = self + 3 neighbors\n", "# Skip self (distance=0)\n", "nn_distances = distances[:, 1:] # [20, 3]\n", "\n", "for i, (label, color) in enumerate(zip(labels_cat, colors_cat)):\n", " cat_dists = nn_distances[i*5:(i+1)*5].mean(axis=1)\n", " axes[1].scatter(range(i*5, (i+1)*5), cat_dists, c=color, s=80, label=label, zorder=3)\n", "axes[1].set_title('Mean Distance to 3 Nearest Neighbors\\n(lower = more similar to neighbors)',\n", " fontweight='bold')\n", "axes[1].set_xlabel('Image index')\n", "axes[1].set_ylabel('Mean cosine distance')\n", "axes[1].legend(fontsize=8)\n", "axes[1].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('k-NN Classification with DINOv2 CLS Features', fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.6 CLS Token vs Average Pooling" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Compare CLS token vs average pooling ──────────────────────────────────────\n", "def extract_avg_pool(image):\n", " \"\"\"Extract average-pooled patch features.\"\"\"\n", " inp = processor(images=image, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " # Average over all patch tokens (exclude CLS)\n", " return out.last_hidden_state[0, 1:].mean(dim=0).cpu().numpy() # [768]\n", "\n", "# Extract both representations\n", "cls_feats = np.array([extract_cls(img) for img, _ in sum(categories.values(), [])])\n", "avg_feats = np.array([extract_avg_pool(img) for img, _ in sum(categories.values(), [])])\n", "\n", "# Compute within/between category similarity for both\n", "def compute_sim_stats(feats, label_ids):\n", " norms = np.linalg.norm(feats, axis=1, keepdims=True)\n", " feats_n = feats / (norms + 1e-8)\n", " sim = feats_n @ feats_n.T\n", " within, between = [], []\n", " for i in range(len(feats)):\n", " for j in range(i+1, len(feats)):\n", " if label_ids[i] == label_ids[j]:\n", " within.append(sim[i,j])\n", " else:\n", " between.append(sim[i,j])\n", " return np.array(within), np.array(between)\n", "\n", "cls_within, cls_between = compute_sim_stats(cls_feats, all_label_ids)\n", "avg_within, avg_between = compute_sim_stats(avg_feats, all_label_ids)\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "# Similarity distributions\n", "axes[0].hist(cls_within, bins=15, alpha=0.7, color='green', label=f'CLS within (μ={cls_within.mean():.3f})')\n", "axes[0].hist(cls_between, bins=15, alpha=0.7, color='red', label=f'CLS between (μ={cls_between.mean():.3f})')\n", "axes[0].set_title('CLS Token: Within vs Between Category Similarity', fontweight='bold')\n", "axes[0].set_xlabel('Cosine Similarity'); axes[0].set_ylabel('Count')\n", "axes[0].legend(fontsize=9); axes[0].grid(True, alpha=0.3)\n", "\n", "axes[1].hist(avg_within, bins=15, alpha=0.7, color='green', label=f'AvgPool within (μ={avg_within.mean():.3f})')\n", "axes[1].hist(avg_between, bins=15, alpha=0.7, color='red', label=f'AvgPool between (μ={avg_between.mean():.3f})')\n", "axes[1].set_title('Average Pooling: Within vs Between Category Similarity', fontweight='bold')\n", "axes[1].set_xlabel('Cosine Similarity'); axes[1].set_ylabel('Count')\n", "axes[1].legend(fontsize=9); axes[1].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('CLS Token vs Average Pooling: Discriminability Comparison',\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "# Separability score\n", "cls_sep = cls_within.mean() - cls_between.mean()\n", "avg_sep = avg_within.mean() - avg_between.mean()\n", "print(f\"Separability (within - between similarity):\")\n", "print(f\" CLS token: {cls_sep:.4f}\")\n", "print(f\" Average pooling:{avg_sep:.4f}\")\n", "print(f\" Winner: {'CLS token' if cls_sep > avg_sep else 'Average pooling'}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9.7 Summary\n", "\n", "| Property | Value |\n", "|----------|-------|\n", "| CLS token dimension | 768 |\n", "| Position in sequence | 0 (first token) |\n", "| How it aggregates | Self-attention over all patches |\n", "| Use case | Classification, retrieval, similarity |\n", "| Within-category similarity | High (similar images → similar CLS) |\n", "| Between-category similarity | Low (different images → different CLS) |\n", "\n", "**Key insight:** DINOv2 CLS features are **linearly separable** for many tasks — a linear classifier on top achieves 86.5% on ImageNet (ViT-g).\n", "\n", "---\n", "**Next:** [`10_attention_head_specialization.ipynb`](10_attention_head_specialization.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }