{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DINOv2 Tutorial — Notebook 4: Dataset Preparation (LVD-142M)\n", "\n", "**Series:** DINOv2 Deep Dive | **Prev:** `03_training_procedure.ipynb` | **Next:** `05_vit_architecture.ipynb`\n", "\n", "---\n", "\n", "## What You Will Learn\n", "- Why data curation matters for self-supervised learning\n", "- The LVD-142M dataset: how 142M images were curated from 1.2B web images\n", "- Deduplication via copy detection\n", "- Retrieval-based curation pipeline\n", "- How to simulate the curation process with DINOv2 features\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Why Data Curation Matters\n", "\n", "The DINOv2 paper's key insight: **data quality matters more than data quantity** for SSL.\n", "\n", "### Problem with Raw Web Data\n", "- Web images contain **duplicates** (same image, different URLs)\n", "- **Near-duplicates** (same image, different crops/watermarks)\n", "- **Low-quality images** (blurry, corrupted, irrelevant)\n", "- **Imbalanced distribution** (some concepts over-represented)\n", "- **NSFW content** and other undesirable images\n", "\n", "### DINOv2 Solution: LVD-142M\n", "- Start with **1.2 billion** raw web images\n", "- Apply **deduplication** to remove near-duplicates\n", "- Apply **retrieval-based curation** to select semantically relevant images\n", "- Result: **142 million** high-quality, diverse images\n", "\n", "```\n", "1.2B raw web images\n", " │\n", " ▼\n", " Deduplication (copy detection)\n", " │\n", " ▼\n", " ~500M deduplicated images\n", " │\n", " ▼\n", " Retrieval-based curation\n", " (using curated seed images)\n", " │\n", " ▼\n", " LVD-142M (142M curated images)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as mpatches\n", "from PIL import Image, ImageDraw, ImageFilter\n", "import torch\n", "from transformers import AutoImageProcessor, AutoModel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Deduplication via Copy Detection\n", "\n", "### How it works:\n", "1. Extract features from all images using a pre-trained model\n", "2. Build an approximate nearest-neighbor index (e.g., FAISS)\n", "3. For each image, find its nearest neighbors\n", "4. If cosine similarity > threshold → mark as duplicate\n", "5. Keep only one image from each duplicate cluster\n", "\n", "### Why cosine similarity?\n", "- Invariant to image brightness/contrast changes\n", "- Robust to minor crops and resizing\n", "- Fast to compute with normalized vectors" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Simulate deduplication ─────────────────────────────────────────────────────\n", "def make_image_variants(base_color, size=64):\n", " \"\"\"Create an original image and several near-duplicates.\"\"\"\n", " variants = []\n", " \n", " # Original\n", " img = Image.new('RGB', (size, size), base_color)\n", " draw = ImageDraw.Draw(img)\n", " draw.ellipse([10, 10, 54, 54], fill=(255, 255, 255))\n", " draw.rectangle([20, 20, 44, 44], fill=base_color)\n", " variants.append(('Original', img))\n", " \n", " # Slightly cropped\n", " cropped = img.crop((4, 4, 60, 60)).resize((size, size), Image.BILINEAR)\n", " variants.append(('Cropped', cropped))\n", " \n", " # Brightness adjusted\n", " arr = np.array(img).astype(float)\n", " bright = Image.fromarray(np.clip(arr * 1.3, 0, 255).astype(np.uint8))\n", " variants.append(('Brighter', bright))\n", " \n", " # Flipped\n", " flipped = img.transpose(Image.FLIP_LEFT_RIGHT)\n", " variants.append(('Flipped', flipped))\n", " \n", " # Slightly blurred\n", " blurred = img.filter(ImageFilter.GaussianBlur(radius=1))\n", " variants.append(('Blurred', blurred))\n", " \n", " return variants\n", "\n", "# Create image groups\n", "group_a = make_image_variants((180, 60, 60)) # red-ish\n", "group_b = make_image_variants((60, 120, 180)) # blue-ish\n", "group_c = make_image_variants((60, 160, 80)) # green-ish\n", "\n", "# Show the variants\n", "fig, axes = plt.subplots(3, 5, figsize=(15, 9))\n", "for row, (group, color) in enumerate(zip([group_a, group_b, group_c],\n", " ['red', 'blue', 'green'])):\n", " for col, (name, img) in enumerate(group):\n", " axes[row, col].imshow(img)\n", " axes[row, col].set_title(f'Group {chr(65+row)}\\n{name}', fontsize=9)\n", " axes[row, col].axis('off')\n", " for spine in axes[row, col].spines.values():\n", " spine.set_edgecolor(color)\n", " spine.set_linewidth(3)\n", "\n", "plt.suptitle('Near-Duplicate Image Groups\\n(same content, different transformations)',\n", " fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Extract features and compute similarity matrix ─────────────────────────────\n", "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n", "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n", "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n", "\n", "# Collect all images\n", "all_images = []\n", "all_labels = []\n", "for group_name, group in [('A', group_a), ('B', group_b), ('C', group_c)]:\n", " for variant_name, img in group:\n", " all_images.append(img.resize((224, 224), Image.BILINEAR))\n", " all_labels.append(f'{group_name}:{variant_name}')\n", "\n", "# Extract CLS features\n", "features = []\n", "for img in all_images:\n", " inp = processor(images=img, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " feat = out.last_hidden_state[0, 0].cpu().numpy()\n", " features.append(feat)\n", "\n", "features = np.array(features) # [15, 768]\n", "\n", "# Cosine similarity matrix\n", "norms = np.linalg.norm(features, axis=1, keepdims=True)\n", "feat_norm = features / norms\n", "sim_matrix = feat_norm @ feat_norm.T\n", "\n", "# Plot similarity matrix\n", "fig, axes = plt.subplots(1, 2, figsize=(16, 7))\n", "\n", "im = axes[0].imshow(sim_matrix, cmap='RdYlGn', vmin=0.5, vmax=1.0)\n", "axes[0].set_xticks(range(15))\n", "axes[0].set_xticklabels(all_labels, rotation=45, ha='right', fontsize=8)\n", "axes[0].set_yticks(range(15))\n", "axes[0].set_yticklabels(all_labels, fontsize=8)\n", "for i in range(15):\n", " for j in range(15):\n", " axes[0].text(j, i, f'{sim_matrix[i,j]:.2f}', ha='center', va='center', fontsize=7)\n", "plt.colorbar(im, ax=axes[0])\n", "# Draw group boundaries\n", "for boundary in [4.5, 9.5]:\n", " axes[0].axhline(boundary, color='black', linewidth=2)\n", " axes[0].axvline(boundary, color='black', linewidth=2)\n", "axes[0].set_title('Cosine Similarity Matrix\\n(DINOv2 CLS features)', fontweight='bold')\n", "\n", "# Deduplication threshold\n", "threshold = 0.92\n", "is_duplicate = (sim_matrix > threshold) & (np.eye(15) == 0)\n", "\n", "# Show which images would be removed\n", "keep = np.ones(15, dtype=bool)\n", "for i in range(15):\n", " if keep[i]:\n", " for j in range(i+1, 15):\n", " if is_duplicate[i, j]:\n", " keep[j] = False\n", "\n", "colors = ['green' if k else 'red' for k in keep]\n", "axes[1].bar(range(15), sim_matrix.max(axis=1) * (1 - np.eye(15)).max(axis=1),\n", " color=colors, alpha=0.8)\n", "axes[1].axhline(threshold, color='black', linestyle='--', linewidth=2,\n", " label=f'Dedup threshold = {threshold}')\n", "axes[1].set_xticks(range(15))\n", "axes[1].set_xticklabels(all_labels, rotation=45, ha='right', fontsize=8)\n", "axes[1].set_title(f'Max Similarity to Other Images\\nGreen=Keep, Red=Remove (threshold={threshold})',\n", " fontweight='bold')\n", "axes[1].set_ylabel('Max Cosine Similarity')\n", "axes[1].legend()\n", "axes[1].grid(True, alpha=0.3)\n", "\n", "plt.suptitle('Copy Detection / Deduplication with DINOv2 Features', fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print(f\"Total images: {len(all_images)}\")\n", "print(f\"After deduplication: {keep.sum()} images kept\")\n", "print(f\"Removed as duplicates: {(~keep).sum()} images\")\n", "print(f\"Kept images: {[all_labels[i] for i in range(15) if keep[i]]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.3 Retrieval-Based Curation\n", "\n", "After deduplication, DINOv2 uses **retrieval-based curation** to select high-quality images:\n", "\n", "### Pipeline:\n", "1. **Curated seed set**: Start with high-quality labeled datasets (ImageNet, Google Landmarks, etc.)\n", "2. **Feature extraction**: Extract features from all seed images using a pre-trained model\n", "3. **Build index**: Create a FAISS index of seed image features\n", "4. **Query uncurated pool**: For each uncurated image, find its k nearest neighbors in the seed set\n", "5. **Select by distance**: Keep uncurated images that are close to seed images\n", "6. **Balance**: Ensure diverse coverage across semantic categories\n", "\n", "### Why this works:\n", "- Seed images are high-quality and semantically meaningful\n", "- Retrieved images are semantically similar to seeds\n", "- This filters out irrelevant/low-quality web images\n", "- The result is a curated set that covers the same semantic space as the seeds" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Simulate retrieval-based curation ─────────────────────────────────────────\n", "np.random.seed(42)\n", "\n", "# Simulate feature space (2D for visualization)\n", "# Seed images: 3 clusters (animals, vehicles, nature)\n", "n_seeds_per_class = 20\n", "seed_centers = np.array([[2, 2], [-2, 2], [0, -2]])\n", "seed_labels = ['Animals', 'Vehicles', 'Nature']\n", "seed_colors = ['#e74c3c', '#3498db', '#2ecc71']\n", "\n", "seeds = []\n", "seed_class = []\n", "for i, center in enumerate(seed_centers):\n", " pts = np.random.randn(n_seeds_per_class, 2) * 0.5 + center\n", " seeds.append(pts)\n", " seed_class.extend([i] * n_seeds_per_class)\n", "seeds = np.vstack(seeds)\n", "\n", "# Uncurated pool: mix of relevant and irrelevant images\n", "n_uncurated = 500\n", "uncurated = np.random.randn(n_uncurated, 2) * 3 # spread across space\n", "\n", "# Compute distance from each uncurated image to nearest seed\n", "dists_to_seeds = np.sqrt(((uncurated[:, None] - seeds[None, :]) ** 2).sum(axis=-1))\n", "min_dist_to_seed = dists_to_seeds.min(axis=1)\n", "nearest_seed_class = np.array(seed_class)[dists_to_seeds.argmin(axis=1)]\n", "\n", "# Curation threshold\n", "threshold = 1.5\n", "selected = min_dist_to_seed < threshold\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n", "\n", "# Seed images\n", "ax = axes[0]\n", "for i, (center, label, color) in enumerate(zip(seed_centers, seed_labels, seed_colors)):\n", " mask = np.array(seed_class) == i\n", " ax.scatter(seeds[mask, 0], seeds[mask, 1], c=color, s=60, label=label,\n", " marker='*', zorder=3, edgecolors='black', linewidths=0.5)\n", "ax.set_title('Curated Seed Images\\n(high-quality labeled data)', fontweight='bold')\n", "ax.legend(); ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Feature Dim 1'); ax.set_ylabel('Feature Dim 2')\n", "ax.set_xlim(-6, 6); ax.set_ylim(-6, 6)\n", "\n", "# Uncurated pool\n", "ax = axes[1]\n", "ax.scatter(uncurated[:, 0], uncurated[:, 1], c='gray', s=10, alpha=0.4, label='Uncurated')\n", "for i, (center, label, color) in enumerate(zip(seed_centers, seed_labels, seed_colors)):\n", " mask = np.array(seed_class) == i\n", " ax.scatter(seeds[mask, 0], seeds[mask, 1], c=color, s=60, label=label,\n", " marker='*', zorder=3, edgecolors='black', linewidths=0.5)\n", "# Draw threshold circles\n", "theta = np.linspace(0, 2*np.pi, 100)\n", "for center, color in zip(seed_centers, seed_colors):\n", " ax.plot(center[0] + threshold*np.cos(theta),\n", " center[1] + threshold*np.sin(theta),\n", " '--', color=color, alpha=0.5, linewidth=1.5)\n", "ax.set_title(f'Uncurated Pool + Seed Images\\nDashed circles = retrieval threshold ({threshold})',\n", " fontweight='bold')\n", "ax.legend(fontsize=8); ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Feature Dim 1'); ax.set_ylabel('Feature Dim 2')\n", "ax.set_xlim(-6, 6); ax.set_ylim(-6, 6)\n", "\n", "# After curation\n", "ax = axes[2]\n", "# Rejected\n", "ax.scatter(uncurated[~selected, 0], uncurated[~selected, 1],\n", " c='lightgray', s=10, alpha=0.3, label=f'Rejected ({(~selected).sum()})')\n", "# Selected, colored by nearest seed class\n", "for i, (label, color) in enumerate(zip(seed_labels, seed_colors)):\n", " mask = selected & (nearest_seed_class == i)\n", " ax.scatter(uncurated[mask, 0], uncurated[mask, 1],\n", " c=color, s=20, alpha=0.7, label=f'Selected {label} ({mask.sum()})')\n", "for i, (center, color) in enumerate(zip(seed_centers, seed_colors)):\n", " mask = np.array(seed_class) == i\n", " ax.scatter(seeds[mask, 0], seeds[mask, 1], c=color, s=80,\n", " marker='*', zorder=3, edgecolors='black', linewidths=0.5)\n", "ax.set_title(f'After Retrieval-Based Curation\\n{selected.sum()}/{n_uncurated} images selected ({selected.mean()*100:.1f}%)',\n", " fontweight='bold')\n", "ax.legend(fontsize=8); ax.grid(True, alpha=0.3)\n", "ax.set_xlabel('Feature Dim 1'); ax.set_ylabel('Feature Dim 2')\n", "ax.set_xlim(-6, 6); ax.set_ylim(-6, 6)\n", "\n", "plt.suptitle('Retrieval-Based Curation Pipeline (Simulated)', fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print(f\"Seed images: {len(seeds)}\")\n", "print(f\"Uncurated pool: {n_uncurated}\")\n", "print(f\"Selected: {selected.sum()} ({selected.mean()*100:.1f}%)\")\n", "print(f\"Rejected: {(~selected).sum()} ({(~selected).mean()*100:.1f}%)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.4 LVD-142M Dataset Statistics\n", "\n", "The final LVD-142M dataset has the following properties:\n", "\n", "| Property | Value |\n", "|----------|-------|\n", "| Total images | 142 million |\n", "| Source | Web crawl (CommonCrawl + others) |\n", "| Seed datasets | ImageNet-22K, Google Landmarks, etc. |\n", "| Deduplication | Copy detection with ViT features |\n", "| Curation | Retrieval-based (k-NN to seed images) |\n", "| Resolution | Variable (min 224×224) |\n", "| Format | JPEG |\n", "| Storage | ~1.5 TB |\n", "\n", "### Comparison with other datasets\n", "\n", "| Dataset | Size | Labels | Curation |\n", "|---------|------|--------|----------|\n", "| ImageNet-1K | 1.2M | Yes (1K classes) | Manual |\n", "| ImageNet-22K | 14M | Yes (22K classes) | Manual |\n", "| JFT-300M | 300M | Yes (noisy) | Semi-auto |\n", "| LAION-400M | 400M | Alt-text only | Minimal |\n", "| **LVD-142M** | **142M** | **No** | **Retrieval-based** |" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize dataset comparison ───────────────────────────────────────────────\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "datasets = ['ImageNet-1K', 'ImageNet-22K', 'JFT-300M', 'LAION-400M', 'LVD-142M']\n", "sizes = [1.2, 14, 300, 400, 142] # millions\n", "colors = ['#3498db', '#2ecc71', '#e74c3c', '#f39c12', '#9b59b6']\n", "has_labels = [True, True, True, False, False]\n", "\n", "bars = axes[0].barh(datasets, sizes, color=colors, alpha=0.8)\n", "for bar, labeled in zip(bars, has_labels):\n", " label_text = '✓ labeled' if labeled else '✗ no labels'\n", " color_text = 'green' if labeled else 'red'\n", " axes[0].text(bar.get_width() + 5, bar.get_y() + bar.get_height()/2,\n", " label_text, va='center', fontsize=10, color=color_text, fontweight='bold')\n", "axes[0].set_xlabel('Dataset Size (millions of images)')\n", "axes[0].set_title('Dataset Size Comparison', fontweight='bold')\n", "axes[0].set_xlim(0, 500)\n", "axes[0].grid(True, alpha=0.3, axis='x')\n", "\n", "# Curation pipeline funnel\n", "stages = ['Raw Web\\nImages', 'After\\nDedup', 'After\\nRetrieval', 'LVD-142M']\n", "counts = [1200, 500, 200, 142] # approximate millions\n", "stage_colors = ['#e74c3c', '#e67e22', '#f1c40f', '#2ecc71']\n", "\n", "bars2 = axes[1].bar(stages, counts, color=stage_colors, alpha=0.8, width=0.6)\n", "for bar, count in zip(bars2, counts):\n", " axes[1].text(bar.get_x() + bar.get_width()/2, bar.get_height() + 10,\n", " f'{count}M', ha='center', fontsize=12, fontweight='bold')\n", "# Reduction arrows\n", "for i in range(len(counts)-1):\n", " reduction = (1 - counts[i+1]/counts[i]) * 100\n", " axes[1].annotate('', xy=(i+1, counts[i+1]+30), xytext=(i, counts[i]+30),\n", " arrowprops=dict(arrowstyle='->', color='black', lw=1.5))\n", " axes[1].text(i+0.5, max(counts[i], counts[i+1])+60,\n", " f'-{reduction:.0f}%', ha='center', fontsize=9, color='red')\n", "\n", "axes[1].set_ylabel('Number of Images (millions)')\n", "axes[1].set_title('LVD-142M Curation Funnel\\n(approximate numbers)', fontweight='bold')\n", "axes[1].grid(True, alpha=0.3, axis='y')\n", "\n", "plt.suptitle('LVD-142M Dataset: Curation from 1.2B Web Images', fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.5 Impact of Data Curation on Model Quality\n", "\n", "The DINOv2 paper shows that **data curation significantly improves model quality**:\n", "\n", "| Training Data | ImageNet k-NN | ImageNet Linear |\n", "|---------------|---------------|------------------|\n", "| ImageNet-1K only | 78.2% | 81.1% |\n", "| Uncurated web data | 79.5% | 82.3% |\n", "| LVD-142M (curated) | **83.5%** | **86.5%** |\n", "\n", "Key finding: **curated data > more uncurated data**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Visualize impact of data curation ─────────────────────────────────────────\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "# Performance comparison\n", "configs = ['ImageNet-1K\\n(1.2M)', 'Uncurated\\nWeb (500M)', 'LVD-142M\\n(curated, 142M)']\n", "knn_acc = [78.2, 79.5, 83.5]\n", "lin_acc = [81.1, 82.3, 86.5]\n", "\n", "x = np.arange(len(configs))\n", "width = 0.35\n", "bars1 = axes[0].bar(x - width/2, knn_acc, width, label='k-NN Accuracy', color='#3498db', alpha=0.8)\n", "bars2 = axes[0].bar(x + width/2, lin_acc, width, label='Linear Probe', color='#e74c3c', alpha=0.8)\n", "\n", "for bar in bars1:\n", " axes[0].text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.1,\n", " f'{bar.get_height():.1f}%', ha='center', fontsize=10, fontweight='bold')\n", "for bar in bars2:\n", " axes[0].text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.1,\n", " f'{bar.get_height():.1f}%', ha='center', fontsize=10, fontweight='bold')\n", "\n", "axes[0].set_xticks(x); axes[0].set_xticklabels(configs)\n", "axes[0].set_ylabel('ImageNet Accuracy (%)')\n", "axes[0].set_title('Impact of Data Curation\\non ImageNet Accuracy (ViT-g)', fontweight='bold')\n", "axes[0].legend()\n", "axes[0].set_ylim(75, 90)\n", "axes[0].grid(True, alpha=0.3, axis='y')\n", "\n", "# Highlight the curated bar\n", "axes[0].add_patch(mpatches.FancyBboxPatch(\n", " (1.6, 75), 1.0, 15, boxstyle='round,pad=0.1',\n", " facecolor='gold', edgecolor='orange', alpha=0.2, linewidth=2, zorder=0\n", "))\n", "axes[0].text(2.1, 89, '★ Best', ha='center', fontsize=12, color='orange', fontweight='bold')\n", "\n", "# Data efficiency curve\n", "data_sizes = np.array([1.2, 10, 50, 100, 142, 200, 500])\n", "# Simulated: curated data is more efficient\n", "acc_curated = 78 + 8 * (1 - np.exp(-data_sizes / 50))\n", "acc_uncurated = 76 + 6 * (1 - np.exp(-data_sizes / 200))\n", "\n", "axes[1].plot(data_sizes, acc_curated, 'g-o', linewidth=2, markersize=6, label='Curated data')\n", "axes[1].plot(data_sizes, acc_uncurated, 'r-s', linewidth=2, markersize=6, label='Uncurated data')\n", "axes[1].axvline(142, color='green', linestyle='--', alpha=0.7, label='LVD-142M')\n", "axes[1].fill_between(data_sizes, acc_curated, acc_uncurated, alpha=0.15, color='green',\n", " label='Curation benefit')\n", "axes[1].set_xlabel('Dataset Size (millions)')\n", "axes[1].set_ylabel('ImageNet Accuracy (%) [simulated]')\n", "axes[1].set_title('Data Efficiency: Curated vs Uncurated\\n(simulated trend)', fontweight='bold')\n", "axes[1].legend(fontsize=9)\n", "axes[1].grid(True, alpha=0.3)\n", "axes[1].set_xscale('log')\n", "\n", "plt.suptitle('Why Data Curation Matters for DINOv2', fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.6 Practical: Simulate Curation with Real DINOv2 Features" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Simulate retrieval-based curation with real DINOv2 features ───────────────\n", "from PIL import Image, ImageDraw\n", "\n", "def make_category_image(category, size=224):\n", " \"\"\"Create synthetic images for different semantic categories.\"\"\"\n", " img = Image.new('RGB', (size, size), (240, 240, 240))\n", " draw = ImageDraw.Draw(img)\n", " \n", " if category == 'animal':\n", " draw.ellipse([40, 60, 184, 164], fill=(180, 120, 80)) # body\n", " draw.ellipse([140, 30, 200, 90], fill=(180, 120, 80)) # head\n", " draw.ellipse([155, 45, 170, 60], fill=(50, 50, 50)) # eye\n", " elif category == 'vehicle':\n", " draw.rectangle([30, 100, 194, 160], fill=(60, 100, 180)) # body\n", " draw.rectangle([60, 70, 164, 110], fill=(60, 100, 180)) # roof\n", " draw.ellipse([50, 150, 90, 190], fill=(50, 50, 50)) # wheel\n", " draw.ellipse([134, 150, 174, 190], fill=(50, 50, 50)) # wheel\n", " elif category == 'nature':\n", " draw.rectangle([0, 140, 224, 224], fill=(80, 160, 80)) # ground\n", " draw.rectangle([0, 0, 224, 140], fill=(135, 206, 235)) # sky\n", " draw.rectangle([100, 80, 124, 140], fill=(100, 70, 40)) # trunk\n", " draw.ellipse([60, 20, 164, 100], fill=(50, 140, 50)) # tree\n", " elif category == 'noise':\n", " arr = np.random.randint(0, 255, (size, size, 3), dtype=np.uint8)\n", " img = Image.fromarray(arr)\n", " \n", " return img\n", "\n", "# Create seed images (curated)\n", "seed_images = {\n", " 'animal_1': make_category_image('animal'),\n", " 'animal_2': make_category_image('animal'),\n", " 'vehicle_1': make_category_image('vehicle'),\n", " 'nature_1': make_category_image('nature'),\n", "}\n", "\n", "# Create uncurated pool (mix of relevant and noise)\n", "uncurated_images = {\n", " 'animal_3': make_category_image('animal'),\n", " 'vehicle_2': make_category_image('vehicle'),\n", " 'nature_2': make_category_image('nature'),\n", " 'noise_1': make_category_image('noise'),\n", " 'noise_2': make_category_image('noise'),\n", " 'noise_3': make_category_image('noise'),\n", "}\n", "\n", "# Extract features\n", "def extract_cls(img):\n", " inp = processor(images=img, return_tensors='pt')\n", " inp = {k: v.to(DEVICE) for k, v in inp.items()}\n", " with torch.no_grad():\n", " out = model(**inp)\n", " return out.last_hidden_state[0, 0].cpu().numpy()\n", "\n", "seed_feats = {k: extract_cls(v) for k, v in seed_images.items()}\n", "uncurated_feats = {k: extract_cls(v) for k, v in uncurated_images.items()}\n", "\n", "# Compute similarity of each uncurated image to seed set\n", "seed_feat_matrix = np.array(list(seed_feats.values()))\n", "seed_feat_norm = seed_feat_matrix / np.linalg.norm(seed_feat_matrix, axis=1, keepdims=True)\n", "\n", "results = {}\n", "for name, feat in uncurated_feats.items():\n", " feat_norm = feat / np.linalg.norm(feat)\n", " sims = seed_feat_norm @ feat_norm\n", " results[name] = {'max_sim': sims.max(), 'best_seed': list(seed_feats.keys())[sims.argmax()]}\n", "\n", "# Visualize\n", "fig, axes = plt.subplots(2, 5, figsize=(20, 8))\n", "\n", "# Seeds\n", "for i, (name, img) in enumerate(seed_images.items()):\n", " axes[0, i].imshow(img)\n", " axes[0, i].set_title(f'SEED: {name}', fontweight='bold', color='green', fontsize=9)\n", " axes[0, i].axis('off')\n", "axes[0, 4].axis('off')\n", "axes[0, 4].text(0.5, 0.5, 'Seed Set\\n(curated, high quality)',\n", " ha='center', va='center', fontsize=12, fontweight='bold',\n", " transform=axes[0,4].transAxes, color='green')\n", "\n", "# Uncurated\n", "threshold = 0.85\n", "for i, (name, img) in enumerate(uncurated_images.items()):\n", " axes[1, i].imshow(img)\n", " sim = results[name]['max_sim']\n", " keep = sim > threshold\n", " color = 'green' if keep else 'red'\n", " status = f'✓ KEEP (sim={sim:.3f})' if keep else f'✗ REJECT (sim={sim:.3f})'\n", " axes[1, i].set_title(f'{name}\\n{status}', fontweight='bold', color=color, fontsize=8)\n", " axes[1, i].axis('off')\n", " for spine in axes[1, i].spines.values():\n", " spine.set_edgecolor(color); spine.set_linewidth(4)\n", "\n", "plt.suptitle(f'Retrieval-Based Curation (threshold={threshold})\\nTop row: seed images | Bottom row: uncurated pool',\n", " fontsize=13, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print(\"\\nCuration Results:\")\n", "for name, res in results.items():\n", " status = 'KEEP' if res['max_sim'] > threshold else 'REJECT'\n", " print(f\" {name:15s}: max_sim={res['max_sim']:.4f}, best_seed={res['best_seed']:12s} → {status}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.7 Summary\n", "\n", "| Step | Method | Purpose |\n", "|------|--------|----------|\n", "| Raw data | Web crawl (1.2B images) | Large-scale unlabeled data |\n", "| Deduplication | Copy detection with ViT features | Remove near-duplicates |\n", "| Curation | Retrieval from curated seeds | Select semantically relevant images |\n", "| Result | LVD-142M (142M images) | High-quality, diverse training set |\n", "\n", "**Key insight:** 142M curated images outperform 500M uncurated images. \n", "**Data quality > Data quantity** for self-supervised learning.\n", "\n", "---\n", "**Next:** [`05_vit_architecture.ipynb`](05_vit_architecture.ipynb) — Deep dive into the Vision Transformer architecture used in DINOv2." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }