{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DINOv2 Tutorial \u2014 Notebook 13: Model Variants\n",
    "\n",
    "**Series:** DINOv2 Deep Dive | **Prev:** `12_downstream_tasks.ipynb` | **Next:** `14_registers_artifact_removal.ipynb`\n",
    "\n",
    "---\n",
    "\n",
    "## What You Will Learn\n",
    "- The four DINOv2 model variants: ViT-S, ViT-B, ViT-L, ViT-g\n",
    "- Parameter counts, FLOPs, and memory requirements\n",
    "- Performance vs compute tradeoffs\n",
    "- How to load different variants from Hugging Face\n",
    "- Scaling laws: how performance improves with model size\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.1 DINOv2 Model Variants Overview\n",
    "\n",
    "DINOv2 was trained in four sizes, all using the same training procedure:\n",
    "\n",
    "| Model | Patch Size | Hidden Dim | Heads | Layers | Params | ImageNet k-NN | ImageNet Linear |\n",
    "|-------|-----------|------------|-------|--------|--------|---------------|------------------|\n",
    "| ViT-S/14 | 14 | 384 | 6 | 12 | 21M | 79.0% | 81.1% |\n",
    "| ViT-B/14 | 14 | 768 | 12 | 12 | 86M | 82.1% | 84.5% |\n",
    "| ViT-L/14 | 14 | 1024 | 16 | 24 | 307M | 83.1% | 86.3% |\n",
    "| ViT-g/14 | 14 | 1536 | 24 | 40 | 1.1B | 83.5% | 86.5% |\n",
    "\n",
    "All variants use:\n",
    "- Patch size **14** (not 16 like standard ViT)\n",
    "- Image size **224\u00d7224** \u2192 **16\u00d716 = 256 patches**\n",
    "- Pre-norm architecture\n",
    "- GELU activation\n",
    "- No classification head (features used directly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "import torch\n",
    "from transformers import AutoImageProcessor, AutoModel\n",
    "\n",
    "# Model specifications\n",
    "variants = {\n",
    "    'ViT-S/14': {\n",
    "        'hidden_dim': 384, 'heads': 6,  'layers': 12, 'mlp_dim': 1536,\n",
    "        'params_M': 21,   'knn_acc': 79.0, 'lin_acc': 81.1,\n",
    "        'hf_name': 'facebook/dinov2-small',\n",
    "        'color': '#3498db',\n",
    "    },\n",
    "    'ViT-B/14': {\n",
    "        'hidden_dim': 768, 'heads': 12, 'layers': 12, 'mlp_dim': 3072,\n",
    "        'params_M': 86,   'knn_acc': 82.1, 'lin_acc': 84.5,\n",
    "        'hf_name': '/mnt/mydata/huggingface/dinov2-base',\n",
    "        'color': '#2ecc71',\n",
    "    },\n",
    "    'ViT-L/14': {\n",
    "        'hidden_dim': 1024, 'heads': 16, 'layers': 24, 'mlp_dim': 4096,\n",
    "        'params_M': 307,  'knn_acc': 83.1, 'lin_acc': 86.3,\n",
    "        'hf_name': 'facebook/dinov2-large',\n",
    "        'color': '#e67e22',\n",
    "    },\n",
    "    'ViT-g/14': {\n",
    "        'hidden_dim': 1536, 'heads': 24, 'layers': 40, 'mlp_dim': 6144,\n",
    "        'params_M': 1100, 'knn_acc': 83.5, 'lin_acc': 86.5,\n",
    "        'hf_name': 'facebook/dinov2-giant',\n",
    "        'color': '#e74c3c',\n",
    "    },\n",
    "}\n",
    "\n",
    "print(\"DINOv2 Model Variants:\")\n",
    "print(f\"{'Model':>10} {'d_model':>8} {'Heads':>6} {'Layers':>7} {'MLP':>6} {'Params':>8} {'k-NN':>7} {'Linear':>8}\")\n",
    "print(\"-\" * 70)\n",
    "for name, v in variants.items():\n",
    "    print(f\"{name:>10} {v['hidden_dim']:>8} {v['heads']:>6} {v['layers']:>7} \"\n",
    "          f\"{v['mlp_dim']:>6} {v['params_M']:>6}M {v['knn_acc']:>6.1f}% {v['lin_acc']:>7.1f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.2 Architecture Comparison Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \u2500\u2500 Architecture comparison \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
    "names  = list(variants.keys())\n",
    "colors = [v['color'] for v in variants.values()]\n",
    "\n",
    "fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n",
    "\n",
    "# Parameter count\n",
    "params = [v['params_M'] for v in variants.values()]\n",
    "bars = axes[0, 0].bar(names, params, color=colors, alpha=0.85)\n",
    "for bar, p in zip(bars, params):\n",
    "    axes[0, 0].text(bar.get_x() + bar.get_width()/2, bar.get_height() + 10,\n",
    "                    f'{p}M', ha='center', fontsize=11, fontweight='bold')\n",
    "axes[0, 0].set_title('Parameter Count', fontweight='bold')\n",
    "axes[0, 0].set_ylabel('Parameters (millions)')\n",
    "axes[0, 0].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# Hidden dimension\n",
    "dims = [v['hidden_dim'] for v in variants.values()]\n",
    "axes[0, 1].bar(names, dims, color=colors, alpha=0.85)\n",
    "for i, d in enumerate(dims):\n",
    "    axes[0, 1].text(i, d + 20, str(d), ha='center', fontsize=11, fontweight='bold')\n",
    "axes[0, 1].set_title('Hidden Dimension (d_model)', fontweight='bold')\n",
    "axes[0, 1].set_ylabel('Dimension')\n",
    "axes[0, 1].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# Number of layers\n",
    "layers = [v['layers'] for v in variants.values()]\n",
    "axes[0, 2].bar(names, layers, color=colors, alpha=0.85)\n",
    "for i, l in enumerate(layers):\n",
    "    axes[0, 2].text(i, l + 0.3, str(l), ha='center', fontsize=11, fontweight='bold')\n",
    "axes[0, 2].set_title('Number of Transformer Layers', fontweight='bold')\n",
    "axes[0, 2].set_ylabel('Layers')\n",
    "axes[0, 2].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# k-NN accuracy\n",
    "knn_accs = [v['knn_acc'] for v in variants.values()]\n",
    "lin_accs = [v['lin_acc'] for v in variants.values()]\n",
    "x = np.arange(len(names))\n",
    "width = 0.35\n",
    "axes[1, 0].bar(x - width/2, knn_accs, width, label='k-NN', color=colors, alpha=0.85)\n",
    "axes[1, 0].bar(x + width/2, lin_accs, width, label='Linear', color=colors, alpha=0.5,\n",
    "               edgecolor=[c for c in colors], linewidth=2)\n",
    "for i, (k, l) in enumerate(zip(knn_accs, lin_accs)):\n",
    "    axes[1, 0].text(i - width/2, k + 0.1, f'{k:.1f}', ha='center', fontsize=9, fontweight='bold')\n",
    "    axes[1, 0].text(i + width/2, l + 0.1, f'{l:.1f}', ha='center', fontsize=9, fontweight='bold')\n",
    "axes[1, 0].set_xticks(x); axes[1, 0].set_xticklabels(names)\n",
    "axes[1, 0].set_title('ImageNet Accuracy', fontweight='bold')\n",
    "axes[1, 0].set_ylabel('Accuracy (%)')\n",
    "axes[1, 0].set_ylim(75, 90)\n",
    "axes[1, 0].legend(); axes[1, 0].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# Scaling: params vs accuracy\n",
    "axes[1, 1].scatter(params, knn_accs, c=colors, s=200, zorder=3, edgecolors='black', linewidths=1)\n",
    "axes[1, 1].scatter(params, lin_accs, c=colors, s=200, zorder=3, marker='s',\n",
    "                   edgecolors='black', linewidths=1)\n",
    "for name, p, k, l, color in zip(names, params, knn_accs, lin_accs, colors):\n",
    "    axes[1, 1].annotate(name, (p, k), textcoords='offset points', xytext=(5, 5), fontsize=9)\n",
    "axes[1, 1].plot(params, knn_accs, 'k--', alpha=0.3, linewidth=1)\n",
    "axes[1, 1].plot(params, lin_accs, 'k:', alpha=0.3, linewidth=1)\n",
    "axes[1, 1].set_xscale('log')\n",
    "axes[1, 1].set_title('Scaling: Parameters vs Accuracy\\n(circles=k-NN, squares=linear)',\n",
    "                      fontweight='bold')\n",
    "axes[1, 1].set_xlabel('Parameters (log scale)')\n",
    "axes[1, 1].set_ylabel('ImageNet Accuracy (%)')\n",
    "axes[1, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# Architecture diagram comparison\n",
    "ax = axes[1, 2]\n",
    "ax.set_xlim(0, 10); ax.set_ylim(0, 10)\n",
    "ax.axis('off')\n",
    "ax.set_title('Architecture Comparison', fontweight='bold')\n",
    "\n",
    "for i, (name, v, color) in enumerate(zip(names, variants.values(), colors)):\n",
    "    x_pos = 1 + i * 2.2\n",
    "    # Draw stacked blocks representing layers\n",
    "    block_h = 8.0 / max(layers)\n",
    "    for layer in range(v['layers']):\n",
    "        y_pos = 0.5 + layer * block_h\n",
    "        width_block = 0.8 + v['hidden_dim'] / 2000\n",
    "        ax.add_patch(plt.FancyBboxPatch(\n",
    "            (x_pos - width_block/2, y_pos), width_block, block_h * 0.85,\n",
    "            boxstyle='round,pad=0.02', facecolor=color, edgecolor='white',\n",
    "            alpha=0.7, linewidth=0.5\n",
    "        ))\n",
    "    ax.text(x_pos, 9.5, name, ha='center', fontsize=9, fontweight='bold', color=color)\n",
    "    ax.text(x_pos, 0.1, f'{v[\"params_M\"]}M', ha='center', fontsize=8, color='black')\n",
    "\n",
    "plt.suptitle('DINOv2 Model Variants: Architecture and Performance Comparison',\n",
    "             fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.3 Load and Inspect the Base Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \u2500\u2500 Load and count parameters of the base model \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
    "MODEL_PATH = \"/mnt/mydata/huggingface/dinov2-base\"\n",
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "processor = AutoImageProcessor.from_pretrained(MODEL_PATH)\n",
    "model = AutoModel.from_pretrained(MODEL_PATH).to(DEVICE).eval()\n",
    "cfg = model.config\n",
    "\n",
    "def count_params(module):\n",
    "    return sum(p.numel() for p in module.parameters())\n",
    "\n",
    "total = count_params(model)\n",
    "print(f\"DINOv2 ViT-B/14 \u2014 Actual parameter count: {total:,} ({total/1e6:.1f}M)\")\n",
    "print()\n",
    "print(\"Breakdown by component:\")\n",
    "print(f\"  {'Component':<35} {'Params':>12} {'%':>6}\")\n",
    "print(\"-\" * 58)\n",
    "for name, module in model.named_children():\n",
    "    p = count_params(module)\n",
    "    print(f\"  {name:<35} {p:>12,} {p/total*100:>5.1f}%\")\n",
    "\n",
    "print()\n",
    "print(\"Breakdown by layer type:\")\n",
    "layer_types = {}\n",
    "for name, module in model.named_modules():\n",
    "    t = type(module).__name__\n",
    "    if t not in layer_types:\n",
    "        layer_types[t] = 0\n",
    "    layer_types[t] += count_params(module)\n",
    "\n",
    "# Show top types\n",
    "for t, p in sorted(layer_types.items(), key=lambda x: -x[1])[:8]:\n",
    "    if p > 0:\n",
    "        print(f\"  {t:<35} {p:>12,} ({p/total*100:.1f}%)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.4 Compute FLOPs Estimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \u2500\u2500 Estimate FLOPs for each variant \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
    "def estimate_flops(hidden_dim, heads, layers, mlp_dim, seq_len=257):\n",
    "    \"\"\"\n",
    "    Estimate FLOPs for a ViT forward pass.\n",
    "    Simplified: count multiply-adds in attention and MLP.\n",
    "    \"\"\"\n",
    "    N = seq_len\n",
    "    D = hidden_dim\n",
    "    \n",
    "    # Per layer:\n",
    "    # QKV projection: 3 * N * D * D\n",
    "    qkv_flops = 3 * N * D * D\n",
    "    # Attention: N * N * D (for each head, summed)\n",
    "    attn_flops = N * N * D\n",
    "    # Output projection: N * D * D\n",
    "    out_flops = N * D * D\n",
    "    # MLP: N * D * mlp_dim * 2\n",
    "    mlp_flops = 2 * N * D * mlp_dim\n",
    "    \n",
    "    per_layer = qkv_flops + attn_flops + out_flops + mlp_flops\n",
    "    total = per_layer * layers\n",
    "    \n",
    "    # Patch embedding: image_size^2 * 3 * D (conv)\n",
    "    patch_embed = 224 * 224 * 3 * D\n",
    "    \n",
    "    return (total + patch_embed) * 2  # multiply-adds -> FLOPs\n",
    "\n",
    "print(\"Estimated FLOPs per forward pass:\")\n",
    "print(f\"{'Model':>10} {'GFLOPs':>10} {'Params':>10} {'FLOPs/Param':>14}\")\n",
    "print(\"-\" * 50)\n",
    "for name, v in variants.items():\n",
    "    flops = estimate_flops(v['hidden_dim'], v['heads'], v['layers'], v['mlp_dim'])\n",
    "    gflops = flops / 1e9\n",
    "    params = v['params_M'] * 1e6\n",
    "    print(f\"{name:>10} {gflops:>10.1f} {v['params_M']:>8}M {flops/params:>12.1f}\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
    "\n",
    "gflops_list = [estimate_flops(v['hidden_dim'], v['heads'], v['layers'], v['mlp_dim'])/1e9\n",
    "               for v in variants.values()]\n",
    "params_list = [v['params_M'] for v in variants.values()]\n",
    "lin_accs    = [v['lin_acc'] for v in variants.values()]\n",
    "\n",
    "# Accuracy vs FLOPs\n",
    "for name, gf, acc, color in zip(names, gflops_list, lin_accs, colors):\n",
    "    axes[0].scatter(gf, acc, c=color, s=200, zorder=3, edgecolors='black', linewidths=1)\n",
    "    axes[0].annotate(name, (gf, acc), textcoords='offset points', xytext=(5, 5), fontsize=9)\n",
    "axes[0].set_xscale('log')\n",
    "axes[0].set_title('Accuracy vs FLOPs\\n(linear probe on ImageNet)', fontweight='bold')\n",
    "axes[0].set_xlabel('GFLOPs (log scale)')\n",
    "axes[0].set_ylabel('ImageNet Linear Accuracy (%)')\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "\n",
    "# Memory estimate (rough: 4 bytes per param, fp32)\n",
    "memory_gb = [p * 4 / 1024 for p in params_list]  # GB\n",
    "axes[1].bar(names, memory_gb, color=colors, alpha=0.85)\n",
    "for i, m in enumerate(memory_gb):\n",
    "    axes[1].text(i, m + 0.05, f'{m:.2f} GB', ha='center', fontsize=10, fontweight='bold')\n",
    "axes[1].set_title('Model Memory (fp32)\\n(weights only, no activations)', fontweight='bold')\n",
    "axes[1].set_ylabel('Memory (GB)')\n",
    "axes[1].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "plt.suptitle('DINOv2 Compute and Memory Requirements', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.5 Scaling Laws"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \u2500\u2500 Scaling law visualization \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
    "\n",
    "params_log = np.log10([v['params_M'] for v in variants.values()])\n",
    "knn_accs   = [v['knn_acc'] for v in variants.values()]\n",
    "lin_accs   = [v['lin_acc'] for v in variants.values()]\n",
    "\n",
    "# Fit log-linear scaling law\n",
    "knn_fit = np.polyfit(params_log, knn_accs, 1)\n",
    "lin_fit = np.polyfit(params_log, lin_accs, 1)\n",
    "\n",
    "x_fit = np.linspace(params_log.min() - 0.2, params_log.max() + 0.2, 100)\n",
    "knn_pred = np.polyval(knn_fit, x_fit)\n",
    "lin_pred = np.polyval(lin_fit, x_fit)\n",
    "\n",
    "for name, p, k, l, color in zip(names, params_log, knn_accs, lin_accs, colors):\n",
    "    axes[0].scatter(10**p, k, c=color, s=150, zorder=3, edgecolors='black', linewidths=1,\n",
    "                    label=f'{name} (k-NN)')\n",
    "    axes[0].scatter(10**p, l, c=color, s=150, zorder=3, marker='s',\n",
    "                    edgecolors='black', linewidths=1)\n",
    "\n",
    "axes[0].plot(10**x_fit, knn_pred, 'b--', alpha=0.5, linewidth=1.5, label='k-NN trend')\n",
    "axes[0].plot(10**x_fit, lin_pred, 'r--', alpha=0.5, linewidth=1.5, label='Linear trend')\n",
    "axes[0].set_xscale('log')\n",
    "axes[0].set_title('Scaling Law: Parameters vs Accuracy\\n(circles=k-NN, squares=linear)',\n",
    "                  fontweight='bold')\n",
    "axes[0].set_xlabel('Parameters (log scale)')\n",
    "axes[0].set_ylabel('ImageNet Accuracy (%)')\n",
    "axes[0].legend(fontsize=8); axes[0].grid(True, alpha=0.3)\n",
    "\n",
    "# Diminishing returns\n",
    "param_ratios = [params_list[i+1]/params_list[i] for i in range(len(params_list)-1)]\n",
    "acc_gains_knn = [knn_accs[i+1] - knn_accs[i] for i in range(len(knn_accs)-1)]\n",
    "acc_gains_lin = [lin_accs[i+1] - lin_accs[i] for i in range(len(lin_accs)-1)]\n",
    "\n",
    "transitions = ['S->B', 'B->L', 'L->g']\n",
    "x = np.arange(len(transitions))\n",
    "width = 0.35\n",
    "axes[1].bar(x - width/2, acc_gains_knn, width, label='k-NN gain', color='#3498db', alpha=0.8)\n",
    "axes[1].bar(x + width/2, acc_gains_lin, width, label='Linear gain', color='#e74c3c', alpha=0.8)\n",
    "for i, (k, l) in enumerate(zip(acc_gains_knn, acc_gains_lin)):\n",
    "    axes[1].text(i - width/2, k + 0.05, f'+{k:.1f}%', ha='center', fontsize=10, fontweight='bold')\n",
    "    axes[1].text(i + width/2, l + 0.05, f'+{l:.1f}%', ha='center', fontsize=10, fontweight='bold')\n",
    "axes[1].set_xticks(x); axes[1].set_xticklabels(transitions)\n",
    "axes[1].set_title('Accuracy Gain per Model Scale-Up\\n(diminishing returns at larger scale)',\n",
    "                  fontweight='bold')\n",
    "axes[1].set_ylabel('Accuracy Gain (%)')\n",
    "axes[1].legend(); axes[1].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "plt.suptitle('DINOv2 Scaling Laws', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "print(\"Scaling analysis:\")\n",
    "for i, (trans, ratio, kg, lg) in enumerate(zip(transitions, param_ratios, acc_gains_knn, acc_gains_lin)):\n",
    "    print(f\"  {trans}: {ratio:.1f}x more params -> +{kg:.1f}% k-NN, +{lg:.1f}% linear\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.6 How to Load Different Variants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \u2500\u2500 Loading different variants \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
    "print(\"How to load different DINOv2 variants:\")\n",
    "print()\n",
    "for name, v in variants.items():\n",
    "    print(f\"# {name} ({v['params_M']}M params)\")\n",
    "    print(f\"processor = AutoImageProcessor.from_pretrained('{v['hf_name']}')\")\n",
    "    print(f\"model = AutoModel.from_pretrained('{v['hf_name']}')\")\n",
    "    print()\n",
    "\n",
    "# Inspect the loaded base model config\n",
    "print(\"Current model (ViT-B/14) config:\")\n",
    "print(f\"  hidden_size:          {cfg.hidden_size}\")\n",
    "print(f\"  num_attention_heads:  {cfg.num_attention_heads}\")\n",
    "print(f\"  num_hidden_layers:    {cfg.num_hidden_layers}\")\n",
    "print(f\"  intermediate_size:    {int(cfg.hidden_size * cfg.mlp_ratio)}\")\n",
    "print(f\"  patch_size:           {cfg.patch_size}\")\n",
    "print(f\"  image_size:           {cfg.image_size}\")\n",
    "print(f\"  head_dim:             {cfg.hidden_size // cfg.num_attention_heads}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.7 Summary\n",
    "\n",
    "| Model | Best For | Tradeoff |\n",
    "|-------|----------|----------|\n",
    "| ViT-S/14 (21M) | Edge devices, fast inference | Lower accuracy |\n",
    "| ViT-B/14 (86M) | Balanced performance/speed | Good all-around |\n",
    "| ViT-L/14 (307M) | High accuracy tasks | Slower, more memory |\n",
    "| ViT-g/14 (1.1B) | Maximum accuracy | Very slow, needs large GPU |\n",
    "\n",
    "**Scaling law insight:** Each 4x increase in parameters gives ~1-2% accuracy gain.  \n",
    "The returns diminish at larger scales \u2014 ViT-B is often the best practical choice.\n",
    "\n",
    "---\n",
    "**Next:** [`14_registers_artifact_removal.ipynb`](14_registers_artifact_removal.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}