Hyperparameter tuning & training deep models

Most failed deep-learning projects do not fail because the architecture was wrong. They fail because the training was misconfigured, the loss did not match the task, or a silent bug in the loop meant the model never saw what the engineer thought it saw. This page is about that layer: which knobs actually matter and in what order, how to choose optimizers and losses, the specific recipes that work for detection, segmentation, ViTs and CLIP, and a systematic way to diagnose a run that is not converging.

Symbols used on this page are defined in the Nomenclature → deep learning & training.

Model-specific deep dives live in Deep Learning; deployment, quantisation and throughput are in models & tooling.

1 · What to tune, and in what order

Hyperparameters are not equally important, and tuning them in the wrong order wastes most of the effort. The sensitivity ranking below is fairly stable across vision tasks:

2 · Learning rate

2.1 Why it dominates

For a quadratic loss with curvature eigenvalues in $[\mu, L]$, gradient descent converges only if

$$ \eta \lt \frac{2}{L}, $$

and the convergence rate is governed by the condition number $\kappa = L/\mu$. Two consequences that carry over to real networks: there is a hard divergence threshold set by the sharpest curvature direction, and the useful range sits just below it — which is why the best learning rate is usually within a factor of 2–3 of the one that blows up. That is an uncomfortably narrow target, and it is why LR is knob #1.

The first simulation makes the three regimes concrete. Worth doing: raise the peak LR until the loss panel turns red (that is a NaN in a real log), then add warmup and watch the same peak LR become survivable.

2.2 Finding it: the LR range test

Do not guess. Smith's LR range test finds the usable band in one short run: start at a very small LR, increase it geometrically every iteration over a few hundred steps, and plot loss against LR.

Pick a value where the loss is still falling steeply, typically a factor of 3–10 below where it turns upward. A common mistake is to pick the minimum of this curve, which is already past the stable region.

2.3 Warmup

Warmup ramps the LR from ~0 to the peak over the first few hundred to few thousand steps. Three reasons it is needed, all about the first moments of training:

• Adam's second-moment estimate is unreliable early. With few samples, $\hat v_t$ is noisy, so the effective step $\eta/\sqrt{\hat v_t}$ can be enormous. Warmup covers the period while the estimate settles.
• Large-batch training needs a large LR (§3), and a large LR applied to a randomly initialised network takes a step far outside the region where the local quadratic model is valid.
• Transformers are especially sensitive. Post-norm architectures can have very large early gradients through the residual path; skipping warmup on a ViT frequently produces an immediate divergence.

Rules of thumb: 3–5 epochs of warmup for ImageNet-scale classification, 500–2000 steps for transformer fine-tuning, ~3 epochs for YOLO-family detectors (which also warm up the momentum term). If a run dies in the first few hundred steps, warmup is the first thing to check.

2.4 Schedules

2.5 Layer-wise and parameter-group rates

A single global LR is rarely optimal when part of the network is pretrained. Two standard patterns:

• Backbone/head split. Fine-tuning gives the pretrained backbone a lower LR (often 10× lower) than the randomly initialised head, so good features are not destroyed while the head is still random.
• Layer-wise LR decay. Scale the LR by $\alpha^{(\text{depth}-\ell)}$ with $\alpha \approx 0.65$–$0.9$, so early layers move least. This is standard for ViT and BERT fine-tuning and typically worth a point or two of accuracy.

A separate, important grouping: never apply weight decay to normalisation parameters or biases. Decaying a BatchNorm scale toward zero actively fights the network. In PyTorch this means constructing two parameter groups (§8.3).

3 · Batch size, and its coupling to the learning rate

3.1 The scaling rules

A minibatch gradient is an estimator of the full gradient with variance $\propto 1/B$. So the noise scale falls as $1/\sqrt B$, which permits a larger step. Two rules are in common use:

$$ \text{linear: } \eta \propto B \qquad\qquad \text{square-root: } \eta \propto \sqrt B $$

Both hold only up to a critical batch size $B_{\text{crit}}$. Below it, doubling the batch nearly halves the number of steps needed — near-perfect scaling. Above it, extra samples per step buy almost nothing, so you pay double the compute for a few percent fewer steps. The simulation shows this directly: the "steps to target" curve flattens while the "samples seen" curve keeps climbing.

3.2 The practical constraints

• BatchNorm breaks at small batch. Below about 8–16 per GPU the batch statistics are too noisy, and detection/segmentation models running at batch 2 are the classic casualty. Fix by switching to GroupNorm, using SyncBatchNorm across GPUs, or freezing the BN statistics inherited from pretraining (which is what standard Faster R-CNN recipes do).
• Gradient accumulation simulates a large batch under a memory limit: run $k$ forward/backward passes, then step once. It is equivalent for the optimizer but not for BatchNorm, whose statistics still come from the small physical batch.
• Small batches regularise. Gradient noise is not purely a cost — it helps escape sharp minima. A very large batch trained to the same loss often generalises slightly worse, which is why extra regularisation or a longer schedule is usually needed at scale.

When changing batch size to fit a new GPU, remember that LR, weight decay's effective strength, warmup length measured in steps, and the total step count all shift. Changing batch size alone and keeping everything else fixed is a very common way to "mysteriously" lose accuracy.

4 · Epochs, early stopping and the data budget

The number of epochs is best thought of as a consequence of the dataset size and the schedule rather than an independent knob. Typical practice:

Early stopping should be implemented as "train long, checkpoint the best validation score, keep training with patience" rather than "stop the first time validation rises". Validation curves are noisy and frequently rise before falling again, especially around an LR drop. Patience of 10–20 evaluations is typical, and the checkpoint you keep is the best one, not the last.

5 · Regularisation

5.1 Weight decay

Weight decay is the strongest regulariser after augmentation, and its correct implementation depends on the optimizer. With plain SGD, L2 regularisation and weight decay are the same thing. With Adam they are not, because L2 added to the gradient gets divided by $\sqrt{\hat v_t}$ along with everything else — so parameters with large gradient history get less regularisation, which is backwards. AdamW's decoupled decay applies it directly to the weights:

$$ \theta_{t+1} = \theta_t - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} \;-\; \eta\,\lambda\,\theta_t . $$

5.2 Dropout

Dropout zeroes activations with probability $p$ at training time and rescales by $1/(1-p)$ so the expected activation is preserved. Its role has narrowed considerably:

5.3 Augmentation is the main event in vision

Empirically, augmentation buys more generalisation per unit of effort than any other regulariser, because it injects real task knowledge (invariances) rather than generic noise.

5.4 Normalisation layers

The choice of normaliser is effectively a hyperparameter, and it interacts directly with batch size. What separates them is only which slice of the tensor the statistics are computed over — and that single difference explains all of their practical behaviour:

BatchNorm's train/eval asymmetry is a frequent source of confusion: at training time it uses the current batch's statistics, at evaluation the running averages. A model that scores well in training mode and poorly in eval mode usually has broken running statistics — too few updates, a momentum set wrong, or model.eval() never called. Conversely, forgetting model.train() after validation leaves BN frozen for the rest of training.

6 · Optimizers

6.1 The three that matter

SGD with momentum accumulates a velocity and steps along it:

$$ \mathbf v_{t+1} = \gamma\,\mathbf v_t + \nabla_\theta \mathcal L, \qquad \theta_{t+1} = \theta_t - \eta\,\mathbf v_{t+1}, \qquad \gamma \approx 0.9 . $$

Adam additionally keeps a per-parameter second moment and normalises by it, making the step size adaptive:

$$ \mathbf m_t = \beta_1 \mathbf m_{t-1} + (1-\beta_1)\,\mathbf g_t, \qquad \mathbf v_t = \beta_2 \mathbf v_{t-1} + (1-\beta_2)\,\mathbf g_t^2, $$ $$ \hat{\mathbf m}_t = \frac{\mathbf m_t}{1-\beta_1^t}, \qquad \hat{\mathbf v}_t = \frac{\mathbf v_t}{1-\beta_2^t}, \qquad \theta_{t+1} = \theta_t - \eta\,\frac{\hat{\mathbf m}_t}{\sqrt{\hat{\mathbf v}_t}+\epsilon} . $$

The bias-correction terms $1-\beta^t$ matter only for the first few hundred steps, but they matter a lot there — without them the early steps are badly scaled, which is part of why warmup helps. AdamW is Adam with the weight decay decoupled from the gradient (§5.1) and is the correct default whenever you would otherwise reach for Adam.

6.2 Choosing

6.3 The parameters that occasionally need attention

6.4 Gradient clipping

Clipping bounds the update when a rare batch produces an enormous gradient:

$$ \mathbf g \leftarrow \mathbf g \cdot \min\!\left(1, \frac{c}{\|\mathbf g\|_2}\right) . $$

Clip by global norm rather than per-element value, so the update direction is preserved. Typical $c$ is 1.0 for transformers and 10 for detection heads. It is cheap insurance: a single bad batch — a corrupt label, an empty ground truth, a degenerate box — otherwise takes one catastrophic step that the run may never recover from. Log the pre-clip gradient norm; a rising trend is an early warning of instability, and frequent clipping means the LR is too high.

7 · Loss functions

7.1 Classification

Cross-entropy is the baseline, $\mathcal L = -\sum_c y_c \log \hat p_c$. Two standard modifications:

Label smoothing replaces the one-hot target with $y_c = 1-\varepsilon$ for the true class and $\varepsilon/(K-1)$ elsewhere, with $\varepsilon \approx 0.1$. This prevents the network from driving logits to infinity to chase a target it can never reach, which improves calibration and slightly improves accuracy. It is standard for ViT and for YOLO classification branches.

Focal loss addresses extreme foreground/background imbalance in dense detection:

$$ \mathcal L_{\text{focal}} = -\alpha_t\left(1 - \hat p_t\right)^{\gamma}\log \hat p_t, \qquad \gamma \approx 2, \ \alpha \approx 0.25 . $$

The factor $(1-\hat p_t)^\gamma$ down-weights already-well-classified examples. With $\gamma = 2$, an example at $\hat p = 0.9$ contributes 100× less than at $\hat p = 0.5$. This matters because a single-stage detector evaluates ~$10^5$ anchors of which a handful are positive — without focal loss the sum of many easy negatives swamps the gradient from the few positives.

7.2 Box regression

7.3 Segmentation

The central issue is class imbalance: a typical mask is overwhelmingly background, so a per-pixel cross-entropy is minimised well by predicting "background everywhere".

The $\varepsilon$ in the Dice formula is not decoration — with an empty ground-truth mask the numerator and denominator are both zero, and without smoothing the loss is NaN. Empty masks occur routinely in real datasets, so this is a practical stability issue rather than a theoretical one.

7.4 Contrastive losses (CLIP)

CLIP trains image and text encoders so matched pairs align. The symmetric InfoNCE loss over a batch of $N$ pairs is

$$ \mathcal L = \tfrac12\left( \mathcal L_{\text{i}\to\text{t}} + \mathcal L_{\text{t}\to\text{i}}\right), \qquad \mathcal L_{\text{i}\to\text{t}} = -\frac1N\sum_{i} \log\frac{\exp\!\left(\mathbf z^{\text{i}}_i \cdot \mathbf z^{\text{t}}_i / \tau\right)} {\sum_{j}\exp\!\left(\mathbf z^{\text{i}}_i \cdot \mathbf z^{\text{t}}_j / \tau\right)} . $$

Two details dominate in practice. The temperature $\tau$ controls how sharply the softmax distinguishes the positive from the negatives — CLIP learns it as a parameter but clamps it, because an unconstrained $\tau$ collapses toward zero and destabilises training. And the effective number of negatives is the batch size, which is why contrastive pretraining is run at batch sizes in the tens of thousands and why gathering embeddings across all GPUs before computing the loss is essential rather than optional.

7.5 Multi-task weighting

Detection and instance segmentation losses are sums of heterogeneous terms:

$$ \mathcal L = \lambda_{\text{cls}}\mathcal L_{\text{cls}} + \lambda_{\text{box}}\mathcal L_{\text{box}} + \lambda_{\text{obj}}\mathcal L_{\text{obj}} + \lambda_{\text{mask}}\mathcal L_{\text{mask}} . $$

The $\lambda$'s matter, and their correct values depend on the numeric scale of each term — which changes if you change a loss function. Swapping smooth-L1 for CIoU, for example, changes the box term's magnitude by a large factor, so an untouched $\lambda_{\text{box}}$ silently re-weights the whole objective. Always log the individual loss terms, not just the total: if one term is 100× the others, it is the only thing being optimised. Uncertainty weighting (Kendall et al.) learns the $\lambda$'s by treating each task's noise as a parameter, and GradNorm balances by gradient magnitude; both are more principled than hand-tuning, though hand-tuned published values are usually fine as a starting point.

8 · Per-architecture recipes

These are starting points drawn from the reference implementations and papers, not universal optima. The value of writing them side by side is seeing why they differ — the pattern is that architecture dictates the normaliser, the normaliser and pretraining dictate the optimizer, and the task dictates the loss.

8.1 YOLO (detection)

8.2 YOLO segmentation and Mask R-CNN

Instance segmentation adds a mask branch, and the practical differences from detection are narrow but important:

• An extra loss term — per-instance mask BCE (Mask R-CNN) or a mask-coefficient formulation (YOLO-seg). Watch its weight relative to the box and class terms (§7.5).
• Mask R-CNN runs at batch 2 per GPU in the standard recipe, which is why it uses frozen BatchNorm or GroupNorm. Leaving BN trainable at batch 2 is a reliable way to get a model that trains but does not generalise.
• Mask R-CNN: SGD, LR 0.02 at batch 16, ×0.1 at epochs 8 and 11 of a 12-epoch ("1×") schedule, weight decay $10^{-4}$. These numbers are so standard that departures from them should be deliberate.
• Mask resolution matters more than mask loss weight — the 28×28 RoI mask is the accuracy ceiling for fine boundaries, and no amount of loss tuning changes that.
• Copy-paste augmentation is unusually effective for rare classes here.

8.3 Faster R-CNN

Anchor mismatch is the most under-diagnosed failure in two-stage detection. If your objects are mostly tiny and the default anchors start at 32 px, recall is capped by the anchor grid no matter how long you train. Cluster your ground-truth box dimensions and compare against the anchor set before touching anything else.

8.4 ViT

8.5 CLIP-style contrastive training

• Batch size is a loss hyperparameter, not just a throughput choice — it sets the number of negatives (§7.4). Gather embeddings across all ranks before computing the loss.
• AdamW with $\beta_2 = 0.98$, weight decay 0.2, cosine schedule with warmup.
• Learn the temperature but clamp it (CLIP caps the logit scale at 100).
• Mixed precision is near-mandatory at this scale, but keep the loss computation and the softmax in fp32 — a large logit matrix in fp16 overflows.
• For fine-tuning CLIP, note that naive full fine-tuning degrades its robustness; interpolating the fine-tuned and original weights (WiSE-FT) recovers much of it.

8.6 Fast / real-time segmentation

Efficient segmentation architectures (FastSeg-style, BiSeNet, DDRNet and similar) share a set of training conventions:

• Auxiliary losses on intermediate outputs, weighted ~0.4, to get gradient into early layers of a deep encoder. Removing them typically costs a point of mIoU.
• Online hard example mining or top-$k$ pixel cross-entropy — most pixels are easy, and averaging over all of them dilutes the signal from boundaries.
• Poly LR schedule $\eta_t = \eta_0(1 - t/T)^{0.9}$ is the segmentation convention, and works about as well as cosine.
• Large crops with scale jitter (0.5–2.0×); SyncBatchNorm across GPUs because batch-per-GPU is small.
• Evaluate at the native resolution, and be explicit about whether reported mIoU uses sliding-window or whole-image inference — the two differ by enough to invalidate comparisons.

9 · Getting the PyTorch loop right

9.1 A correct baseline loop

Every line flagged above is a real bug someone has shipped. The ordering of unscale_ before clipping is the subtlest: clipping a scaled gradient clips to the wrong threshold, so the clipping silently does nothing useful.

9.2 The correctness checklist

9.3 Mixed precision

Mixed precision roughly doubles throughput and halves activation memory, and there is rarely a reason not to use it. The two variants behave differently:

Keep in fp32 regardless: loss reduction over a large batch, softmax over many classes or a large contrastive logit matrix, and any accumulation of a long sum. Also raise Adam's $\epsilon$ to $10^{-6}$ (§6.3). If loss becomes NaN only under AMP, the cause is almost always an overflow in one of those places rather than a genuine modelling problem.

9.4 Reproducibility

Full determinism costs some speed but makes debugging tractable — without it you cannot tell a real regression from run-to-run variance:

A pragmatic middle ground for real projects: seed everything and accept cuDNN nondeterminism, but always report the mean and spread over at least 3 seeds. Single-run comparisons of two configurations differing by less than a point are not evidence of anything — seed variance on a detection task is routinely a few tenths of mAP.

9.5 Multi-GPU

• Use DistributedDataParallel, not DataParallel. The latter is single-process, unbalanced and slow.
• The effective batch is per-GPU batch × world size, so the LR scaling rule (§3.1) applies to the product. Forgetting this when moving from 1 to 8 GPUs is a classic accuracy regression.
• Use SyncBatchNorm when the per-GPU batch is small, or GroupNorm to sidestep the issue.
• Call sampler.set_epoch(epoch) or every rank sees the same order every epoch.
• Log and checkpoint from rank 0 only; validate on all ranks and reduce, or the metric is computed on a fraction of the data.
• For contrastive losses, gather across ranks (§8.5) — otherwise the effective negative count is the per-GPU batch, not the global one.

10 · Underfitting, overfitting and reading the curves

10.1 The diagnosis

The shape of the train and validation curves, not their absolute values, tells you which problem you have — and the two call for opposite responses, which is why misdiagnosing costs so much time.

10.2 Overfitting remedies, in order of effectiveness

1. More data. Nothing else comes close. Often cheaper than it sounds via targeted collection of the failure cases rather than uniform collection.
2. Stronger augmentation. The closest substitute for more data, because it encodes real invariances.
3. Transfer learning. A pretrained backbone is regularisation in the form of a prior.
4. Weight decay, then early stopping.
5. Stochastic depth / dropout in the head.
6. Smaller model — last, because it also lowers the ceiling.
7. Label smoothing, EMA of weights, ensembling — each a small, reliable gain.

11 · When training will not converge

11.1 The diagnostic ladder

Work through these in order. Each rung isolates one class of cause, and the order is chosen so that the cheapest and most common causes are eliminated first.

11.2 Instability and NaN

Two guards worth adding permanently to any long training job: assert that the loss is finite every step and dump the offending batch if not; and keep a rolling checkpoint of the last few epochs so a late divergence does not cost the whole run. Both are a few lines and have saved a great many GPU-days.

11.3 What to log

A run you cannot diagnose afterwards is a run you will have to repeat. The minimum useful set:

• Each loss term separately, not just the total (§7.5).
• Learning rate per step — this catches schedule bugs instantly, and they are common.
• Gradient norm before clipping, and how often clipping fires.
• Validation metrics on the metric you actually care about, not only the loss. Loss and mAP do not always move together.
• Throughput (samples/s) and GPU utilisation — a data-bound run wastes most of its budget.
• The full config and the git commit, attached to the run.
• Sample predictions as images, periodically. Numbers hide failure modes that a picture makes obvious.

12 · Search strategy on a budget

Two practical points. Tune on a proxy task first — a subset of the data, a smaller model, a shorter schedule — because most hyperparameter rankings transfer even when absolute values do not; then confirm the top few configurations at full scale. And hold out a genuine test set that is never used for selection: tuning hundreds of configurations against a validation set overfits to that set, and the reported number stops being an estimate of generalisation.

13 · Further reading

• Google Research, Deep Learning Tuning Playbook (Godbole, Dahl, Gilmer, Shallue, Nado). The best single practical document on this subject — trial budgets, what to tune in what order, and the batch-size/LR relationship treated carefully.
• Karpathy, "A Recipe for Training Neural Networks." The origin of the overfit-one-batch discipline and the incremental-complexity approach in §11.1.
• PyTorch documentation — the AMP recipe, torch.optim notes, reproducibility page, and the DDP tutorial. The AMP page in particular documents the unscale_-before-clip ordering.
• Bergstra & Bengio (2012), "Random Search for Hyper-Parameter Optimization," JMLR 13. Why grid search wastes the budget.
• Smith (2017), "Cyclical Learning Rates for Training Neural Networks" — the LR range test; and Smith & Topin (2019) on one-cycle / super-convergence.
• Goyal et al. (2017), "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." Linear LR scaling and why warmup is required.
• Shallue et al. (2019), "Measuring the Effects of Data Parallelism on Neural Network Training," JMLR. The critical-batch-size result behind §3.1.
• Loshchilov & Hutter (2019), "Decoupled Weight Decay Regularization" (AdamW); and (2017) "SGDR: Stochastic Gradient Descent with Warm Restarts" (cosine schedules).
• Lin et al. (2017), "Focal Loss for Dense Object Detection."
• Zheng et al. (2020), "Distance-IoU Loss," and Rezatofighi et al. (2019) on GIoU.
• Wu & He (2018), "Group Normalization." The clearest analysis of why BatchNorm fails at small batch.
• Ioffe & Szegedy (2015) (BatchNorm) and Ba, Kiros & Hinton (2016) (LayerNorm).
• Steiner et al. (2021), "How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers." A large, careful study of exactly the trade-offs in §8.4.
• Radford et al. (2021), "Learning Transferable Visual Models From Natural Language Supervision" (CLIP) — see the appendix for the training configuration.
• Li et al. (2020), "A System for Massively Parallel Hyperparameter Tuning" (ASHA).
• Ultralytics and Detectron2 configuration defaults — worth reading as documentation of what a well-tuned recipe actually contains.

Related pages: Deep Learning · mAP, recall & MOTA · Model port & testing · Perception stack