Camera model & projection

Every geometric vision algorithm on this site — PnP, stereo, SfM, bundle adjustment — is built on one equation that maps a 3-D world point to a pixel. This page derives that equation completely: the rigid transform that moves the point into the camera frame, the perspective division that discards range, the intrinsic matrix that converts to pixels, and the distortion model that describes what a real lens does differently. Getting the conventions right here is what makes everything downstream work.

Symbols used on this page are defined in the Nomenclature → camera & multi-view geometry.

Notation follows the OpenCV / Hartley–Zisserman convention throughout, including the sign and ordering choices that cause most of the practical confusion.

The whole model in one line. $$ s\,\tilde{\mathbf x} = \mathbf K \left[\mathbf R \mid \mathbf t\right] \tilde{\mathbf X} $$ A world point $\tilde{\mathbf X}$ (homogeneous, 4-vector) is rotated and translated into the camera frame by $[\mathbf R \mid \mathbf t]$, then mapped to a homogeneous pixel $\tilde{\mathbf x}$ by $\mathbf K$. The scalar $s$ is the lost depth: projection is a many-to-one map, and recovering $s$ is the entire business of stereo, triangulation and depth sensing.

1 · The pinhole camera

1.1 Geometry

Idealise the lens as a single point — the optical centre $C$ — through which all rays pass. Place the camera frame at $C$ with $Z$ along the optical axis, $X$ right and $Y$ down. A point at $(X_c, Y_c, Z_c)$ and the image plane at distance $f$ in front of $C$ give, by similar triangles,

$$ x = f\,\frac{X_c}{Z_c}, \qquad y = f\,\frac{Y_c}{Z_c} . $$

image plane │ world point P ●────────┼─────────────────● C (optical centre) ╲ │p ╱ │ ╲ │ ╱ │ ╲ │ ╱ │ optical axis (Z) ╲ │ ╱ ▼ ╲ │ ╱ ● p'│ ╱ │ ├── f ──┤ Z_c ├─────────────────────────────┤ similar triangles: x / f = X_c / Z_c → x = f·X_c / Z_c

Two consequences worth stating explicitly. The division by $Z_c$ makes projection nonlinear, which is why everything in this section ends up as iterative least squares rather than a linear solve. And $Z_c$ is destroyed by the division, so a single image determines only the ray a point lies on, not its position — a camera is a bearing sensor.

The physical image plane sits behind the pinhole and the image is inverted. The convention above places a virtual image plane in front at the same distance, which removes the double sign flip and is what every library implements. The "camera looks down $+Z$, with $Y$ down" convention is OpenCV's; some graphics systems use $-Z$ forward and $Y$ up, and mixing them is a classic source of upside-down reconstructions.

1.2 Normalised image coordinates

It pays to name the intermediate stage where the perspective division has happened but no camera-specific units have been applied. Setting $f = 1$:

$$ x_n = \frac{X_c}{Z_c}, \qquad y_n = \frac{Y_c}{Z_c}, \qquad \tilde{\mathbf x}_n = \begin{bmatrix} x_n \\ y_n \\ 1 \end{bmatrix}. $$

These normalised coordinates are dimensionless and camera-independent: two different cameras looking along the same ray produce the same $(x_n, y_n)$. This is the natural space for geometry — the essential matrix on the SfM page and the law-of-cosines P3P derivation on the PnP page both live here — and, importantly, it is where distortion is defined (§5). The pipeline is therefore:

world ──[R|t]──► camera 3-D ──÷Z──► normalised ──distort──► distorted ──K──► pixels X_w X_c, Y_c, Z_c x_n, y_n x_d, y_d u, v metres dimensionless dimensionless px

2 · Extrinsics — where the camera is

2.1 The rigid transform

The extrinsics express the world frame in camera coordinates:

$$ \mathbf X_c = \mathbf R\,\mathbf X_w + \mathbf t, \qquad \mathbf R \in SO(3), \quad \mathbf t \in \mathbb{R}^3 . $$

Six degrees of freedom: three for rotation, three for translation.

$\mathbf t$ is not the camera position. This is the single most common sign error in camera code. $\mathbf t$ is the origin of the world frame expressed in camera coordinates. The camera centre in world coordinates is $$ \mathbf C = -\,\mathbf R^\top \mathbf t , $$ obtained by setting $\mathbf X_c = \mathbf 0$. Likewise $\mathbf R$ maps world directions into the camera frame, so the camera's viewing direction in world coordinates is the third row of $\mathbf R$, i.e. $\mathbf R^\top \hat{\mathbf z}$. Plotting $\mathbf t$ as a camera position produces a trajectory that looks plausible but is reflected through the origin and rotated.

2.2 Parameterising the rotation

Representation	Params	Notes
Rotation matrix	9 (6 constraints)	What the algebra uses; over-parameterised for optimisation
Rodrigues / axis-angle	3	OpenCV's `rvec`. Minimal and singularity-free near zero; the natural increment
Quaternion	4 (1 constraint)	Good for storage and interpolation; needs renormalisation
Euler angles	3	Avoid — gimbal lock and 24 ordering conventions

For optimisation the increment is taken in the Lie algebra and applied multiplicatively, $\mathbf R \leftarrow \exp([\delta\boldsymbol\omega]_\times)\,\mathbf R$, which keeps the matrix exactly orthonormal — the same discipline used in bundle adjustment and derived on the attitude page.

3 · Intrinsics — the camera matrix

3.1 From metres to pixels

Normalised coordinates must be scaled to pixels and shifted so the origin is the corner of the image rather than the optical axis. With pixel sizes $s_x, s_y$ (metres per pixel) and focal length $F$ in metres, $f_x = F/s_x$ and $f_y = F/s_y$. In matrix form:

$$ \mathbf K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}, \qquad \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \mathbf K \begin{bmatrix} x_n \\ y_n \\ 1 \end{bmatrix} . $$

Expanded, this is simply

$$ u = f_x\,x_n + s\,y_n + c_x, \qquad v = f_y\,y_n + c_y . $$

Symbol	Meaning	Units	Typical
$f_x, f_y$	focal length in pixels	px	Equal to within ~1% for square pixels
$c_x, c_y$	principal point — where the optical axis meets the sensor	px	Near the image centre, but not exactly
$s$	skew — non-perpendicular sensor axes	px	0 for any modern sensor
$F$	physical focal length	mm	lens specification
$s_x, s_y$	pixel pitch	µm	1–6 µm

Note that $f_x \ne f_y$ encodes non-square pixels, not an optical property — the lens has one focal length. A large discrepancy in a calibration result usually means the image was anisotropically resized, or the calibration is poorly conditioned.

3.2 What each parameter does to the image

$\mathbf K$ is an affine map of the image plane, so it cannot change any 3-D relationship — only how the projected picture is laid out in pixels:

Change	Effect on the image	Effect on geometry
$f_x, f_y$ up	scales the image about the principal point — narrower FOV	none — but perspective becomes weaker, hurting depth observability
$c_x, c_y$ shift	translates the whole image rigidly	none
$s \ne 0$	shears the image	none

The simulation below applies the complete pipeline to a wireframe box and walks one corner through every stage numerically. Worth doing: shift the principal point and confirm the shape is unchanged while the whole projection slides; then push the focal length up and watch the field of view collapse while the box flattens toward an affine (parallel-projection) look — the regime where PnP loses depth observability.

3.3 Field of view

Focal length and sensor size together determine the angular coverage:

$$ \mathrm{FOV}_h = 2\arctan\!\left(\frac{w}{2 f_x}\right), \qquad \mathrm{FOV}_v = 2\arctan\!\left(\frac{h}{2 f_y}\right), $$

with $w, h$ the image dimensions in pixels. The inverse is more useful in practice — given a required field of view, the focal length you need is $f_x = w / \left(2\tan(\mathrm{FOV}_h/2)\right)$:

Horizontal FOV	$f_x$ for $w = 1280$	Character
120°	369 px	Fisheye-ish; pinhole model starting to strain
90°	640 px	Wide — good parallax, strong distortion
60°	1109 px	Typical machine vision
30°	2389 px	Narrow; weak perspective, depth poorly observable

Resizing an image changes the intrinsics. Scaling by $\alpha$ scales $f_x, f_y, c_x, c_y$ and $s$ by $\alpha$; cropping by $(\Delta u, \Delta v)$ subtracts from $c_x, c_y$ and leaves the focal lengths alone. The distortion coefficients are unchanged, because they act on normalised coordinates (§5). A resize that forgets to rescale $\mathbf K$ is a very common bug in learned-perception pipelines, and it shows up as a systematic depth or pose bias rather than an obvious failure.

4 · The projection matrix

Composing intrinsics and extrinsics gives the $3\times4$ camera projection matrix:

$$ \mathbf P = \mathbf K\left[\mathbf R \mid \mathbf t\right] \in \mathbb{R}^{3\times4}, \qquad s\,\tilde{\mathbf x} = \mathbf P\,\tilde{\mathbf X}. $$

Written out, the full chain from world to pixel is

$$ s\begin{bmatrix} u \\ v \\ 1\end{bmatrix} = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_1 \\ r_{21} & r_{22} & r_{23} & t_2 \\ r_{31} & r_{32} & r_{33} & t_3 \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}, $$

and the recovered scale is $s = Z_c$, the depth in the camera frame. $\mathbf P$ has 11 degrees of freedom (12 entries, defined up to scale) — 5 intrinsic and 6 extrinsic — which is exactly why the DLT needs at least 6 point correspondences to solve for it linearly.

Two operations are worth knowing. Given $\mathbf P$, the intrinsics and rotation are recovered by an RQ decomposition of its left $3\times3$ block, since $\mathbf K$ is upper triangular and $\mathbf R$ orthogonal. And the camera centre is the right null vector, $\mathbf P\tilde{\mathbf C} = \mathbf 0$, which is the homogeneous statement of $\mathbf C = -\mathbf R^\top\mathbf t$.

4.1 Back-projection

Inverting the model recovers a ray, not a point:

$$ \mathbf X_c(\lambda) = \lambda\,\mathbf K^{-1}\tilde{\mathbf x}, \qquad \lambda \gt 0 , $$

and in world coordinates the ray starts at $\mathbf C$ with direction $\mathbf d = \mathbf R^\top\mathbf K^{-1}\tilde{\mathbf x}$. Recovering $\lambda$ needs an extra constraint — a second view (stereo, triangulation), a known scene plane, a known object size, or an active depth sensor. If the point is known to lie on the plane $Z_w = 0$, for instance, the ray-plane intersection is a closed-form solve, which is the basis of ground-plane monocular ranging.

5 · Lens distortion

5.1 The model

Real lenses are not pinholes. The standard Brown–Conrady model, which is what OpenCV implements, applies a correction to the normalised coordinates before $\mathbf K$. With $r^2 = x_n^2 + y_n^2$:

$$ x_d = x_n\underbrace{\left(1 + k_1 r^2 + k_2 r^4 + k_3 r^6\right)}_{\text{radial}} + \underbrace{2p_1 x_n y_n + p_2\left(r^2 + 2x_n^2\right)}_{\text{tangential}} $$ $$ y_d = y_n\left(1 + k_1 r^2 + k_2 r^4 + k_3 r^6\right) + p_1\left(r^2 + 2y_n^2\right) + 2p_2 x_n y_n $$ $$ \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x x_d + s\,y_d + c_x \\ f_y y_d + c_y \end{bmatrix} $$

Coefficient	Models	Behaviour
$k_1$	dominant radial term	Negative → barrel (straight lines bow outward; wide-angle lenses). Positive → pincushion (telephoto)
$k_2, k_3$	higher-order radial	Matter only for strong distortion and near the corners. $k_3$ is often fixed at 0
$p_1, p_2$	tangential / decentring	Sensor not exactly parallel to the lens. Usually tiny; estimating them on clean data can add noise

The key structural point: distortion is a function of radius. It is zero at the principal point and grows steeply toward the corners, as $r^2$ at least. The simulation makes this concrete — the displacement curve is flat in the middle and turns sharply upward at the edge.

A reprojection error averaged over the image hides distortion error. Because the displacement is concentrated at the periphery, a calibration with a good mean RMS can still be badly wrong in the corners — tens of pixels, as the sim shows. Two consequences: check reprojection residuals as a function of image radius, not just their mean; and ensure the calibration images actually cover the corners, or the higher-order coefficients are extrapolating from data that was never collected.

5.2 Undistortion is not analytically invertible

The forward model maps ideal to distorted. Undistortion needs the inverse, and the polynomial has no closed-form inverse. Two standard approaches:

Iterate per point. Start with $(x_n, y_n) = (x_d, y_d)$ and repeatedly re-apply the inverse relation; a handful of iterations converges for moderate distortion. This is what undistortPoints does, and it is the right choice for sparse features.
Build a lookup map once. For each output pixel of the undistorted image, compute where to sample in the distorted image (the forward direction, which is closed form), and store that map. Then undistorting a frame is a single interpolated remap. This is initUndistortRectifyMap + remap, and it is how dense per-frame undistortion is made cheap.

Note the direction carefully: the map is built from the destination to the source, which is the opposite of what most people first write. Forward-warping each source pixel to a destination leaves holes; inverse-mapping each destination pixel does not.

Undistortion also forces a framing choice, because the undistorted image is not rectangular. A free-scaling parameter trades black border against lost field of view: keep all pixels and accept curved black edges, or crop to the largest valid rectangle and throw away the periphery. Either is defensible, but the resulting $\mathbf K$ is different from the original and must be carried forward — a new principal point and often a new focal length.

5.3 When the pinhole model runs out

Beyond roughly 120° of field of view the polynomial model fits poorly, because the true projection is no longer even approximately $\tan\theta$. Fisheye and omnidirectional lenses need a different parameterisation built on the incidence angle $\theta$ itself:

$$ \theta = \arctan r, \qquad \theta_d = \theta\left(1 + k_1\theta^2 + k_2\theta^4 + k_3\theta^6 + k_4\theta^8\right), $$

with the distorted radius then $\theta_d$ rather than a scaling of $r$. This is the Kannala–Brandt model (OpenCV's fisheye module); the double-sphere and unified camera models are common alternatives for very wide and catadioptric systems. Everything else on this page — extrinsics, triangulation, the epipolar constraint — carries over unchanged once the projection function is replaced, because those parts only need some map from ray to pixel.

6 · Calibration

6.1 What it solves

Calibration estimates the intrinsics and distortion from images of a known target — and, as a by-product, the pose of the target in every view. With $N$ views of a planar target holding $M$ points:

$$ \min_{\mathbf K,\,\mathbf d,\,\{\mathbf R_i, \mathbf t_i\}} \sum_{i=1}^{N}\sum_{j=1}^{M} \left\| \mathbf x_{ij} - \pi\!\left(\mathbf K, \mathbf d, \mathbf R_i, \mathbf t_i, \mathbf X_j\right) \right\|^2 . $$

This is a bundle-adjustment problem with the structure known and fixed. The unknown count is $4 + 5$ intrinsic and distortion parameters plus $6N$ extrinsic — so more views add unknowns as well as data, and the benefit comes only from diverse views.

6.2 Zhang's method

The classical procedure, and still what most implementations do:

1. detect target points in each image (corners, circles, ChArUco) 2. for each view, fit a HOMOGRAPHY H_i from the target plane to the image (4-point DLT + normalisation) 3. each H_i constrains the image of the absolute conic: h₁ᵀ ω h₂ = 0 , h₁ᵀ ω h₁ = h₂ᵀ ω h₂ where ω = K⁻ᵀK⁻¹ → 2 linear constraints per view on the 5 unknowns of ω 4. solve linearly for ω with N ≥ 3 views, then get K by Cholesky factorisation of ω ── closed form, no distortion yet 5. recover R_i, t_i per view from H_i and K 6. initialise distortion at 0, then NONLINEAR REFINEMENT of everything on true reprojection error ── the step that matters

The pattern is the one seen throughout these pages: a linear method ignoring the real noise model to get an initial estimate, then iterative refinement of the statistically correct cost. Steps 1–5 alone are not accurate enough to use.

6.3 Conditioning — why calibrations go wrong

The failures are almost always about the data, not the solver:

Problem	Symptom	Fix
All views at the same orientation	$f$ and $Z$ trade off; focal length is badly determined	Tilt the target strongly, ±30–45°, in several directions. This is the single most important rule
Target only in the image centre	Distortion coefficients are unconstrained, then extrapolate wildly	Cover the corners deliberately
Target too small in frame	Everything poorly conditioned	Fill a large fraction of the image
Purely frontal planar views	$f_x/f_y$ and skew ambiguous	Tilt; or fix $s = 0$ and $f_x = f_y$ as priors
Too many free parameters	Low RMS, poor generalisation — overfitting	Fix $k_3 = 0$, $s = 0$; estimate tangential terms only if justified
Non-planar target assumed planar	Systematic residual pattern	Use a rigid, flat target; paper on a wall is not flat enough

RMS reprojection error is a necessary but weak check. A low RMS means the model fits the calibration images; it does not mean the parameters are right. More diagnostic: look at residuals versus image radius (catches distortion mis-fit), inspect the covariance of the estimated parameters (catches the $f$–$Z$ trade), and validate on a held-out measurement — image a known length at a known distance and check the recovered value. This is the same distinction drawn on the PnP page §5: fit quality and parameter determination are different things.

7 · Where the model enters each algorithm

Algorithm	Uses	Assumes known
PnP	full forward projection; $\partial\mathbf x/\partial\mathbf X_c$	$\mathbf K$, distortion
Stereo	rectification from two $\mathbf K$'s and the relative pose	both intrinsics and the extrinsic baseline
SfM	normalised coordinates for the essential matrix	$\mathbf K$ (else it estimates it)
Bundle adjustment	the projection Jacobian per observation	nothing — can refine $\mathbf K$ and distortion too
Visual SLAM	measurement model $\mathbf h(\cdot)$ and its Jacobian	$\mathbf K$, distortion

The projection Jacobian is worth writing once, because it appears in all of them. Differentiating $u = f_x X_c/Z_c + c_x$, $v = f_y Y_c/Z_c + c_y$:

$$ \frac{\partial(u,v)}{\partial \mathbf X_c} = \frac{1}{Z_c}\begin{bmatrix} f_x & 0 & -f_x X_c/Z_c \\ 0 & f_y & -f_y Y_c/Z_c \end{bmatrix} . $$

The $1/Z_c$ factor is the quantitative statement that distant points carry little information: their image position barely responds to camera translation, which is why far features constrain rotation well and translation poorly.

8 · Conventions that cause bugs

$\mathbf t$ versus camera position. $\mathbf C = -\mathbf R^\top\mathbf t$ (§2.1). Check this first when a trajectory looks mirrored.
World-to-camera versus camera-to-world. Both are called "the extrinsics". OpenCV's rvec/tvec are world-to-camera; many SLAM systems store camera-to-world. They are inverses, not negatives.
Pixel centre convention. Is integer $(0,0)$ the centre of the first pixel or its corner? A half-pixel offset is small, systematic, and survives every averaging.
$Y$ down versus $Y$ up. OpenCV has $Y$ down, $Z$ forward. OpenGL has $Y$ up, $Z$ backward. Mixing them flips the image vertically and changes the rotation's handedness.
Distortion applied at the wrong stage. It acts on normalised coordinates, before $\mathbf K$ — never on pixels.
Intrinsics not updated after resize or crop (§3.3).
Row-major versus column-major when moving matrices between libraries; a silently transposed $\mathbf R$ is still orthonormal, so no assertion fires.

A cheap and very effective guard against most of these: write a round-trip unit test that projects a set of known 3-D points, back-projects the pixels to rays, and asserts the rays pass through the original points. It catches sign errors, transposes and stage-ordering mistakes immediately.

9 · Further reading

OpenCV documentation — "Camera Calibration and 3D Reconstruction" (the calib3d module overview). The clearest statement of the exact equations and coefficient ordering used by the most widely deployed implementation, with the classic barrel/pincushion and pinhole-geometry illustrations. Also the fisheye module page for the wide-FOV model.
Hartley & Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., chapter 6 ("Camera Models") and chapter 8. The definitive treatment, including the RQ decomposition of $\mathbf P$ and the absolute conic used by Zhang's method.
Zhang (2000), "A Flexible New Technique for Camera Calibration," IEEE TPAMI 22(11). The planar-target method behind essentially every calibration tool in use.
Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., chapter 2 for the image-formation model and chapter 11 for calibration. Freely available online.
Brown (1966), "Decentering Distortion of Lenses." The origin of the radial/tangential polynomial model.
Kannala & Brandt (2006), "A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses," IEEE TPAMI 28(8).
Usenko, Demmel & Cremers (2018), "The Double Sphere Camera Model," 3DV. A practical modern wide-FOV model with cheap projection and unprojection.
Bouguet, Camera Calibration Toolbox for MATLAB — the documentation remains one of the best explanations of calibration conditioning and what good target coverage looks like.