PnP — pose from correspondences

Given $n$ known 3-D points and where they appear in one calibrated image, recover the camera's six degrees of freedom. This is the estimator at the heart of visual localisation, marker tracking, spacecraft relative navigation and every SLAM system's tracking thread. The geometry is exact and the closed-form solvers are fast — which makes it easy to miss that PnP's real difficulty is conditioning: the six numbers it returns are never equally well determined, and a small reprojection error does not mean a good pose.

Symbols used on this page are defined in the Nomenclature → camera & multi-view geometry.

PnP almost always runs inside a RANSAC loop; the triangulation that produces the 3-D points in the first place is on the SfM page.

1 · Problem setup

1.1 The projection model

A world point $\mathbf X_i \in \mathbb{R}^3$ maps to image coordinates through a rigid transform and a perspective division:

$$ \mathbf X^c_i = \mathbf R\,\mathbf X_i + \mathbf t, \qquad \mathbf x_i = \pi\!\left(\mathbf X^c_i\right) = \begin{bmatrix} f_x \dfrac{X^c_i}{Z^c_i} + c_x \\[2mm] f_y \dfrac{Y^c_i}{Z^c_i} + c_y \end{bmatrix}. $$

The intrinsics $\mathbf K$ are known — that is what distinguishes PnP from full camera calibration. The unknowns are $\mathbf R \in SO(3)$ and $\mathbf t \in \mathbb{R}^3$. The estimate we actually want minimises reprojection error:

$$ \left(\hat{\mathbf R}, \hat{\mathbf t}\right) = \arg\min_{\mathbf R,\,\mathbf t} \sum_{i=1}^{n} \left\| \mathbf x_i - \pi\!\left(\mathbf R \mathbf X_i + \mathbf t\right) \right\|^2 . $$

1.2 Why reprojection error is the right cost

The measurement noise lives in the image — it comes from feature localisation, a few tenths of a pixel. If that noise is isotropic Gaussian in the image plane, then minimising the sum of squared reprojection errors is exactly the maximum-likelihood estimate. Costs defined in 3-D (for example minimising distance between back-projected rays and points) are algebraically convenient but correspond to a noise model nothing actually has, and they weight distant points incorrectly. The practical consequence: use algebraic costs to initialise, then always polish against reprojection error.

2 · P3P — the minimal case

2.1 The law-of-cosines system

With three points, work in terms of the unknown depths $d_i = \|\mathbf X^c_i\|$. The angles between the back-projected rays are known from the image and the calibration:

$$ \cos\theta_{ij} = \hat{\mathbf u}_i^\top \hat{\mathbf u}_j, \qquad \hat{\mathbf u}_i = \frac{\mathbf K^{-1}\tilde{\mathbf x}_i}{\left\|\mathbf K^{-1}\tilde{\mathbf x}_i\right\|}, $$

and the distances $L_{ij} = \|\mathbf X_i - \mathbf X_j\|$ between the world points are also known. The law of cosines on each of the three triangles gives three equations in three unknown depths:

$$ L_{ij}^2 = d_i^2 + d_j^2 - 2\,d_i d_j \cos\theta_{ij}, \qquad (i,j) \in \{(1,2),(1,3),(2,3)\}. $$

This is a system of three quadratics. Eliminating two unknowns yields a quartic in the remaining one — hence up to four real positive solutions. Each admissible depth triple fixes the three points in the camera frame, and the rotation and translation then follow from the same Procrustes/SVD solve used for point-cloud registration, since we now have three 3-D–3-D correspondences.

2.2 The danger circle

There is a worse case. If the camera centre lies on the circle passing through the three world points (in their plane), the problem becomes degenerate — a continuum of poses explains the observations, not merely four. Near that circle the solution is ill-conditioned even if not exactly singular. This matters practically for planar marker tracking, where a camera orbiting a three-point target can pass through the danger configuration. It is another instance of the degeneracy discussed in RANSAC §6: the residual is small, the answer is meaningless.

3 · Linear and closed-form solvers for general $n$

3.1 DLT — the direct linear transform

The simplest general approach ignores the structure of $SO(3)$ and solves for the 12 entries of the projection matrix $\mathbf P = [\mathbf R \mid \mathbf t]$ linearly. Each correspondence contributes two rows to a homogeneous system $\mathbf A\mathbf p = \mathbf 0$, solved by the singular vector of the smallest singular value, then $\mathbf R$ is recovered by projecting the upper $3\times3$ block back onto $SO(3)$ via SVD.

It needs $n \ge 6$, it is trivial to implement, and it is statistically poor: minimising an algebraic residual is not minimising reprojection error, and discarding the orthonormality constraint throws away known structure, so noise is amplified. DLT is acceptable as an initialiser for iterative refinement and nothing more. As with all linear vision solvers, Hartley normalisation of the coordinates is mandatory or the conditioning is dreadful.

3.2 EPnP — $O(n)$ and accurate

EPnP (Lepetit, Moreno-Noguer & Fua, 2009) is the standard modern choice, and its idea is elegant. Express every world point as a weighted sum of four virtual control points:

$$ \mathbf X_i = \sum_{j=1}^{4} \alpha_{ij}\,\mathbf C_j, \qquad \sum_{j=1}^{4}\alpha_{ij} = 1 . $$

Because the relation is affine, the same barycentric weights hold in the camera frame. So instead of $n$ unknown depths, the unknowns are the 4 control points in the camera frame — 12 numbers, independent of $n$. The projection equations become a linear system whose solution lies in the null space of a $2n\times12$ matrix. That null space has dimension 1 to 4 depending on configuration; EPnP writes the solution as a combination of null-space vectors and fixes the coefficients using the known rigidity constraints (the distances between control points must be preserved). The result is $O(n)$ in time, far more accurate than DLT, and needs only $n \ge 4$. This is what OpenCV's default solvePnP flag family is built around.

3.3 Choosing a solver

4 · Iterative refinement

4.1 Gauss-Newton on the manifold

The closed-form solvers minimise algebraic proxies. The maximum-likelihood estimate requires iterating on the true reprojection cost. Parameterise the update in the Lie algebra so the rotation stays on $SO(3)$ by construction:

$$ \mathbf R \leftarrow \exp\!\left([\delta\boldsymbol\omega]_\times\right)\mathbf R, \qquad \mathbf t \leftarrow \mathbf t + \delta\mathbf t, \qquad \delta\mathbf x = \begin{bmatrix}\delta\boldsymbol\omega \\ \delta\mathbf t\end{bmatrix} \in \mathbb{R}^6 . $$

The Jacobian of one point's 2-D residual factors into the projection derivative times the rigid-motion derivative:

$$ \frac{\partial \mathbf x}{\partial \mathbf X^c} = \frac{1}{Z^c}\begin{bmatrix} f_x & 0 & -f_x X^c/Z^c \\[1mm] 0 & f_y & -f_y Y^c/Z^c \end{bmatrix}, \qquad \frac{\partial \mathbf X^c}{\partial \delta\mathbf x} = \begin{bmatrix} -[\mathbf X^c]_\times & \mathbf I_3 \end{bmatrix}, $$ $$ \mathbf J_i = \frac{\partial \mathbf x}{\partial \mathbf X^c}\, \begin{bmatrix} -[\mathbf X^c_i]_\times & \mathbf I_3\end{bmatrix} \in \mathbb{R}^{2\times6}. $$

Stacking gives $\mathbf J \in \mathbb{R}^{2n\times6}$, and each Gauss-Newton step solves a $6\times6$ system:

$$ \left(\mathbf J^\top \mathbf J + \lambda \mathbf I\right)\delta\mathbf x = -\,\mathbf J^\top \mathbf r . $$

The $\lambda$ term is Levenberg–Marquardt damping — essential here because $\mathbf J^\top\mathbf J$ is frequently near-singular in exactly the configurations discussed in §5. Convergence from a decent initialisation takes 3–5 iterations, and each costs a single pass over the points plus a $6\times6$ Cholesky, so refinement is cheap relative to the closed-form solve it improves.

Note the $1/Z^c$ in the projection derivative: distant points have small Jacobians and therefore contribute little information about translation. This is the algebraic origin of the observation in §5 that a long-focal, far-away target barely constrains depth.

4.2 Pose covariance

A pose without an uncertainty is not usable by a filter. For isotropic image noise $\sigma$, the first-order covariance of the 6-vector is

$$ \operatorname{Cov}\!\left(\delta\mathbf x\right) \approx \sigma^2 \left(\mathbf J^\top \mathbf J\right)^{-1} . $$

Two things follow, and both are practically important. First, this is nearly free — $\mathbf J^\top \mathbf J$ was already formed and factored for the last Gauss-Newton step. Second, its anisotropy is the real diagnostic: the diagonal entries tell you which degrees of freedom the geometry pinned down and which it did not, information that a single scalar reprojection residual completely hides.

4.3 The sim

The simulation builds $\mathbf J$ numerically, forms $\mathbf J^\top\mathbf J$, inverts it, and plots the per-parameter $1\sigma$. The experiments that matter:

• Drive target depth spread to zero. The target becomes planar and the out-of-plane rotations and depth degrade sharply — this is the ambiguity of §5.2 showing up in the information matrix.
• Increase focal length while keeping the target shallow. The information spread explodes: a narrow field of view sees almost parallel rays, and translation along the optical axis becomes nearly indistinguishable from scaling the target.
• Increase camera distance. Same effect — the target subtends a smaller angle, so depth information collapses.
• Increase $n$. Everything improves as $1/\sqrt{n}$, but it does not fix a degenerate configuration: more points on a plane is still a plane.

5 · Conditioning and ambiguity

5.1 What makes a configuration good

The conditioning of PnP is governed by how much the projection changes when the pose changes. Concretely:

5.2 The planar two-fold ambiguity

Planar targets — chessboards, fiducial markers, printed patterns — are extremely common and carry a specific trap. For a plane viewed under weak perspective there are generally two distinct poses that produce nearly identical projections, related by a reflection of the plane's normal about the viewing direction. Under a pure affine (parallel-projection) approximation the two are exactly indistinguishable; what breaks the tie is only the small amount of genuine perspective in the image.

So the ambiguity is resolved by a weak signal, and its strength falls as the target gets smaller, flatter to the camera, or further away, and as focal length increases. The observable symptom is well known to anyone who has tracked a marker: the estimated pose flips between two orientations from frame to frame while reprojection error stays low in both. Mitigations:

• Use IPPE, which is derived for the planar case and returns both solutions with their residuals, so the caller can make an informed choice rather than being handed one at random.
• Apply temporal consistency: prefer the solution closer to the previous frame's pose.
• Make the target non-planar. Even a small out-of-plane offset removes the ambiguity — the most robust fix, when the target is under your control.
• Use a larger target, a shorter focal length, or get closer; all increase the perspective signal.

6 · Robustness

6.1 RANSAC-PnP

Correspondences come from feature matching, so a substantial fraction are wrong, and a single gross outlier destroys a least-squares pose. The standard architecture wraps a minimal solver in RANSAC:

Using $s=3$ rather than $s=6$ is a large win: at 50% inliers the trial count is 35 versus 293. The extra P3P candidates cost a little scoring, but scoring is cheap compared with the trials avoided. This is exactly the "minimal sample" argument from RANSAC §2.2.

6.2 The pose-behind-camera test

A free and highly effective filter: any solution placing a visible point at $Z^c \le 0$ is physically impossible. Checking the sign of the depths eliminates a large share of spurious P3P roots and mismatched samples before the expensive inlier count runs. Always include it.

6.3 When points are not equally reliable

Features have different localisation accuracy — a corner at fine scale is far better localised than a blob at coarse scale — and 3-D points from triangulation have very different uncertainties depending on their parallax (see SfM). Both should enter as weights:

$$ \min_{\mathbf R,\mathbf t}\sum_i \mathbf r_i^\top \boldsymbol\Sigma_i^{-1} \mathbf r_i , \qquad \boldsymbol\Sigma_i = \boldsymbol\Sigma^{\text{feat}}_i + \mathbf J^{X}_i \boldsymbol\Sigma^{\text{3D}}_i \mathbf J^{X\top}_i , $$

where the second term propagates the 3-D point's uncertainty into the image through the projection Jacobian. Ignoring it makes poorly triangulated distant points count as much as well-constrained nearby ones, which is a common source of quiet bias in visual odometry.

7 · Applications and what each one cares about

The pattern across these: the algorithm is the same, and the engineering effort goes into the configuration. A calibration that collects fifty images of a target at the same pose has fifty times the data and almost none of the extra information — the classic mistake, and one the covariance of §4.2 would have caught immediately.

8 · Practical checklist

• Undistort first, or use a solver that models distortion. Radial distortion is not small at the image periphery, and it biases the pose systematically rather than randomly.
• Always finish with iterative refinement on reprojection error. Closed-form solvers are initialisers.
• Use P3P/AP3P inside RANSAC, then EPnP or LM on the inliers.
• Include the depth-positivity test — free and effective.
• Use IPPE for planar targets and handle both solutions explicitly.
• Report the covariance, not just the residual (§4.2, §5.1). A pose consumed by a filter without an honest uncertainty will be either ignored or over-trusted.
• Monitor the condition number. A rising condition number is an early warning that the geometry is degrading, well before the pose looks wrong.
• Weight by feature and 3-D point uncertainty (§6.3).
• Design the target and the trajectory for observability: depth variation, wide angular coverage, varied poses.
• Test the degenerate cases deliberately: coplanar points, clustered points, a distant small target, the P3P danger circle.

9 · Further reading

• Hartley & Zisserman, Multiple View Geometry, 2nd ed., chapters 7 and 22. Pose estimation and the degenerate configurations.
• Fischler & Bolles (1981) — the same paper that introduced RANSAC also introduced the "perspective-$n$-point" terminology and analysed P3P's multiple solutions.
• Haralick et al. (1994), "Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem," IJCV 13. The careful treatment of the four solutions and the danger cylinder.
• Lepetit, Moreno-Noguer & Fua (2009), "EPnP: An Accurate O(n) Solution to the PnP Problem," IJCV 81.
• Kneip, Scaramuzza & Siegwart (2011), "A Novel Parametrization of the Perspective-Three-Point Problem for a Direct Computation of Absolute Camera Position and Orientation," CVPR. The basis of AP3P.
• Collins & Bartoli (2014), "Infinitesimal Plane-Based Pose Estimation," IJCV. IPPE and the clearest analysis of the planar ambiguity.
• Schweighofer & Pinz (2006), "Robust Pose Estimation from a Planar Target," IEEE TPAMI. Explains the two-minimum structure and why flipping happens.
• Terzakis & Lourakis (2020), "A Consistently Fast and Globally Optimal Solution to the Perspective-n-Point Problem," ECCV. SQPnP.
• Lu, Hager & Mjolsness (2000), "Fast and Globally Convergent Pose Estimation from Video Images," IEEE TPAMI. The orthogonal-iteration approach.

Related pages: RANSAC · Structure from motion · Bundle adjustment · Stereo · Attitude & rotations · Joint observability