NASA Navigation Filter Best Practices

A working guide to the hard-won lessons in NASA/TP⁠-2018⁠-219822, “Navigation Filter Best Practices” by J. Russell Carpenter and Christopher N. D’Souza (with F. Landis Markley and Renato Zanetti). The report distills decades of flight experience — Apollo, Shuttle, ISS, Orion — into concrete recommendations for building a Kalman filter that actually stays consistent in the field. This page summarizes each chapter, keeps the essential math, and adds interactive illustrations of the four ideas engineers get wrong most often: bias modeling, covariance stability, measurement underweighting, and the consider filter.

Symbols used on this page are defined in the Nomenclature → estimation & filtering.

Source document: Navigation Filter Best Practices (PDF). Everything below is a hand-written summary — read the report for the full derivations and flight anecdotes.

1 · The Kalman / Extended Kalman filter

The report frames the linearized (LKF) and extended (EKF) Kalman filter as the baseline. The state and its covariance are propagated, then corrected by each measurement. In discrete form the time update is

$$ \hat{\mathbf x}_k^- = f(\hat{\mathbf x}_{k-1}^+),\qquad \mathbf P_k^- = \boldsymbol\Phi_k\,\mathbf P_{k-1}^+\,\boldsymbol\Phi_k^{\mathsf T} + \mathbf Q_k $$

with state-transition matrix $\boldsymbol\Phi_k=\exp(\mathbf F\,\Delta t)$, $\mathbf F=\partial f/\partial\mathbf x$. The measurement update computes the gain, corrects the state with the innovation, and updates the covariance:

$$ \mathbf K_k = \mathbf P_k^-\mathbf H_k^{\mathsf T}\big(\mathbf H_k\mathbf P_k^-\mathbf H_k^{\mathsf T}+\mathbf R_k\big)^{-1} $$ $$ \hat{\mathbf x}_k^+ = \hat{\mathbf x}_k^- + \mathbf K_k\big(\mathbf z_k - h(\hat{\mathbf x}_k^-)\big),\qquad \mathbf P_k^+ = (\mathbf I-\mathbf K_k\mathbf H_k)\,\mathbf P_k^- $$

2 · The covariance matrix

Chapter 2 is the heart of the report: how to propagate $\mathbf P$, how to keep it valid, and how to measure whether it is telling the truth.

2.1 Meaningful covariance metrics

Raw diagonal variances in Cartesian position are rarely what you care about. The report recommends projecting $\mathbf P$ onto physically meaningful directions — e.g. for an orbit, the variance of the semi-major axis or the flight-path angle. If $a=g(\mathbf x)$ is such a scalar, its variance is the linear map

$$ \sigma_a^2 = \Big(\frac{\partial g}{\partial\mathbf x}\Big)\,\mathbf P\,\Big(\frac{\partial g}{\partial\mathbf x}\Big)^{\mathsf T} $$

Semi-major-axis variance is a superb health check because $a$ is (to first order) invariant under along-track position error but extremely sensitive to velocity error — exactly the direction that degrades an orbit prediction.

2.2 Process noise: SNC, DMC, and Vasicek

A perfect dynamics model needs no $\mathbf Q$; a real one always does. The report catalogs three ways to inject it:

Model	Idea	When
SNC — State Noise Compensation	Add white acceleration noise directly to velocity states.	Unmodeled accelerations that are roughly white over the step.
DMC — Dynamic Model Compensation	Estimate an extra correlated (FOGM) acceleration state that soaks up the mismodeling.	Slowly-varying unmodeled forces (drag, SRP mismodel).
Vasicek	Mean-reverting process with an explicit non-zero mean.	Biases that drift toward a known equilibrium.

The continuous SNC contribution to the discrete $\mathbf Q$ over a step $\Delta t$, for a white acceleration of PSD $q$ driving a position/velocity pair, is the familiar

$$ \mathbf Q_k = q\begin{bmatrix}\tfrac13\Delta t^3 & \tfrac12\Delta t^2\\[2pt]\tfrac12\Delta t^2 & \Delta t\end{bmatrix} $$

The report warns: process noise is a tuning knob, not a fudge factor. Too little and the filter goes optimistic and rejects good data (“smug”); too much and it never converges.

2.3 Keep P symmetric and positive — the Joseph form

The short update $\mathbf P^+=(\mathbf I-\mathbf K\mathbf H)\mathbf P^-$ is algebraically correct but numerically fragile: finite precision breaks symmetry and can drive $\mathbf P$ indefinite, after which the filter diverges silently. The report’s standing recommendation is the Joseph-stabilized form, which is symmetric and positive-definite by construction and valid for any gain (not just the optimal one):

$$ \boxed{\;\mathbf P^+ = (\mathbf I-\mathbf K\mathbf H)\,\mathbf P^-\,(\mathbf I-\mathbf K\mathbf H)^{\mathsf T} + \mathbf K\mathbf R\mathbf K^{\mathsf T}\;} $$

2.4 The consider (Schmidt–Kalman) filter

Some error sources — a survey-tie bias, an ephemeris error, an un-calibrated station delay — you cannot or should not estimate, but you must not ignore. The consider filter partitions the state into solve-for $\mathbf x$ and consider $\mathbf c$ parameters, and folds the consider covariance $\mathbf P_{cc}$ into the innovations without ever updating $\mathbf c$:

$$ \mathbf W = \mathbf H_x\mathbf P_{xx}\mathbf H_x^{\mathsf T} + \mathbf H_c\mathbf P_{cc}\mathbf H_c^{\mathsf T} + \mathbf R $$

The gain uses this inflated $\mathbf W$, so the estimate covariance grows just enough to stay honest about the error the un-estimated parameter injects. Ignore the bias and the filter becomes optimistic; consider it and the error stays inside its band.

3 · Processing measurements

Chapter 3 covers the mechanics of feeding data to the filter:

• Latency & time-tagging. A measurement is valid at the instant of collection, not receipt. Propagate the state to the true measurement epoch (or carry a delayed-state) before updating — a mistagged GPS fix is a classic silent bias.
• Order invariance & scalar processing. If the measurement noise $\mathbf R$ is diagonal (or you first decorrelate it), you can process a vector measurement one scalar component at a time. This avoids a matrix inverse and, done right, gives an identical answer regardless of order.
• Decorrelation. When $\mathbf R$ is not diagonal, factor $\mathbf R=\mathbf L\mathbf L^{\mathsf T}$ and transform $\tilde{\mathbf z}=\mathbf L^{-1}\mathbf z$ so the transformed noise is unit-diagonal.
• Filter cascades are discouraged. Feeding one filter’s output as another’s measurement double-counts information and produces wildly optimistic covariance. Prefer a single, larger filter.
• IMU model replacement. Rather than treating gyro/accel outputs as noisy measurements of motion, the report treats the IMU as the dynamics driver (model-replacement mode): the measured $\Delta\theta,\Delta v$ propagate the state, and the IMU error states (bias, scale factor) are estimated.

4 · Measurement underweighting

A very precise measurement (tiny $\mathbf R$) applied against a large, nonlinearly-mapped covariance makes the linearized update over-correct: the true post-update error is larger than the filter believes, so it becomes optimistic. Lear’s remedy, endorsed by the report, is to underweight the measurement by inflating the predicted innovations covariance with a factor $\beta$:

$$ \mathbf W = \mathbf H\mathbf P^-\mathbf H^{\mathsf T}\,(1+\beta) + \mathbf R,\qquad \mathbf K = \mathbf P^-\mathbf H^{\mathsf T}\mathbf W^{-1} $$

This deliberately takes a smaller step, trading a little convergence speed for staying consistent while the covariance is still large — precisely the situation during initial convergence or after a big maneuver. As $\mathbf P$ shrinks, the nonlinearity matters less and $\beta$ can be reduced.

5 · Bias modeling

Nearly every real sensor has slowly-varying errors. Chapter 5 is a taxonomy of stochastic models, from crudest to most useful:

Model	SDE	Behavior
Random constant	$\dot b = 0$	Fixed unknown; never adapts.
Random ramp	$\ddot b = 0$	Constant unknown rate.
Random walk	$\dot b = w$	Variance grows without bound ($\propto t$).
FOGM (first-order Gauss–Markov)	$\dot b = -\tfrac1\tau b + w$	Mean-reverting; bounded steady-state variance.
SOGM / Vasicek	2nd-order / non-zero mean	Oscillatory or drifts to a known equilibrium.

The FOGM is the workhorse. Its continuous equation and the discrete propagation of its variance are

$$ \dot b(t) = -\frac{1}{\tau}\,b(t) + w(t),\qquad b_{k} = e^{-\Delta t/\tau}\,b_{k-1} + w_k,\qquad \sigma_{ss}^2 = \frac{q\,\tau}{2} $$

where $\tau$ is the correlation time and $q$ the driving PSD. The steady-state variance $\sigma_{ss}^2=q\tau/2$ is finite — that boundedness is why FOGM is preferred over a random walk for anything that physically saturates (thermal bias, drag coefficient, clock drift). The simulation drives all three with the same noise so you can watch the random walk wander off while the FOGM stays inside its envelope.

6 · State representations

• Choose solve-for states deliberately. Every extra state costs observability and can make the filter fragile. Estimate a parameter only if the measurements actually see it.
• Units, scaling, and precision. Mixing kilometers and radians in one covariance produces enormous condition numbers. Scale states so $\mathbf P$ is well-conditioned; use double precision for the covariance.
• Coordinate & time systems. Be explicit and consistent (ECI vs. ECEF, TAI vs. UTC, which epoch/frame the ephemeris uses). Frame confusion is a leading cause of “mystery” biases.
• Absolute vs. relative. For rendezvous, a dual-inertial formulation (both vehicles in inertial states) is numerically cleaner than differencing; the linearized relative motion (Clohessy–Wiltshire / HCW) is useful for analysis but the report favors estimating full states.
• IMU error states. Gyro and accelerometer models carry bias, scale-factor, and misalignment states, and must account for coning (rotation-rectification) and sculling (velocity-rectification) when integrating high-rate data. See the IMU page for the mechanization.

7 · Factorization methods

To guarantee a positive-definite covariance even in single precision, the report recommends propagating a factor of $\mathbf P$ rather than $\mathbf P$ itself. The UDU factorization keeps $\mathbf P=\mathbf U\mathbf D\mathbf U^{\mathsf T}$ with $\mathbf U$ unit-upper-triangular and $\mathbf D$ diagonal (a square-root-free cousin of Cholesky). Rank-one measurement updates are done in-factor with the

• Agee–Turner rank-one update of $\mathbf U\mathbf D\mathbf U^{\mathsf T}$, and the
• Carlson rank-one update — both process a scalar measurement while preserving the factored form, so $\mathbf P$ can never go negative.

A consider covariance can be carried in the same factored framework. The practical payoff: effectively double the dynamic range of the arithmetic, which mattered enormously on flight computers and still guards against subtle round-off on modern ones.

The payoff is visible the moment precision gets tight. Below, the same update is run conventionally and in factored form as the mantissa shrinks: the conventional covariance's smallest eigenvalue eventually crosses zero — the filter is dead — while the factored form stays positive because it never forms $\mathbf P$ at all:

8 · Attitude estimation

Attitude gets its own chapter because it does not live in a vector space. The report recommends estimating a small-angle error about a reference quaternion — the Multiplicative EKF (MEKF) — and resetting the reference after each update so the three-component error stays small and its covariance stays meaningful:

$$ \mathbf q = \delta\mathbf q(\delta\boldsymbol\theta)\otimes\hat{\mathbf q},\qquad \delta\mathbf q \approx \begin{bmatrix}\tfrac12\delta\boldsymbol\theta\\ 1\end{bmatrix} $$

This keeps a minimal $3\times3$ attitude covariance (no redundant fourth component, no singular constraint) and is the standard for spacecraft. Full derivations, plus the other attitude parameterizations (Euler, axis-angle, Rodrigues/CRP/MRP, $SO(3)$ exp/log), are on the Attitude & Rotations page.

9 · Usability

• Measurement editing. Gate each measurement on its normalized innovation (Mahalanobis distance) $\;\gamma = \nu^{\mathsf T}\mathbf W^{-1}\nu\;$ and reject when it exceeds a $\chi^2$ threshold — but log rejections, because a wall of rejects means the filter, not the data, is wrong.
• Reinitialization & restart. Design the filter so it can be cold-started or warm-restarted from a saved state/covariance without hand-tuning.
• Ground-system interplay. Onboard and ground filters must agree on conventions; the report stresses documented interfaces and reproducible configuration.

10 · Smoothing

When you can process data offline, a smoother uses future measurements to improve past estimates. The report describes the two standard forms:

• Fraser–Potter (two-filter) — run a forward filter and an independent backward filter, then combine the two Gaussian estimates by information addition: $\;\mathbf P_s^{-1}=\mathbf P_f^{-1}+\mathbf P_b^{-1},\; \mathbf P_s^{-1}\hat{\mathbf x}_s=\mathbf P_f^{-1}\hat{\mathbf x}_f+\mathbf P_b^{-1}\hat{\mathbf x}_b.$
• Rauch–Tung–Striebel (RTS) — a single backward recursion over the stored forward estimates using the smoother gain $\mathbf C_k=\mathbf P_k^+\boldsymbol\Phi^{\mathsf T}(\mathbf P_{k+1}^-)^{-1}$.

Smoothing is easiest to appreciate against a data gap. The forward filter must coast blindly through it, while the smoother interpolates using measurements from both sides — the improvement is largest exactly where the filter was weakest:

11 · Advanced topics

The final chapter surveys nonlinear alternatives to the EKF — sigma-point (unscented) filters and the additive-divided-difference sigma-point filter (ADDSPF) — which capture nonlinearity to higher order without Jacobians. The report’s guidance: reach for these only when a well-tuned EKF/MEKF genuinely fails the consistency tests, because they cost more and add their own tuning parameters. See the Filtering page for the UKF sigma-point construction and a particle-filter comparison.

Related: Filtering (KF/EKF/UKF/PF) · EKF deep dive · Attitude & MEKF · IMU error models · full report: NASA/TP-2018-219822 (PDF).