3D Gaussian Splatting

NeRF showed that a scene can be stored as a neural field and rendered by integrating along rays — accurate, and far too slow to be interactive. Gaussian Splatting reaches the same image quality at hundreds of frames per second by changing the representation rather than the optimiser: store the scene as millions of explicit anisotropic 3-D Gaussians, project each one to a screen-space ellipse in closed form, and composite them with a sorted rasteriser. No ray marching, no network evaluation per sample. This page works through the representation, the exact projection algebra, the full render pipeline from camera matrices to pixels, and how the whole thing is trained.

Symbols used on this page are defined in the Nomenclature → Gaussian splatting.

The camera conventions used throughout — intrinsics, extrinsics, the projection Jacobian — are derived on the camera model page; the point cloud that initialises training comes from structure from motion.

1 · The representation

1.1 What one Gaussian stores

The scene is a set of primitives $\{\mathcal G_i\}$, each defined by a 3-D Gaussian with an opacity and a view-dependent colour. The spatial part is

$$ G(\mathbf x) = \exp\!\left(-\tfrac12\left(\mathbf x - \boldsymbol\mu\right)^\top \boldsymbol\Sigma^{-1}\left(\mathbf x - \boldsymbol\mu\right)\right), $$

with no normalising constant — the amplitude is carried separately by the opacity $\alpha$, so the two are not redundant.

At the standard SH degree $\ell = 3$ that is

$$ \underbrace{3}_{\boldsymbol\mu} + \underbrace{3}_{\mathbf s} + \underbrace{4}_{\mathbf q} + \underbrace{1}_{\alpha} + \underbrace{3 \times 16}_{\text{SH}} = \mathbf{59} \ \text{floats per Gaussian} . $$

1.2 Model size

Two things are worth noting about this table. The parameter counts are comparable to a large neural network, but they are not weights — there is no network, and each parameter has a direct geometric meaning. And training memory is roughly 3× the model, because Adam keeps first and second moments for every parameter; that factor, not the model itself, is usually what exhausts a GPU.

2 · The covariance

2.1 Why it is factored, not stored

A covariance matrix must be symmetric positive semi-definite. Optimising its six free entries directly would let gradient descent wander out of that set and produce a meaningless Gaussian. So $\boldsymbol\Sigma$ is never a parameter — it is constructed from a scale and a rotation:

$$ \boxed{\;\boldsymbol\Sigma = \mathbf R\,\mathbf S\,\mathbf S^\top \mathbf R^\top\;} \qquad \mathbf S = \operatorname{diag}(s_x, s_y, s_z), \quad \mathbf R = \mathbf R(\mathbf q) . $$

This is an eigendecomposition written forward: $\mathbf S\mathbf S^\top$ holds the variances along the principal axes and $\mathbf R$ orients them. The form is PSD for any values of $\mathbf s$ and $\mathbf q$, which is exactly the property needed — the optimiser can take any step it likes and the result is still a valid ellipsoid. The geometric reading is direct: the Gaussian is an ellipsoid with semi-axes $s_x, s_y, s_z$ rotated by $\mathbf R$.

The quaternion-to-matrix map is the standard one:

$$ \mathbf R(\mathbf q) = \begin{bmatrix} 1 - 2(q_j^2 + q_k^2) & 2(q_i q_j - q_r q_k) & 2(q_i q_k + q_r q_j) \\ 2(q_i q_j + q_r q_k) & 1 - 2(q_i^2 + q_k^2) & 2(q_j q_k - q_r q_i) \\ 2(q_i q_k - q_r q_j) & 2(q_j q_k + q_r q_i) & 1 - 2(q_i^2 + q_j^2) \end{bmatrix} . $$

Anisotropy is the whole point. If the Gaussians were isotropic spheres, representing a flat wall would need an enormous number of them. An anisotropic Gaussian can flatten into a disc and cover a planar surface with one primitive, which is why 3DGS needs millions rather than billions. The cost is that extremely elongated Gaussians produce visible streaks (§7.2), so the representation's strength and its main artefact have the same source.

2.2 Projecting to screen space — the EWA step

Rendering needs the Gaussian's footprint in pixels. Perspective projection is nonlinear, so a projected Gaussian is not exactly Gaussian — but take its first-order (affine) approximation about the Gaussian's centre and the covariance transforms by the usual rule. With $\mathbf W$ the view rotation and $\mathbf J$ the Jacobian of the projection evaluated at the centre:

$$ \boxed{\;\boldsymbol\Sigma' = \mathbf J\,\mathbf W\,\boldsymbol\Sigma\, \mathbf W^\top \mathbf J^\top\;} $$

where, for a pinhole camera with the point at camera coordinates $(x_c, y_c, z_c)$,

$$ \mathbf J = \frac{\partial \pi}{\partial \mathbf x_c} = \begin{bmatrix} \dfrac{f_x}{z_c} & 0 & -\dfrac{f_x x_c}{z_c^2} \\[3mm] 0 & \dfrac{f_y}{z_c} & -\dfrac{f_y y_c}{z_c^2} \end{bmatrix} \in \mathbb{R}^{2\times3} . $$

This is precisely the projection Jacobian from the camera model page §7 — the same matrix that appears in PnP and bundle adjustment. Here it is doing a different job: pushing a covariance forward rather than propagating a pose correction. The technique is EWA splatting (elliptical weighted average), from Zwicker et al.'s volume-splatting work, which predates 3DGS by two decades.

The result $\boldsymbol\Sigma'$ is a $2\times2$ covariance in pixel units. Dropping the third row and column of the $3\times3$ product is what reduces the ellipsoid to the screen ellipse, and the off-axis terms in $\mathbf J$ are what make a splat at the image periphery shear — real perspective distortion, captured to first order.

2.3 The screen-space low-pass filter

A Gaussian whose projected extent is below one pixel cannot be sampled correctly and produces aliasing and flicker. The reference implementation adds a small dilation to the screen covariance before use:

$$ \boldsymbol\Sigma'_{\text{used}} = \boldsymbol\Sigma' + h\,\mathbf I_2, \qquad h \approx 0.3\ \text{px}^2 , $$

which guarantees a minimum footprint of roughly half a pixel. It is a crude but effective anti-aliasing measure, and the dashed ellipse in the simulation above shows the undilated covariance for comparison — at sub-pixel scales the dilation term dominates entirely. This approximation is also the specific thing Mip-Splatting improves on (§8), because a fixed dilation in screen space is not the correct filter when the sampling rate changes with distance.

2.4 Evaluating the 2-D Gaussian at a pixel

With the projected centre $\boldsymbol\mu'$ and covariance $\boldsymbol\Sigma'$, the contribution at pixel $\mathbf p$ is

$$ G'(\mathbf p) = \exp\!\left(-\tfrac12\,\mathbf d^\top \boldsymbol\Sigma'^{-1}\mathbf d\right), \qquad \mathbf d = \mathbf p - \boldsymbol\mu' , $$

and since $\boldsymbol\Sigma'$ is $2\times2$ its inverse is closed form — one determinant and three multiplies, the cheapest possible inner loop:

$$ \boldsymbol\Sigma' = \begin{bmatrix} a & b \\ b & c\end{bmatrix} \ \Longrightarrow\ \boldsymbol\Sigma'^{-1} = \frac{1}{ac - b^2} \begin{bmatrix} c & -b \\ -b & a \end{bmatrix} . $$

The effective opacity at that pixel is then the product of the learned opacity and the spatial falloff:

$$ \alpha_i(\mathbf p) = \alpha_i \cdot G'_i(\mathbf p) . $$

If $ac - b^2 \le 0$ the splat is degenerate — numerically flat — and must be discarded rather than inverted. This check is in every implementation and is one reason extreme anisotropy is dangerous.

3 · Rendering, end to end

3.1 The pipeline

This is the complete path from stored parameters to a pixel, with nothing hidden. Every stage is differentiable, which is what allows the whole thing to be trained by backpropagating image error.

Three design choices in that listing are what make it fast, and all three are about memory access rather than arithmetic:

• Tile binning. Splitting the screen into 16×16 tiles means each GPU block handles a small set of splats that it can load into shared memory once and reuse across 256 pixels.
• One global sort, not per-pixel. Packing the tile index and the depth into a single integer key lets a single radix sort produce per-tile depth-ordered lists simultaneously. Sorting per pixel would be hopeless; sorting per tile is one pass.
• Early termination. Once accumulated transmittance falls below a threshold, the remaining splats cannot change the pixel, so the loop stops. In an opaque scene this skips most of the work.

3.2 Alpha compositing

The colour of a pixel is the front-to-back blend over the splats covering it:

$$ \boxed{\;C = \sum_{i \in \mathcal N} c_i\,\alpha_i \prod_{j=1}^{i-1}\left(1 - \alpha_j\right)\;} \qquad T_i = \prod_{j \lt i}\left(1 - \alpha_j\right) $$

This is the discrete volume-rendering equation — the same one NeRF evaluates — but the samples are primitives rather than points queried along a ray. That is the entire performance difference: NeRF must evaluate a network at dozens of sample positions per ray to find out what is there, while 3DGS already knows what is there and only has to blend it.

3.3 Cost model

$$ \text{work} \;\approx\; \underbrace{N}_{\text{project}} + \underbrace{\sum_i n_{\text{tiles}}(i)}_{\text{duplicate}} + \underbrace{M\log M}_{\text{sort}} + \underbrace{\sum_{\text{pixels}} n_{\text{splats/pixel}}}_{\text{blend}} $$

with $M$ the number of (Gaussian, tile) pairs. The last term normally dominates, and it depends on the depth complexity of the view — how many semi-transparent splats overlap a typical pixel. This is why a scene with many large low-opacity Gaussians renders far more slowly than one with the same count of compact opaque ones, and why pruning is a performance optimisation as much as a quality one. Note that a single very large splat is duplicated into every tile it touches, so one badly-scaled Gaussian can measurably hurt frame time.

4 · View-dependent colour

4.1 Spherical harmonics

A single RGB value per Gaussian would make the scene perfectly Lambertian — no specular highlights, no reflections, no sheen. Real appearance depends on viewing direction, so colour is stored as a function on the sphere, expanded in spherical harmonics:

$$ c(\mathbf d) = \sum_{\ell=0}^{L}\sum_{m=-\ell}^{\ell} k_{\ell m}\, Y_{\ell m}(\mathbf d) , $$

with $\mathbf d$ the unit vector from the Gaussian to the camera and $k_{\ell m}$ the learned coefficients (three sets, one per colour channel). The first few real basis functions, as implemented:

$$ Y_{00} = 0.2821, \qquad Y_{1m} = 0.4886 \cdot \{-y,\; z,\; -x\}, $$ $$ Y_{2m} = \{1.0925\,xy,\ -1.0925\,yz,\ 0.3154(2z^2 - x^2 - y^2),\ -1.0925\,xz,\ 0.5463(x^2 - y^2)\} . $$

The accounting matters: at degree 3, colour is 48 of a Gaussian's 59 floats — 81% of the entire model size. That is why SH degree is the first thing reduced when memory is constrained, and why many compression methods target the SH coefficients specifically.

4.2 Progressive band growth

SH bands are not all optimised from the start. Training begins with degree 0 only and promotes one band every 1000 iterations. The reason is a familiar one: the high-frequency angular terms are perfectly capable of memorising per-view appearance differences — exposure variation, moving shadows, a passing pedestrian — instead of representing genuine specularity. Forcing the model to first explain the scene with a single flat colour per Gaussian means the geometry has to be approximately right before the appearance model gains the capacity to cheat. It is curriculum learning applied to the representation, and removing it measurably increases floater artefacts.

5 · Training

5.1 Initialisation

Training does not start from nothing. The standard pipeline runs COLMAP first, which produces calibrated camera poses and a sparse point cloud, and each SfM point becomes one Gaussian:

The nearest-neighbour scale initialisation is a neat detail: it makes each Gaussian roughly the size of the local point spacing, so the initial scene is approximately covered without large overlaps. Starting from random positions instead largely fails — the camera poses and the SfM cloud are prerequisites, not conveniences, and poor COLMAP output is the most common cause of a bad reconstruction.

5.2 The loss

$$ \mathcal L = (1-\lambda)\,\mathcal L_1 + \lambda\,\mathcal L_{\text{D-SSIM}}, \qquad \lambda = 0.2 $$

Purely photometric — rendered image against ground-truth image, with no geometric supervision at all. The two terms do different jobs: $\mathcal L_1$ on raw pixel differences gives stable, well-scaled gradients everywhere, while D-SSIM compares local means, variances and covariance over a window and so rewards getting structure right rather than average intensity. L1 alone tends to produce blurry reconstructions; SSIM alone is less stable early on. The 80/20 mix is the published default and is rarely worth changing.

Note what is absent: no depth loss, no normal loss, no mask. The geometry that emerges is whatever explains the images, which is why 3DGS produces beautiful renders but poor meshes — nothing in the objective asks the Gaussians to lie on surfaces. §8 covers the variants that add exactly that.

5.3 Per-parameter learning rates

Adam is used, but with a different learning rate for each parameter type — because they have wildly different units and sensitivities. A step that is right for an opacity logit is catastrophic for a world position.

Two of these deserve emphasis. Position LR is scaled by the scene's spatial extent, so the same config works for a tabletop object and a city block — without it, every scene would need retuning. And the higher SH bands get a learning rate 20× smaller than the DC term, which is the same anti-overfitting instinct as the progressive band growth in §4.2.

5.4 Adaptive density control

Gradient descent can move, reshape and fade Gaussians, but it cannot create or delete them — the parameter count is fixed by the model structure. 3DGS therefore adds a non-differentiable heuristic, run every 100 iterations between iterations 500 and 15 000. This is arguably the most important engineering component of the method: without it the reconstruction stays at SfM resolution.

The clone/split distinction is the clever part. A large positional gradient means "this Gaussian is being pulled in inconsistent directions by different views", i.e. one primitive is trying to explain a region that needs more than one. Whether the right response is more coverage or finer detail depends on whether the Gaussian is currently small or large — hence the scale test.

The healthy shape in that simulation is growth followed by saturation. Runaway growth to the cap means the gradient threshold is too permissive; immediate stagnation means it is too strict, and the reconstruction will stay coarse.

5.5 The schedule

5.6 Gradients

Every stage above is differentiable, and the chain runs backwards through the whole pipeline: image error → accumulated colour → per-splat $\alpha_i$ and $c_i$ → $\boldsymbol\Sigma'$ and $\boldsymbol\mu'$ → $\boldsymbol\Sigma$ and $\boldsymbol\mu$ → finally the stored scale, quaternion, opacity and SH coefficients. The derivative through the covariance construction is the only awkward one, since $\partial\boldsymbol\Sigma/\partial\mathbf q$ requires differentiating the quaternion-to-matrix map, and the reference implementation writes it out explicitly in CUDA rather than relying on autograd — this is a hand-written backward pass, which is part of why it is fast.

One practical consequence worth knowing: the positional gradient is accumulated and averaged over views between densification passes, not used instantaneously. A single view's gradient is noisy; the quantity that signals "this region needs more primitives" is the consistent pull across many views.

6 · Compared with NeRF

7 · Failure modes

7.1 Artefacts and what causes them

7.2 Capture guidance

Because the only supervision is photometric, reconstruction quality is bounded by the capture. The rules follow directly from the failure modes above:

• Cover the object from many angles, including elevation. Floaters form in exactly the regions that only a few views constrain.
• Keep exposure and white balance locked. Otherwise the SH coefficients absorb the variation as fake view-dependence — the thing progressive band growth is fighting.
• Avoid motion blur and rolling shutter. COLMAP's poses degrade, and bad poses are unrecoverable downstream.
• Do not photograph a scene that moves. The representation is static; a moving subject becomes floaters or ghosting.
• Check the COLMAP reconstruction before training. If the sparse cloud is wrong, no amount of training fixes it. This is the single highest-value check.
• Mind reflective and transparent surfaces. A mirror is reconstructed as geometry behind the mirror plane, because that is a consistent explanation of the images.

8 · Variants worth knowing

The pattern across this list: almost every variant is fixing one of the specific approximations identified above — the affine projection, the fixed screen-space filter, the centre-depth sort, or the absence of any surface prior. Knowing which approximation a paper targets is usually enough to predict what it improves.

9 · Practical checklist

• Validate the SfM reconstruction first. Poses and the sparse cloud are prerequisites (§5.1).
• Lock camera exposure and white balance during capture.
• Budget VRAM at 3× the model size for Adam's moments (§1.2).
• Keep the per-parameter learning rates, including the scene-extent scaling on position — these are not arbitrary (§5.3).
• Keep progressive SH growth and opacity reset. Both are anti-overfitting mechanisms, and disabling them visibly increases floaters.
• Monitor the Gaussian count. Saturation is healthy; hitting a cap means the densification threshold needs raising.
• Prune by anisotropy as well as opacity if needles appear.
• Reduce SH degree before reducing Gaussian count when compressing — 81% of the size is colour.
• Do not expect a usable mesh from vanilla 3DGS; use 2DGS or SuGaR if geometry is the goal.
• Evaluate on held-out views, and inspect for floaters from novel angles — training-view metrics hide exactly the failure mode the method is prone to.

10 · Further reading

• Kerbl, Kopanas, Leimkühler & Drettakis (2023), "3D Gaussian Splatting for Real-Time Radiance Field Rendering," SIGGRAPH. The original paper; §5 and the supplementary material contain the exact hyperparameters and the analytic gradients.
• Zwicker, Pfister, van Baar & Gross (2001), "EWA Volume Splatting," IEEE Visualization. The projection mathematics of §2.2, two decades earlier.
• Zwicker et al. (2002), "EWA Splatting," IEEE TVCG. The screen-space filtering treatment that §2.3 approximates.
• Mildenhall et al. (2020), "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV. The volume-rendering equation and the baseline for every comparison.
• Barron et al. (2022), "Mip-NeRF 360." The unbounded-scene benchmark 3DGS is usually evaluated against.
• Yu, Chen, Huang, Sattler & Geiger (2024), "Mip-Splatting: Alias-free 3D Gaussian Splatting," CVPR.
• Huang, Yu, Chen, Geiger & Gao (2024), "2D Gaussian Splatting for Geometrically Accurate Radiance Fields," SIGGRAPH.
• Guédon & Lepetit (2024), "SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction," CVPR.
• Lu et al. (2024), "Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering," CVPR.
• Radl et al. (2024), "StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering," SIGGRAPH.
• Schönberger & Frahm (2016), "Structure-from-Motion Revisited" (COLMAP) — the front end 3DGS depends on.
• Ramamoorthi & Hanrahan (2001), "An Efficient Representation for Irradiance Environment Maps." Why low-order SH suffices for smooth appearance.
• The graphdeco-inria/gaussian-splatting reference implementation — the CUDA rasteriser is the clearest statement of the tiling, sorting and backward pass.

Related pages: Camera model & projection · Structure from motion · Bundle adjustment · GraphSLAM & pose graphs · Hyperparameter tuning