What does it mean to reconstruct the world?

Eleven modules, in order. Module 01 is today’s lecture; everything after it is scaffolding we will fill in as the course progresses. The animations marked below are just the shape of the course — the content of each lecture will be developed in full, exactly like this one.

We’ll develop the foundations today, including our first real equations for reconstructing 3D from photographs — and explaining why 4D is significantly harder.

Imagine a photograph of a street. The image contains buildings, roads, windows, and possibly vehicles. You can recognize these objects immediately.

But imagine I ask you something different. Can you tell me the exact distance from the camera to every building? Can you reconstruct the building surfaces that aren’t visible? Could you move a virtual camera around the scene and see what the street looks like from a completely different viewpoint?

A regular image cannot directly provide all that information.

It records a two-dimensional projection of a three-dimensional world. This creates the fundamental problem of 3D reconstruction.

Mathematically, let the real three-dimensional scene be \(\mathcal S\), the camera parameters be \(C\), and the image formation process be \(\mathcal R\). The camera produces an image:

• \(I\) is the observed photograph.
• \(\mathcal S\) is the 3D scene, including its geometry and appearance.
• \(C\) describes the camera position, orientation, and optical properties.
• \(\mathcal R\) is the process of converting the 3D scene into pixels, including visibility, lighting, and projection.

This is the forward problem: going from a known world to an image. Reconstruction reverses that process:

The symbol \(\widehat{\mathcal S}\) means our estimate of the actual scene. And here is the first fundamental insight:

Why? Because many different 3D scenes can produce exactly the same photograph. We’ll quantify that claim precisely in §4.

Consider a ball on a table.

In 3D, a point has three spatial coordinates:

Here \(X,Y,Z\) specify its position along the three spatial axes. In 4D reconstruction, the scene evolves with time \(t\):

For a ball moving at constant velocity, we could write:

where \(\mathbf X_0\) is its initial position, \(\mathbf v\) is its 3D velocity vector, and \(t\) is elapsed time. For example: \(\mathbf X_0=(1,2,3)\), \(\mathbf v=(0.5,0,0)\). Then at \(t=2\) seconds:

The ball has moved one spatial unit along the X-axis. For a deforming object, such as a waving cloth, the problem is more complex: thousands of points may change position differently over time. A useful abstract representation is:

Here \(\mathcal G(t)\) describes the geometry at time \(t\), and \(\mathcal A(t)\) describes its appearance — which might also change with time. The resulting sequence of observations is:

Notice that the camera \(C_t\) can move as well. That introduces one of the hardest challenges in 4D reconstruction:

One subtle point: 4D reconstruction does not always output a single, persistent, physically accurate mesh. Some methods output depth maps and motion fields; others represent a time-varying radiance field, Gaussian particles, or deforming surfaces. These are related but different outputs, and the differences will matter enormously when we read research papers (§10).

This is probably the most important mathematical foundation of the entire field. If you understand this section properly, the equations in NeRF, Gaussian splatting, DUSt3R, VGGT, and many 4D reconstruction models will become much easier to understand.

3.1 · The pinhole camera model

Imagine a camera looking at a ball. Light from the ball travels toward the camera and is projected onto an image plane.

Let a point in camera coordinates be \(P=(X,Y,Z)\). The camera looks along the positive \(Z\) direction in our convention. Suppose the image plane is located at distance \(f\) from the camera center. Using similar triangles, \(\dfrac{x}{f}=\dfrac{X}{Z}\) — and rearranging:

These are the fundamental perspective projection equations.

Notice something important:

As depth increases, the apparent size or displacement in the image becomes smaller. This explains why distant buildings look small even when they’re physically enormous.

3.2 · Moving from image-plane coordinates to pixels

Actual digital images don’t use coordinates centered at the camera’s optical axis. They use pixel coordinates. A common simplified formulation is:

Here \(u,v\) are pixel coordinates, \(f_x,f_y\) are focal lengths measured in pixels, and \(c_x,c_y\) are the principal point coordinates — where the optical axis intersects the image plane.

For example, suppose \(f_x=800\), \(X=0.5\) m, \(Z=4\) m, \(c_x=640\). Then \(u=800\cdot\tfrac{0.5}{4}+640 = 740\). The point appears at horizontal pixel 740.

Now double the object’s depth and horizontal coordinate: \(X=1\) m, \(Z=8\) m. The result is still \(u=800\cdot\tfrac{1}{8}+640=740\).

Suppose I observe a pixel at horizontal location \(u\). From the projection equation (10):

For every possible positive value of \(Z\), there is a corresponding value of \(X\). For example, all of these are valid for the pixel \(u=740\):

This means one pixel corresponds not to a unique 3D point, but to an entire ray extending from the camera into the world. In vector form:

Here \(\widetilde{\mathbf u}\) is the pixel in homogeneous coordinates and \(K\) is the camera intrinsic matrix, which we will derive in full in §6.

The scale ambiguity

Even for a calibrated camera, projecting a point and projecting the same point scaled by any positive constant can produce the same pixel:

where \(\pi\) is perspective projection. This is one form of scale ambiguity. There are other ambiguities too: hidden surfaces, material properties, lighting, and even object shape can be uncertain.

Here is where the reconstruction problem becomes much more constrained. Imagine photographing the same stationary object from two different positions. Because the camera moves, the object appears in different pixel locations in the two images. This displacement is called disparity in rectified stereo vision.

5.1 · Stereo triangulation

Suppose we have two calibrated, horizontally aligned cameras. Their distance apart is the baseline \(B\), and both have focal length \(f\). For a 3D point at depth \(Z\), the disparity between its two pixel positions is:

For rectified stereo, we obtain:

This equation is incredibly important. It tells us that two views provide geometric information about depth through parallax.

Numerical example. Suppose \(f=800\) px, \(B=0.2\) m, \(d=40\) px. Then \(Z=\dfrac{800\cdot 0.2}{40}=4\) m. We have recovered the object’s depth using image correspondence and known camera geometry — and unlike ordinary single-image estimation, this depth is constrained by an observed displacement between two camera views.

But there are still failure modes:

Consider \(Z=\dfrac{fB}{d}\). The sensitivity of depth to disparity is:

For a small disparity error \(\Delta d\):

As disparity becomes small, depth estimation becomes increasingly sensitive to correspondence errors. This is one reason distant objects, low-parallax camera motion, and repetitive surfaces are challenging. Even with multiple images, reconstruction is not automatically reliable.

Now let’s introduce the equation that appears throughout research in 3D computer vision:

This equation combines the camera’s internal geometry, position, orientation, and perspective projection. Let’s unpack it.

6.1 · Camera intrinsics

The camera intrinsic matrix is usually written:

This common form assumes zero pixel skew; more general formulations can include skew. The intrinsic parameters tell us how the camera maps points expressed in camera coordinates onto pixel coordinates — exactly equations (10)–(11) in matrix clothing.

6.2 · Camera extrinsics

The extrinsic parameters are the rotation \(R\) and translation \(t\). A 3D point in the world coordinate system is transformed into camera coordinates using:

• \(\mathbf X_w\) is the point in world coordinates.
• \(\mathbf X_c\) is the same point relative to the camera.
• \(R\in SO(3)\) is a \(3\times 3\) rotation matrix.
• \(t\in\mathbb R^3\) is a translation vector.

We’re using the world-to-camera convention. Under this convention, the camera center in world coordinates is \(-R^\top t\).

6.3 · Perspective division

Once we transform a point into camera coordinates \(\mathbf X_c=(X_c,Y_c,Z_c)\), we project it with equations (10)–(11). The unknown scalar \(\lambda\) in the homogeneous camera equation (20) accounts for the fact that homogeneous vectors represent image coordinates only up to scale; in this simple projection model it is related to camera-space depth.

Before neural networks became dominant, methods such as Structure-from-Motion (SfM) and Multi-View Stereo (MVS) were already capable of reconstructing large real-world scenes. A foundational reference is Structure-from-Motion Revisited (Schönberger & Frahm, CVPR 2016); its implementation is the well-known COLMAP system.

The classical pipeline can be summarized as:

The especially interesting step is bundle adjustment.

7.1 · Bundle adjustment: reconstruction as optimization

Suppose we have several cameras observing the same 3D points. Each observed pixel is associated with an unknown 3D point. We want to find camera poses and 3D positions such that projecting the reconstructed points into each camera reproduces the observed pixels. One standard objective is:

Inside the objective, the expression

is the reprojection error: how far our predicted pixel location is from the observed one. Bundle adjustment tries to minimize these errors jointly across all cameras and all reconstructed points.

This distinction is central to the evolution from COLMAP toward DUSt3R and VGGT — we’ll chart it in §10.

Now return to our moving ball. Suppose it appears to move to the right between two video frames. There are at least three possibilities:

A single observed displacement doesn’t tell us which explanation is correct. This is why dynamic reconstruction is substantially harder than static reconstruction: the ambiguity becomes a counterfactual ambiguity — every frame pair supports multiple physical stories.

8.1 · Optical flow: motion in image coordinates

Suppose a point appears at pixel \(\mathbf u_t=(u_t,v_t)\) at time \(t\), and at \(\mathbf u_{t+1}\) in the next frame. Its optical flow is:

For example, if a point moves from pixel \((100,120)\) to \((110,125)\), then \(\mathbf w_t=(10,5)\): ten pixels horizontally, five vertically. But optical flow is 2D motion, not necessarily actual 3D motion. If the camera moves while the world remains stationary, the observed optical flow can still be nonzero — Bench C of the lab let you reproduce this.

8.2 · Scene flow: motion in 3D

Scene flow describes the movement of scene points in three-dimensional space. For a point \(\mathbf X\), its displacement during one time interval is:

Here \(\Delta t\) is the interval between observations. If we divide displacement by elapsed time, we obtain an average velocity: \(\mathbf v_t(\mathbf X)=\mathbf F_t(\mathbf X)/\Delta t\).

The observed pixel displacement is related to 3D motion by:

The logic, step by step:

1. Start with the original point \(\mathbf X\).
2. Add its 3D displacement \(\mathbf F_t\).
3. Transform that new point into the next camera’s coordinates (\(R_{t+1}, t_{t+1}\)).
4. Project it onto the next image with the next intrinsics.
5. Subtract the originally projected pixel position.

The result is the observed 2D displacement — for readability, the equation suppresses explicit dependence on \(\mathbf X\), and both projection terms use the appropriate timestamped camera.

Equation (27) reveals why 4D reconstruction is difficult: the observed displacement is influenced simultaneously by depth, camera pose, and object motion. If any of these are incorrect, the others may compensate. For example, a system might estimate that the camera has moved too far and compensate by assigning artificial motion to static scene points.

That gives you the intuition behind the problem tackled by MoRe (Motion-aware Feed-forward 4D Reconstruction Transformer): explicitly disentangling dynamic motion from static geometry with an attention-forcing strategy — an approach we will read closely in Module 09.

So far we’ve talked about reconstructing points. But how do we represent an entire scene inside a computer? Several representations define the field’s vocabulary:

The distinction between explicit and implicit representations is useful. A mesh explicitly stores surface geometry as vertices and faces. A neural signed-distance field, by contrast, represents a surface as the zero level set of a function:

Here \(s(\mathbf X)\) is signed distance and \(\mathcal M\) is the recovered surface. A NeRF doesn’t directly store mesh triangles either — instead, it learns a continuous function that can be queried at spatial locations.

9.1 · Neural Radiance Fields (NeRF)

NeRF (Mildenhall et al., ECCV 2020) is one of the foundational developments behind modern neural scene reconstruction. Its scene function can be written as:

• \(F_\theta\) is a neural network with learned parameters \(\theta\).
• \(\mathbf X\) is a 3D spatial location.
• \(\mathbf d\) is a viewing direction.
• \(\sigma\) is volume density.
• \(\mathbf c\) is RGB radiance.

We can render an image by tracing camera rays and accumulating predicted colors according to the density along each ray. In continuous form, the rendered color along a ray \(\mathbf r(s)\) is:

where:

The quantity \(T(s)\) is transmittance: approximately, how much light travels along the ray without being absorbed or occluded under this emission–absorption model. This formula is at the heart of volume rendering. It makes the rendering operation differentiable, enabling the scene network to be optimized directly against photographs — NeRF demonstrated that optimizing such a representation from posed images produces impressive novel-view synthesis.

9.2 · From NeRF to dynamic NeRF

The original NeRF assumes a static scene. What if the world changes over time? A natural extension is:

However, directly learning a time-varying scene is difficult, because the same physical point occupies different spatial positions over time. A useful approach is to introduce a canonical representation: the model learns a reference configuration plus a deformation field that maps coordinates at different timestamps into that reference:

This idea appears in D-NeRF: Neural Radiance Fields for Dynamic Scenes (CVPR 2021), which learns a canonical scene together with time-dependent deformation. The key advantage: geometry and appearance are shared across timestamps rather than independently learned for each frame.

9.3 · From NeRF to Gaussian splatting

The 2023 paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering (Kerbl et al., ACM TOG) introduces a different scene representation. Instead of repeatedly evaluating a neural network along camera rays, we represent the scene as many 3D Gaussian primitives. A Gaussian has a center \(\boldsymbol\mu\) and a covariance matrix \(\Sigma\) describing its shape and orientation:

In 3DGS, Gaussians are optimized and efficiently projected onto the image plane for rasterization-style rendering — achieving high-quality, real-time novel-view synthesis. For dynamic scenes, Gaussian positions (and sometimes their rotations, scales, and appearances) evolve over time. For example:

Here \(\boldsymbol\mu_i^0\) is the reference center of Gaussian \(i\) and \(\Delta\boldsymbol\mu_i(t)\) is its predicted displacement. This is the intuition behind several dynamic Gaussian approaches, including 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (CVPR 2024), which combines 3D Gaussians, learned spatiotemporal features, and predicted deformations rather than independently optimizing a complete set of Gaussians for every timestamp.

Here is a research-oriented timeline of papers worth understanding. We will eventually read the methodological sections of these papers, not just their abstracts.

The broader evolution is especially interesting:

This is a conceptual history, not a claim that one family has completely replaced the others. Optimization-based reconstruction and feed-forward models remain complementary.

At this stage I want you to notice three scientific problems — rather than immediately choosing a new architecture.

First · reconstruction is an underdetermined inverse problem

We observe images, but many 3D worlds may explain those images (§4 was a proof of concept). When there is insufficient evidence, reconstruction systems must use geometric assumptions or learned priors — creating a tension between what the observations support and what a model believes is plausible. The recent Building Rome from a Single Image line of work generates complete meshes containing both visible surfaces and unobserved regions using learned 3D priors and scene-scale generation — squarely inside this tension.

Second · motion and camera geometry are coupled

In a dynamic video, we often don’t know the true geometry, motion, or camera trajectory. An error in one can affect the others (equation (27) made this precise). MoRe and Any4D address important aspects of this problem — but their existence doesn’t mean dynamic-scene reconstruction has become perfectly identifiable or error-free.

Third · visual realism and geometric accuracy are different objectives

Suppose a model reconstructs a moving person’s hand. The generated video may appear smooth and photorealistic, yet the recovered hand trajectory could be physically inaccurate. For entertainment or view synthesis that might be acceptable; for robot planning, manipulation, or physical interaction it could be a serious problem.