Light-Field Imaging in Microscopy

Light-field microscopy records both spatial and angular information of a sample in a single exposure, enabling reconstruction of its 3D structure. This article follows the historical and conceptual route from integral photography to light-field cameras, light-field microscopy, and Fourier light-field microscopy, and summarizes the imaging principles, sampling strategies, and duality between light-field microscopy and Fourier light-field microscopy.

Conceptual Origins and Evolution of Light-Field Imaging

Integral Photography: Capturing and Reproducing 3D Information

The origin of light-field imaging is usually traced back to Lippmann’s proposal of integral photography. In his 1908 article in Comptes rendus of the French Academy of Sciences¹, Lippmann compared the photography of his time with real human visual perception and proposed the idea of integral photography. He envisioned a photosensitive medium with a double-sided microstructure: the front surface consisted of many tiny spherical elements, while the photosensitive layer on the back was placed along the corresponding curved focal surfaces. In this way, rays arriving from different directions would be refracted by the spherical elements on front surface and focused onto different positions on the photosensitive layer. Each spherical element therefore recorded not merely local intensity, but the local distribution of directions.

When the exposed plate is illuminated from the back, different positions on the photosensitive layer re-emit light corresponding to the originally recorded directions. Because optical paths are reversible, these rays pass again through the same spherical elements and can form a 3D visual impression with parallax in the observer’s eyes. This corresponds to Lippmann’s central idea: the complete 3D information is distributed across a large number of tiny elements, or Tota in minimis existit natura.

This lensless recording strategy also has clear limitations. Each spherical element collects light independently, leading to low light throughput and limited image resolution. The system also has difficulty controlling magnification, field of view (FOV), and focus distance, while nearby objects no longer satisfy the approximation of parallel incidence and therefore produce defocus. From today’s light-field perspective, the key point of integral photography is not the specific use of spherical microstructures, but the fact that it was the first to treat “light arriving at one spatial location from different directions” as a recordable information object.

Light-Field Camera: Recording the Full Four-Dimensional Light Field

Later, as Adelson, Wang, and others formalized the concept of the plenoptic function, such imaging problems began to be understood from the perspective of sampling a set of light rays. To describe the propagation state of light in space, one may use the spatial coordinate $(x,y,z)$ of a point together with the propagation angles $(\theta, \phi)$ of the ray passing through that point. A simplified equivalent description introduces two parallel planes: a ray is described by its intersection coordinate $(x,y)$ with the first plane and its intersection coordinate $(u,v)$ with the second plane.

Building on this concept, the work of Levoy, Hanrahan, Ren Ng, and others further advanced the development of modern handheld light-field cameras. Such cameras use a high-quality main lens to form an intermediate image plane and then use a microlens array (MLA) to capture angular information. In this architecture, the position index of the MLA corresponds to spatial sampling $(x,y)$ on the image plane of the main lens, while the positions of different pixels behind each microlens correspond to different incident directions $(u,v)$. The camera therefore records not a 2D intensity image, but a discretely sampled 4D light field $L(x,y,u,v)$. This microlens-based light-field camera forms a spatial/angular sampling duality with camera-array-based light-field cameras, a relation analogous to the duality between light-field microscopy and Fourier light-field microscopy discussed later.

Light-Field Microscopy: From Geometric Models to Wave-Optics Models

In 2006, Levoy and colleagues introduced the idea of the light-field camera into microscopy[1] and proposed light-field microscopy (LFM). Here, we refer to the focal plane of the microscope, namely the plane focused by the objective, as the native object plane (NOP), and the corresponding image plane as the native image plane (NIP). The basic design of LFM places an MLA at the NIP and places the sensor one focal length behind the MLA, thereby recording both spatial and angular information of the sample in a single exposure. In this configuration, the $N_{angle}\times N_{angle}$ pixel array behind each microlens records angular information of the rays entering that microlens, whereas different microlenses record spatial positions of the light field.

Early LFM mainly relied on geometric-optics models for digital refocusing: rays recorded on the sensor can be reprojected to different depths to synthesize images at different focal planes. At microscopic scales, however, geometric optics provides only an approximation. A more accurate model of LFM imaging requires wave optics. For monochromatic light with wavelength $\lambda$, the propagation direction can be described by the wave vector $\mathbf{k}=(k_x,k_y,k_z)$. Since its magnitude is fixed at $\lvert\mathbf{k}\rvert=\frac{2\pi}{\lambda}$, only two of the three directional components are independent, and the plenoptic function can still retain a similar form, $L(x,y,k_x,k_y)$.

Based on this observation, LFM still preserves a sampling structure similar to that of macroscopic light-field cameras: the MLA position corresponds to spatial sampling of the sample, while the pixel distribution behind each microlens corresponds to angular sampling of different incident directions. In their 2013 paper, Broxton et al. used scalar diffraction theory from wave optics to model the LFM imaging process[2], calculating the point spread function (PSF). The detailed derivation is omitted here. Combined with deconvolution algorithms, the raw light-field image can be used to reconstruct volumetric data and significantly improve LFM reconstruction quality. It is worth noting that the PSF of LFM is space-variant: its 2D response on the camera plane depends on the 3D object-space position of the point source. It is therefore a 5D PSF, which makes both forward and backward projection computationally expensive during deconvolution.

Nevertheless, LFM still faces several fundamental limitations when used for 3D microscopic imaging.

1. Focal-plane artifacts: A point source near the focal plane forms a diffraction-limited Airy pattern near the corresponding NIP. Its intensity distribution is highly localized and falls into only a few microlens apertures. This makes the reconstruction near the focal plane vulnerable to the sampling pitch of the MLA, causing resolution degradation or artifacts.
2. Trade-off between spatial and angular resolution: In LFM, the $N_{angle}\times N_{angle}$ pixels behind each microlens record angular information of the local region. To obtain sufficient axial parallax information, $N_{angle}$ is usually chosen between $11\sim 19$. Under a fixed sensor pixel budget, this greatly reduces the sampling rate of spatial information. Therefore, balancing spatial and angular resolution has always been a key issue in LFM design.

From Integral Microscopy to Fourier Light-Field Microscopy

To bypass the above limitations of conventional LFM, researchers in optics and neuroscience developed new light-field microscopic methods around 2016. One category became known as Fourier integral microscopy (FIMic)[3], [5], while another was represented by extended field-of-view light-field microscopy (XLFM)[4]. The optical configuration of FIMic was already very close to what later became commonly known as Fourier light-field microscopy, whereas XLFM was mainly motivated by large-FOV, high-speed observations in neuroscience and introduced two groups of microlenses with different focal lengths into the MLA to expand the depth of field (DOF).

In 2019, Guo et al. unified this family of light-field imaging paradigms, derived a complete wave-optics imaging model, and quantitatively analyzed the system performance[6]. They systematically formulated this imaging method as Fourier light-field microscopy (FLFM).

In a standard FLFM optical system, the MLA is placed at the Fourier plane of the objective to segment the pupil into sub-apertures, while the sensor is placed one focal length behind the MLA to sample spatial information. Unlike in LFM, where each microlens records the angular distribution within a local field of view, each microlens in FLFM corresponds to a global angular view. In other words, FLFM does not sacrifice a local spatial region behind each microlens to obtain angular information; instead, each microlens corresponds to a sub-aperture view and preserves a complete spatial image from that view. This sampling strategy brings an immediate advantage: FLFM can maintain good axial localization capability with fewer angular samples, thereby reserving more sensor pixels for spatial sampling and improving spatial resolution under a limited pixel budget.

Another important advantage of FLFM comes from the structure of its imaging model. Because the sensor directly samples the spatial domain, the system PSF can be regarded as spatially invariant under common model assumptions and within the effective field of view. Its imaging process can therefore be simplified as a 3D convolution. This property greatly reduces the computational complexity of 3D reconstruction and deconvolution, making FLFM particularly attractive for large-volume and high-speed reconstruction tasks.

Duality Between the Two Light-Field Microscopy Strategies

However, FLFM should not be viewed simply as an “improved LFM”. Rather, it represents a different strategy for sampling the same light field. Just as camera-array-based and MLA-based light-field cameras form a spatial/angular sampling duality, LFM and FLFM also exhibit several forms of duality.

The first level of duality lies in MLA sampling. LFM places the MLA at the NIP, where each microlens collects angular information from a local light field. FLFM places the MLA at the Fourier plane, where each microlens divides the aperture into angular channels and forms a spatial image behind each channel. From the perspective of phase space, which jointly represents spatial position and propagation direction, LFM indexes microlenses by image-plane position. Each microlens records relatively dense angular information within a local region, and individual microlenses are often small. It is therefore a strategy with relatively sparse spatial sampling and dense angular sampling, where $N_{angle}$ is typically $11 \sim 19$. FLFM, in contrast, indexes microlenses by Fourier-plane position. Each microlens corresponds to a global angular channel and forms a spatial image within that channel; the number of microlenses is relatively small. It is therefore a strategy with relatively sparse angular sampling and dense spatial sampling, where $N_{angle}$ is typically $3 \sim 7$. The two are not simply better or worse than each other, but represent different ways of slicing and allocating the same 4D light field under a limited sensor pixel budget.

The second level of duality appears in the location of artifacts. The typical artifacts of LFM concentrate near $z=0$: when the object lies near the NOP, the image has already focused at the MLA plane, and the microlens array discretizes local spatial information too early, leading to resolution degradation near the focal plane. In contrast, degradation in FLFM tends to appear when the object is far away from the NOP; in a limiting sense, it can be understood as degradation near $z\rightarrow\infty$. One may interpret this as follows: LFM cuts spatial information too early at the image plane, while FLFM cuts angular information too early at the Fourier plane. Their failure modes therefore fall in mutually dual depth regions.

Thus, the relationship between LFM and FLFM is not merely one between an old method and a newer replacement. More precisely, they are two strategies for sampling the same four-dimensional light field at two mutually dual planes: the image plane and the Fourier plane. LFM is advantageous in preserving richer angular parallax, while FLFM allocates more pixel budget to spatial sampling and benefits from a spatially invariant imaging model that is more suitable for fast deconvolution. Understanding this duality helps place LFM, FIMic, XLFM, and FLFM along a coherent technical route: from the “local directional recording” of integral photography, to the “four-dimensional light-field sampling” of light-field cameras, and finally to the choice of sampling strategies around the image plane and Fourier plane in microscopy.

Reference

1. Levoy, M., Ng, R., Adams, A., Footer, M. & Horowitz, M. Light Field Microscopy. in (2006).
2. Broxton, M. et al. Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Express 21, 25418 (2013).
3. A. Llavador, J. Sola-Pikabea, G. Saavedra, B. Javidi, and M. Martínez-Corral, “Resolution improvements in integral microscopy with Fourier plane recording,” Opt. Express 24, 20792 (2016).
4. L. Cong, Z. Wang, Y. Chai, W. Hang, C. Shang, W. Yang, L. Bai, J. Du, K. Wang, and Q. Wen, “Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (Danio rerio),” eLife 6, e28158 (2017).
5. G. Scrofani, J. Sola-Pikabea, A. Llavador, E. Sanchez-Ortiga, J. C. Barreiro, G. Saavedra, J. Garcia-Sucerquia, and M. Martínez-Corral, “FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples,” Biomed. Opt. Express 9, 335 (2018).
6. Guo, C., Liu, W., Hua, X., Li, H. & Jia, S. Fourier light-field microscopy. Opt Express 27, 25573–25594 (2019).
7. Stefanoiu, A., Scrofani, G., Saavedra, G., Martínez-Corral, M. & Lasser, T. What about computational super-resolution in fluorescence Fourier light field microscopy? Opt. Express 28, 16554 (2020).

Acknowledgement

Polished by LLM. Proofread by Zhouyu Jin.