The Challenge of Reproducing Depth Perception on a Light Field Display

In the October 2021 issue of The SMPTE Motion Imaging Journal, a featured technical paper describes all the ways natural human vision creates depth perception by exploiting information called depth cues. The brain captures information through the eyes, and analyzes these cues to understand a complex 3D scene. In real life, we view objects as light field content, a collection of light rays flowing in every direction through every point in space. A perfect light field display should recreate natural depth cues that are precise, simultaneous and consistent with each other, allowing the viewer’s brain to correctly interpret the scene.

These are huge constraints for a 3D display, however, and currently none can fully render all these cues simultaneously with the expected resolution. So the challenge, for both current displays and future technology, is to balance the tradeoffs and provide the information to simulate most of the depth cues, including:

Visual Depth Cues, which are provided by the objects being viewed

• The Binocular Cue, the depth cue provided by stereoscopic 3D displays, is based on retinal disparity where each eye captures the same scene from a slightly different angle.
• Monocular Cues include Static Cues, the classic pictorial cues available on a standard 2D display, including shading and lighting, linear and aerial perspectives, texture gradients, occlusions, and absolute, relative and familiar sizes. Motion-based Cues like motion parallax (as when a close-up object appears to move faster than an identical object that’s further away) and dynamic occlusion (when one object passes in front of another) are exploited when one part of the scene is moving relative to another.

Oculomotor Depth Cues, which are related to physiological aspects of vision and eye movements

• Accommodation is created by a change in the shape of the eye’s lens which changes its focal length. This maintains a clear focus on an object as its distance varies, and is equivalent to the auto-focus system on a camera.
• Convergence, a change in the horizontal rotation angle of both eyes, keeps the viewed object in focus at the same position on both retinas. Accommodation and Convergence are consistent when viewing real objects, but on a display, objects are always at screen position. This may cause a clear conflict between the two depth cues which may be uncomfortable for the viewer.
• Myosis involves a change in the size of the pupil which serves as a variable aperture for the eye, increasing the depth of focus as the pupil constricts.

To fully recreate our viewing in real life, a display should consistently satisfy all these depth cues at the same time, with a high-resolution picture that ensures the eye cannot perceive the pixel structure of the display. To accomplish this, a light field display would need to provide hundreds or thousands of views of the same scene simultaneously – a huge challenge requiring massively parallel rendering architectures, and affecting the entire volumetric video workflow. Though no current display technology can create a light field display fulfilling even a few percent of these requirements, a reduced version of all the constraints may be feasible in the coming years with developing technologies.

Multi-projector Displays combine a large number of projectors to provide motion parallax effects. Integral Imaging technology combines a high-resolution display with an array of lenslets placed on top of the display, each tiny lens covering a set of pixels. A Directional Backlight system generates rays of light to create the light field, using an LCD panel as a focal plane shutter. Micro-Display Tiling could greatly improve display resolution, with micro-LEDs delivering high brightness, high contrast, low response time and the key parameter of reduced pixel size. While these approaches all show promise, a technological breakthrough is needed to address the tradeoff between complexity, capacity and efficiency, in order to create a commercial solution.

Get a deeper dive, exploring the technological requirements of creating depth perception with light field displays. Read the article  in The SMPTE Motion Imaging Journal.