Real-Time Photorealistic 3D Holographic Display using Deep Neural Rendering

Technology

The invention, “tensor holography”, enables the creation of real-time, photorealistic 3d holography using deep learning. The inventors introduced the first large-scale 3D hologram dataset with 4000 RGB-D images and paired 3D CGHs. The RGB-D scene consists of various shapes, colors, shading, texture, and a new physical simulation algorithm is introduced to handle occlusion. Using this photorealistic training dataset, the inventors developed the first deep learning based rendering system that uses a convolution neural network to transform RGB-D input to a 3D hologram. The CNN performs this task in tens of milliseconds on a consumer-grade graphics processing unit (GPU). The CNN is very memory-efficient (<1mb) and can run interactively (<1s) on mobile devices (i.e. an iPhone 11 Pro).

Problem Addressed

Virtual reality headsets have gained tremendous popularity. However, one common problem tends to occur— the headsets can make users feel sick. Nausea can be experienced because users’ eyes are being tricked into seeing a 3D scene, when, in fact, they are staring at a 2D fixed-distance display. One possible solution for a better 3D viewing experience would be to have these headsets display 3D computer generated holograms (CGHs). However, creating these 3D CGHs has been no easy feat. CGH algorithms based on physical simulation is very time-consuming, and can take minutes to complete on current high-end graphics processing units (GPUs). This prevents real-time application of CGHs for aforementioned applications, particularly for virtual and augmented reality. Existing CGH algorithms also don’t handle occlusion well and are hard to achieve photorealistic results.

Advantages

• Photorealistic 3D holograms that can be created in milliseconds
• The first convolutional neural network created for 3D holography
• The first large-scale 3D hologram dataset that consists of custom-made 4,000 pairs of RGB-D images and matched CGHs
• Prevent nausea in users who utilize devices with 3D holographic display

Video

MIT Computational Fabrication Group 

Publications

Liang Shi, Beichen Li, Changil Kim, et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature. 2021 Mar 10. doi: 10.1038/s41586-020-03152-0