Real-Time Photorealistic 3D Holographic Display using Deep Neural Rendering
The tensor holography technology has a wide range of applications, especially in the consumer electronics industry. Within this industry, the applications for this technology are most readily applied to virtual and augmented reality. This technology could have further applications within the fields of 3D printing, microscopy, and healthcare.
Researchers
-
real-time photorealistic 3d holography with deep neural networks
United States of America | Published application
Figures
Tensor Holography Process
Technology
The invention, “tensor holography”, enables the creation of real-time, photorealistic 3d holography using deep learning. The inventors introduced the first large-scale 3D hologram dataset with 4000 RGB-D images and paired 3D CGHs. The RGB-D scene consists of various shapes, colors, shading, texture, and a new physical simulation algorithm is introduced to handle occlusion. Using this photorealistic training dataset, the inventors developed the first deep learning based rendering system that uses a convolution neural network to transform RGB-D input to a 3D hologram. The CNN performs this task in tens of milliseconds on a consumer-grade graphics processing unit (GPU). The CNN is very memory-efficient (<1mb) and can run interactively (<1s) on mobile devices (i.e. an iPhone 11 Pro).
Problem Addressed
Virtual reality headsets have gained tremendous popularity. However, one common problem tends to occur— the headsets can make users feel sick. Nausea can be experienced because users’ eyes are being tricked into seeing a 3D scene, when, in fact, they are staring at a 2D fixed-distance display. One possible solution for a better 3D viewing experience would be to have these headsets display 3D computer generated holograms (CGHs). However, creating these 3D CGHs has been no easy feat. CGH algorithms based on physical simulation is very time-consuming, and can take minutes to complete on current high-end graphics processing units (GPUs). This prevents real-time application of CGHs for aforementioned applications, particularly for virtual and augmented reality. Existing CGH algorithms also don’t handle occlusion well and are hard to achieve photorealistic results.
Advantages
- Photorealistic 3D holograms that can be created in milliseconds
- The first convolutional neural network created for 3D holography
- The first large-scale 3D hologram dataset that consists of custom-made 4,000 pairs of RGB-D images and matched CGHs
- Prevent nausea in users who utilize devices with 3D holographic display
Video
MIT Computational Fabrication Group
Publications
Liang Shi, Beichen Li, Changil Kim, et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature. 2021 Mar 10. doi: 10.1038/s41586-020-03152-0
License this technology
Interested in this technology? Connect with our experienced licensing team to initiate the process.
Sign up for technology updates
Sign up now to receive the latest updates on cutting-edge technologies and innovations.