Real-Time Photorealistic 3D Holographic Display using Deep Neural Rendering


The tensor holography technology has a wide range of applications, especially in the consumer electronics industry. Within this industry, the applications for this technology are most readily applied to virtual and augmented reality. This technology could have further applications within the fields of 3D printing, microscopy, and healthcare.  

Problem Addressed

Virtual reality headsets have gained tremendous popularity. However, one common problem tends to occur— the headsets can make users feel sick. Nausea can be experienced because users’ eyes are being tricked into seeing a 3D scene, when, in fact, they are staring at a 2D fixed-distance display. One possible solution for a better 3D viewing experience would be to have these headsets display 3D computer generated holograms (CGHs). However, creating these 3D CGHs has been no easy feat. CGH algorithms based on physical simulation is very time-consuming, and can take minutes to complete on current high-end graphics processing units (GPUs). This prevents real-time application of CGHs for aforementioned applications, particularly for virtual and augmented reality. Existing CGH algorithms also don’t handle occlusion well and are hard to achieve photorealistic results.



The invention, “tensor holography”, enables the creation of real-time, photorealistic 3d holography using deep learning. The inventors introduced the first large-scale 3D hologram dataset with 4000 RGB-D images and paired 3D CGHs. The RGB-D scene consists of various shapes, colors, shading, texture, and a new physical simulation algorithm is introduced to handle occlusion. Using this photorealistic training dataset, the inventors developed the first deep learning based rendering system that uses a convolution neural network to transform RGB-D input to a 3D hologram. The CNN performs this task in tens of milliseconds on a consumer-grade graphics processing unit (GPU). The CNN is very memory-efficient (<1mb) and can run interactively (<1s) on mobile devices (i.e. an iPhone 11 Pro).


  • Photorealistic 3D holograms that can be created in milliseconds
  • The first convolutional neural network created for 3D holography
  • The first large-scale 3D hologram dataset that consists of custom-made 4,000 pairs of RGB-D images and matched CGHs
  • Prevent nausea in users who utilize devices with 3D holographic display

Intellectual Property

IP Type: US Provisional Application

IP Title: Real-time photorealistic 3D holographic display using deep neural rendering

Video by MIT Computational Fabrication Group