"Modal Images" (Image-Space Modal Analysis of Deformations for Video Synthesis)

Technology

The Inventors apply techniques from 3D physical simulation and structural analysis to the task of video synthesis. Small deformations of most solid objects can be well-approximated with linear modal analysis. The Inventors show that by applying this analysis to small deformations captured in video they can extract projections of an object’s deformation modes and use these projections as a basis for the image-space simulation of that object’s dynamics.

The algorithm works by extracting a set of observed mode shapes from small deformations due to unknown excitations forces in an input video and using these as a modal basis for simulation of an object’s response to user defined forces. Local image space deformations are measured using phase variations in a complex steerable pyramid based on the input video. Displacement measures from the object’s rest state are evaluated for each video frame, then are filtered and denoised. The temporal FFT of the displacement signals are calculated and an output modal analysis is conducted on the resulting displacement spectra to select and normalize for the spectrum of forces that caused motion in the input video. The response of an object to force is its displacement and momentum, which can be represented as a superposition of recovered modes. Users can either pull or apply an impulse on the object using a computer interface, upon which a simulation is conducted based on the object’s recovered modes that make up its resultant displacement field.

Problem Addressed

Image and video are easy to capture and provide an accurate depiction of our immediate surroundings. However, they lack the interactivity and physical realism that is possible with virtual environments. The Inventors show that some of the structural information available to 3D simulations can be observed in videos and used to synthesize physically plausible dynamics directly in image space without prior knowledge of scene geometry. 

Advantages

• Minimal user input required to synthesize physically plausible object motion simulations

Publications

Davis, A., Chen, J. G., & Durand, F. "Image-Space Modal Bases for Plausible Manipulation of Objects in Video." ACM Transactions on Graphics 34, no. 6 (2015): Article 239. doi: 10.1145/2816795.2818095.