Image and video are easy to capture and provide an accurate depiction of our immediate surroundings. However, they lack the interactivity and physical realism that is possible with virtual environments. The Inventors show that some of the structural information available to 3D simulations can be observed in videos and used to synthesize physically plausible dynamics directly in image space without prior knowledge of scene geometry.