The research team from MIT's Computer Science and Artificial Intelligence Lab is a bit mind-blowing. The team has created a way to record video and make individual objects in that video interactive and capable of being manipulated by the viewer independently of the recorded video. MIT News explains how the team has been able to create this interactive dynamic video (IDV):
To simulate the objects, the team analyzed video clips to find "vibration modes" at different frequencies that each represent distinct ways that an object can move. By identifying these modes' shapes, the researchers can begin to predict how these objects will move in new situations.
Davis [the lead researcher] used IDV on videos of a variety of objects, including a bridge, a jungle gym, and a ukelele. With a few mouse-clicks, he showed that he can push and pull the image, bending and moving it in different directions. He even demonstrated how he can make his own hand appear to telekinetically control the leaves of a bush.
The team behind IDV says there are many real-world uses for the technique, including in gaming and animation, but also in engineering whereby engineers could use IDV of a bridge to show how it would respond to high winds or an earthquake. Check out the video below at the 1:00 mark to see the IDV process in action.