FYI.

This story is over 5 years old.

Tech

A New Algorithm Reveals the Hidden World of Imperceptible Motion

Everything is moving all of the time, and usually we miss it.
Image: Durand et al

The world is full of motion that goes unfelt and unseen—the minute trembles and vibrations that become swallowed up by larger and more general perturbations. They aren't wiped away by these larger forces, but are instead obscured from view. Which means that they can theoretically be retrieved and made observable. It's difficult business, however, and an excellent signal processing problem.

Enter William Freeman and Frédo Durand, MIT professors of engineering and computer science, respectively. For years now they've been hot on a trail of a video processing algorithm that can separate out the tiniest motions of structures from very large motions. They've crafted algorithms that can make the human pulse visible and can recover the content of human speech from vibrations carried through otherwise soundproof glass, but these have all had the limitation of assuming that every motion was small and going from there rather than differentiating between big and general motions and little and specific motions. Small movements that are part of big ones are difficult to isolate.

Advertisement

The duo's latest algorithm, presented earlier this month at the Computer Vision and Pattern Recognition conference in Boston in a paper titled "Video Magnification in Presence of Large Motions," offers just that. Here's the rough outline, via the aforementioned (open-access) paper:

The world is full of small temporal variations that are hard to see with naked eyes. Variations in skin color occur as blood circulates, structures sway imperceptibly in the wind [11], and human heads wobble with each heart beat. While usually too small to notice, such variations can be magnified computationally to reveal a fascinating and meaningful world of small motions. Current video magnification approaches assume that the objects of interest have very small motion. However, many interesting deformations occur within or because of larger motion. For example, our skin deforms subtly when we make large body motion. A toll gate that closes exhibits tiny vibrations in addition to the large rotational motion. And microsaccades are often combined with large-scale eye movements.

Furthermore, videos or objects might be shot by handheld cameras and may not be perfectly still, and a standard video magnification technique will amplify handshake in addition to the motion of interest. When applied to videos that contain large motions, current magnification techniques result in large artifacts such as haloes or ripples, and the small motion remains hard to see because it is overshadowed by the then magnified large motion and its artifacts

Freeman and Durand's method is called DVMAG, which is short for Dynamic Video Motion Magnification. It allows users to select a particular region of interest within some video and then decomposes the scene into different layers. "Our layer-based magnification is based on decomposing an image into a foreground, background through an alpha matte," the paper explains. "We magnify each layer and generate a magnified sequence through matte inversion. We use texture synthesis to fill in image holes revealed by the magnified motion. Finally, we de-warp the magnified sequence back to the original space-time co-ordinates." You should probably just watch the video above.