Engineers Prove There's No Such Thing as a Silent Film

​The visual world has sound information even if it doesn't have actual sound sound. Just as we, on looking at a photograph of some common setting—a busy subway station or neighborhood coffeeshop—might imagine what it sounds like with some accuracy, so too can an algorithm. Thanks to separate engineering teams at the Catholic University of America and MIT, that algorithm now exists.

The result is a "simple and fast optical technique" that's capable of extracting audio information from silent high-speed video. "The basic principle is that the sound waves can stimulate objects encountered in the traveling path to vibrate," the CUA researchers, led by engineering professor Zhaoyang Wang, write in the current Optical Engineering. "The vibrations, although usually with small amplitudes, can be detected by using an image matching process."

Information about these vibrations is gleaned from an image by observing the relative motions of points on different surfaces. Some more pliable materials, like a sheet of paper, might deliver fairly high-quality audio information. And as these vibrations are directly related to sound waves, the CUA engineers explain, the original audio information can be brought back to life.

So, the technique, which the CUA team has verified experimentally, is really quite a bit more than just imagining the acoustic environment, it's literally observing it.

There are some clear limitations, such as background noise. "This technique has a very high sensitivity and it can easily detect sub-micron-scale to nano-scale vibrations," Wang told me, "so vibrations from other sources may affect the results. [While] many of them can be easily removed, but some can be hard to separate. Air density change[s] over long distances may also affect the captured videos, which may bring large errors to the results."

The hope, he says, is that the technique might someday be used to passively detect conversations inside of a building at a large distance. All it takes is a window.

In August, researchers from MIT achieved something similar—albeit way less precise, according to Wang—peeling away acoustic information from visual sources such as a glass of water, aluminum foil, and the leaves of a potted plant. In one experiment, they were able to recover intelligible speech from the vibrations of a potato chip bag recorded from 15 feet away and on the other side of a pane of soundproof glass.

Abe Davis, the lead MIT researcher, has bigger ideas than snooping, forecasting a "new kind of imaging." "We're recovering sounds from objects," Davis said. "That gives us a lot of information about the sound that's going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways."

So, we're not just revealing the hidden acoustic information of an environment, but entirely new information about the composition of the environment itself.