​This ‘Pokémon Go’ Hack Lets Pokémon Interact with the Real World

Everyone's already crazy about Pokémon Go, the augmented reality phone game that superimposes virtual characters on top of the real world environment. But what if the Pokémon could go one step further and interact with the authentic environment? Could Zubat flutter through leaves in a tree?

Abe Davis, a researcher at MIT CSAIL (Computer Science and Artificial Intelligence Laboratory), developed a technique called "interactive dynamic video" that allows people to "reach in" and interact with objects in the video frame. With the new technology, outlined in a report with researcher Fredo Durand and doctoral student Justin G. Chen, the character Charmander jumping in a real life puddle would create a splash on screen.

Interactive Dynamic Video works by capturing and using the tiny object vibrations—almost undetectable movements—in order to create video simulations that users can interact with on their devices. And it makes the line between virtual reality and actual reality even more blurry.

"This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space," Davis said. "By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos."

While 3D modeling has been touted as the most common way to simulate motion in objects, it's also expensive and sometimes inaccessible. But Davis's method shows us that just five seconds of video is enough to achieve simulated motion.

The CSAIL team analyzed clips in order to find "vibration modes" at different frequencies, illustrating the specific ways an object can move, and then predicting how the object would react in a new situation—like if Pikachu were to hop on top of a swing and it were to sway a bit.

"If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond to existing forces and assume that it will respond in a consistent way to new ones," Davis said.

This new method could also be useful in other situations where the real world meets virtual reality. Filmmakers could use it to produce new visual effects, or engineers could use it to to simulate how a building or bridge would respond to an earthquake or other event compromising their architecture.

"The ability to put real world objects into virtual models is valuable for not just the obvious entertainment applications," Davis said, "but also for being able to test the stress in a safe virtual environment, in a way that doesn't harm the real-word counterpart."

This interactive technology has yet to reach most of our fingertips. But the characters on your screen are closer to your real world than you think.