Your Brain Is Not a Video Camera

Announced by a push of air and a screeching sound reverberating through the tunnel, the 6 train arrives and passes. Through headphones you are listening to "Day In The Life" by The Beatles. The 6 accelerates to your left, fwoosh, fwoosh, fwoosh, and then it is gone, taking the air and sound with it.  As it recedes into the tunnel, you shift your gaze to a structural beam on on the platform. It wobbles slightly side to side.

After the beam appears motionless again, through your headphones you hear the start of the famous middle-eight in "Day In The Life", for which John Lennon, Paul McCartney and George Martin instructed each member of an orchestra to start at the lowest note of their instrument and ascend to the highest note. As you listen, you return your gaze forward towards the empty track. When the middle-eight finishes, you happen to turn your gaze back again to the platform beam. This time it wobbles momentarily downwards.

You think to yourself, I am not drunk, and even if I was, platform beams do not wobble. But how then do you explain what you saw?

"Our perception corresponds to a model of the world, not the world as it is," said Pascal Wallisch, a neuroscientist at New York University. "This model is constructed by the brain. Put differently, the eye is not a video camera and the brain does not just passively record its input.”

Wallisch and colleagues at the University of Chicago observed this based on experimental findings they published earlier this year in their paper “Music Can Elicit A Vision Motion Aftereffect.” The beam wobbling is a motion aftereffect (MAE).  Right after you look at a large moving scene, what you look at will move in the opposite direction. It's a subtle effect that affects our perception of the world around us, and according to research, it can be elicited by music.

Aristotle is credited with the first recording of an aftereffect when, after looking at a waterfall, he wrote, “Even when the external object of perception has departed, the impressions it has made persist." More than 1000 years later, Jan Evangelista Purkyně made the earliest known recorded observation of an opposite direction aftereffect after watching a cavalry parade, popular in those days. More recently, researchers in the 20th century have published aftereffects elicited by subtle and even implied motion.

But Wallisch and colleagues are the first to report a disruption in people’s judgment of visual motion from listening to music. The experiment was simple.  Participants listened through headphones to ascending and descending piano scales for sixty seconds. Then they had to judge the direction of moving dots on a computer screen.  The authors found that subjects who listened to scales that moved “up” the piano perceived the dots to move down. Those who listened to scales that moved “down” perceived the dots to move up.

When I suggested that the same result might happen if they listened to the middle-eight of “Day In The Life," Wallisch agreed, though noted that they did not test this possibility experimentally. The experiment they did do highlights the fundamental truth that we are creatures collecting information about our physical and emotional surroundings to reconstruct reality. Yet on occasion we miss the mark.

“This reconstruction is inherently imperfect, as the brain loses information up front; for example, the device that captures signals from the visual world, the retina, is a 2-dimensional,” said Wallisch. “But the spatial world, as well as our visual perception is 3-dimensional. One spatial dimension is lost right off the bat and has to be inferred from other information. To enhance the reconstruction, the brain takes advantage of cues in other modalities. Put differently, not only is the brain not a video camera, the audio feed is not even independent of the visual feed. It is inherently tangled up. Already on the frontend. In most cases, this is beneficial to disambiguate the world. We are simply taking advantage of this in this experiment.”

Neuroscientist Mark Changizi, who is not affiliated with the paper, said that the finding is consistent with the state of the art.

“A number of experiments, some going back nearly fifty years, have come to realize that visual cortex has auditory selective neurons in it, for example, brain uses whatever cues it can to figure out what the scene is,” said Changizi.

The main draw of the paper, he added, is that it explores what melody means to the brain. Most likely, effective melody loosely maps human movement—for example, the arrival or departure of footsteps—and the brain is confusing melody for true motion.

“If the brain is really coding it and thinking of it as movement, we should see an aftereffect," he said.

The finding not only informs us about the distance between our perceptions and reality. It also invites us to think about how we shape music and how music shapes us. To learn more, Wallisch invited me to the Center for Neural Science at NYU.

The center was once a trio of buildings housing garment factories, one block away from the Triangle Shirtwaist Factory, where an infamous 1911 fire killed 146 garment workers in the deadliest industrial disaster in New York City. Today the three buildings are fused together into a veritable research complex where fashionable young scientists and some economists from around the country and around the world hunt data and find answers and drink beer and smoke cigarettes. Many play in bands too.

Wallisch, who I first met at the Society for Neuroscience in New Orleans, is friendly and accommodating. If you could be the guinea pig for only one research study in your life, make it his. Once he gave each subject a copy of their brainscan to make Christmas cards. Peers agree that such bonhomie is rare in a competition as ruthless as science. He even indulged a tour of the facilities and discussed how researchers spread out across at least 15 laboratories compete for time at data collection machines like eye movement trackers and fMRI scanners, which, said the operator not affiliated with NYU, cost about $450 per hour during prime usage hours.

Science lies in thoughtful methodology, not improvised explanations. As such, procedures in neuroscience experiments—the imaginative designs, the grizzly politics, the seizure-inducing power of machines—are best understood when thoroughly expounded upon by researchers. So we cut short the tour to get afternoon tea at department headquarters, where Wallisch expanded on the possible implications of motion aftereffects elicited by music.

“This ties in with a more general notion that what you do with your brain matters,” Wallisch said. “This is generally true for the brain, even in extreme cases. For instance, the brain won’t let brain regions lie fallow. So people who are congenitally blind reappropriate the visual cortex, where visual processing happens, for other functions, like hearing and language processing.”

This means that the music you put into your brain matters because it influences how you perceive reality. Mozart will not make you better a mathematician any more than Slipknot will make you a murderer, but listening regularly to either will influence the model of reality your brain is constructing for you.

What this could mean is that listening to classical music might improve your ability to suss patterns and the like, while listening to heavy metal will make you more aggressive than your baseline. Alongside difficulty making friends, your perception will be drastically different still if you never listen to music.

“In the time of Mozart, people didn't listen to music then unless you could go to a performance,” Changizi says.  “Some pieces may have been so famous that fiddlers on the street could be performing them, but the way they spread through culture, it was a once a year thing.”

Information dissemination creates possibilities for influence.  At an early age music is exposed to you; at a later age, you expose yourself to music. This suggests that your perception could be heavily influenced by other people, assuming for example you only let other people choose your music. And because minds inhabiting brains reconstruct reality, we probably underestimate how much other people influence us, or stated otherwise, how much other models of reality influence our personal model of reality.

That said, raising yourself or your kids on Mozart instead of pop music doesn't guarantee development of enhanced and unique cognition. If that is your goal, listen to as much jazz as possible.

“Much of pop music has similar chord structure, and pop music is much more like Mozart than jazz, which blows away the principles and constraints of classical music,” said Changizi.

But why does your brain confuse melody with motion? The motion aftereffect elicited by confusion between your brain’s audio and visual feeds results from your use of vertical motion to describe sonic movement.  When you hear ascending scales, you say the music is going “up” the keyboard and “down” the keyboard when scales descend. But on a keyboard, hands playing scales move left and right, not up and down.

Composers have used this understanding to their advantage. Symphony No. 6 by Franz Joseph Haydn conveys sunrise through an ascending melody. Little Richard played the piano to imitate trains moving on tracks. Quechua Indians in Andean South America play the charrango to illustrate rain. Yet interpreting music in terms of movement is not true for all cultures, noted Wallisch.

“We believe that tribes who haven’t been expose to this wouldn’t show this effect,” he said.

“Some cultures have different associations with pitch,” Changizi added. “The main reason for pitch changes is Doppler shifts. Pitch modulations are not due to height in an individual field.  What pitch tells you is what direction an object is moving in.”

Changizi wrote a book, Harnessed, partly about music and your brain. He argues that music has culturally evolved to mimic human movement, because movement is, for evolutionary reasons, what your brain is good at listening to.

“You could have ‘Twinkle, Twinkle Little Star,' as the footsteps of the mover,” he said. “You can do a Mozart style of ‘Twinkle, Twinkle Little Star’ versus a hard rock version where the difference is in the personality of the gait you're describing with the tone of the instrument. You can make it sound like a soldier, or make it more fanciful, or put more bad attitude in it. In each case, you can describe the music on the basis of gait and other sounds made when people move.”

Music—through your headphones, in concert halls, from buskers—brings you to humanity. Interpreting music as movement helps you develop emotional connections.

“Music has evolved to be evocative storytelling of a human moving and doing emotional shit around you,” he said.  “It’s evolved to tap into people’s emotions, and the way to tap into people’s emotions is to sound or look like people who are like you or people you have sex with.”

Go back now to the subway.  You are on the 4 train.  Look up at the subway map. Each stop takes you somewhere different in the city of eight million models of reality.

“What you do—the experience you expose yourself to—changes your brain and effectively who you are,” said Wallisch.