How Visual Stereotyping Warps Our Literal View of the World

It's a natural, safe assumption: I see you as you see me. Our eyes might impose differing degrees of limitation on how we see—or how well we see—but when it comes to what we see, we can claim some basic level of objectivity.

This is, of course, bogus. Vision isn't simply a matter of taking optical information and projecting it wholesale into our awareness to be reasoned about. No, vision is itself interpretation, and the brain is tasked with continually making inferences and automatically, algorithmically filling in the gaps for the things that our eyes may not have properly seen but we know to exist. This usually works out.

But it also means that vision can be tricked. And this trickery goes much deeper than trippy optical illusions. According to a paper published today in Nature Neuroscience, a team of researchers from NYU led by psychology professor Jonathan Freeman have demonstrated how our deeply held stereotypes about others function to systematically alter the brain's visual representation of a face.

In other words, before we even have a chance to see a face, our brains have already determined to some extent what that face looks like.

"Individuals extract a wealth of information from another's face, including social categories such as sex, race, and emotion," Freeman and co. write. "The traditional view is that each social category dimension is represented independently, which is sensible as social category dimensions are indeed all distinct and orthogonal [not interdependent] in reality."

The traditional view now seems to be wrong, however. Newer computational models show how our brains manage to make these different categories interdependent, or "entangled," through stereotyping.

As the current paper explains, these models view visual perception of social categories as an exchange between lower-level face processing and higher-order social cognition, including stereotypes. These different components/levels constrain each other until a compromise of sorts is reached.

"As facial cues activate categories, categories activate related stereotypes; the stereotypes, in turn, constrain initial category activation itself, while categories and stereotypes recurrently pass activation back and forth," the authors note. "Accordingly, the extraction of one category dimension (e.g., sex) will activate stereotypical associations that in turn bias the perception of other category dimensions."

The example given in the paper is of the categories "black" and "male" and the shared stereotypical association with "aggressive." The idea is that "black" and "male" become linked via "aggressive" all the way down to the level of visual perception. That is, a neutral black male face will appear angry at a basic sensory level, as determined by stereotyping.

The researchers explored this idea using fMRI scans and mouse-tracking analyses. The basic idea of the latter is that it's possible to reveal biases by asking participants different questions via computer terminal and then checking their conscious responses against the very slight motions of their hands controlling a computer mouse. Using this technique, they found that participants often categorized black men as angry even though they were shown for-sure not angry faces. Likewise, women were often perceived as "happy" even though the faces show were objectively, for-sure not happy. Meanwhile, asian faces were perceived as female, regardless of gender, and black faces were perceived as male, regardless of gender.

Experiments with fMRI imaging backed this up. In observing activity in the fusiform cortex, the region of the brain responsible for visual processing, the researchers found similar activation patterns for subjects show, for example, black male faces and objectively angry faces. The extent of this correlation matched the correlation seen in the mouse-tracking version of the experiment.

A second study using a larger group of participants backed up the findings of first by eliminating some possible alternative explanations and adding additional controls. We see our stereotypes, and we form stereotypes based on what we see.

"In short, our findings suggest that the fundamental structure of social categories when perceiving a face can become warped by social-conceptual knowledge that binds ostensibly unrelated categories together," the paper concludes. "Thus, although stereotyping has long been considered a consequence of initially perceiving others via categories, our stereotypes can affect even our initial categorizations."