There’s nothing cute about stereotypes, but as a species, we seem to struggle to live without them.
In a clever new study led by Jordan Carpenter, who is now a postdoctoral fellow at Duke, a University of Pennsylvania team of social psychologists and computer scientists figured out a way to test just how accurate our stereotypes about language use might be, using a huge collection of real tweets and a form of artificial intelligence called “natural language processing.”
Word clouds show the words in tweets that raters mistakenly attributed to Female authors (left) or Males (right). The larger the word appears, the more often the raters were fooled by it. Word color indicates the frequency of the word; gray is least frequent, then blue, and dark red is the most frequent. <url> means they used a link in their tweet.
Starting with a data set that included the 140-character bon mots of more than 67,000 Twitter users, they figured out the actual characteristics of 3,000 of the authors. Then they sorted the authors into piles using four criteria – male v. female; liberal v. conservative; younger v. older; and education (no college degree, college degree, advanced degree).
A random set of 100 tweets by each author over 12 months was loaded into the crowd-sourcing website Amazon Mechanical Turk. Intertubes users were then invited to come in and judge what they perceived about the author one characteristic at a time, like age, gender, or education, for 2 cents per rating. Some folks just did one set, others tried to make a day’s wage.
The raters were best at guessing politics, age and gender. “Everybody was better than chance,” Carpenter said. When guessing at education, however, they were worse than chance.
Jordan Carpenter is a newly-arrived Duke postdoc working with Walter Sinnott-Armstrong in philosophy and brain science.
“When they saw the word S*** [this is a family blog folks, work with us here] they most often thought the author didn’t have a college degree. But where they went wrong was they overestimated the importance of that word,” Carpenter said. Raters seemed to believe that a highly-educated person would never tweet the S-word or the F-word. Unfortunately, not true! “But it is a road to people thinking you’re not a Ph.D.,” Carpenter wisely counsels.
The raters were 75 percent correct on gender, by assuming women would be tweeting words like Love, Cute, Baby and My, interestingly enough. But they got tricked most often by assuming women would not be talking about News, Research or Ebola or that the guys would not be posting Love, Life or Wonderful.
Female authors were slightly more likely to be liberal in this sample of tweets, but not as much as the raters assumed. Conservatism was viewed by raters as a male trait. Again, generally true, but not as much as the raters believed.
Youthful authors were correctly perceived to be more likely to namedrop a @friend, or say Me and Like and a few variations on the F-bomb, but they could throw the raters for a loop by using Community, Our and Original.
And therein lies the social psychology takeaway from all this: “An accurate stereotype should be one with accurate social judgments of people,” but clearly every stereotype breaks down at some point, leading to “mistaken social judgement,” Carpenter said. Just how much stereotypes should be used or respected is a hot area of discussion within the field right now, he said.
The other value of the paper is that it developed an entirely new way to apply the tools of Big Data analysis to a social psychology question without having to invite a bunch of undergraduates into the lab with the lure of a Starbucks gift card. Using tweets stripped of their avatars or any other identifier ensured that the study was testing what people thought of just the words, nothing else, Carpenter said.
The paper is “Real Men Don’t Say “Cute”: Using Automatic Language Analysis To Isolate Inaccurate Aspects Of Stereotypes.” You can see the paper in Social Psychology and Personality Science, if you have a university IP address and your library subscribes to Sage journals. Otherwise, here’s a press release from the journal. (DOI: 10.1177/1948550616671998 )
Post by Karl Leif Bates