Shulou(Shulou.com)06/02 Report--

Produced by big data Digest

Source: nature

Compiled by Zhao Jike, Ma Li, Li Lei, Qian Tianpei

AI facial emotion recognition software is ubiquitous, but psychologists question its underlying hypothesis.

Can we identify emotions by facial expressions?

Smile when you are happy, frown when you are angry, and stare when you are surprised. Although the expression of human emotion seems to be not exactly the same, it is still uniform.

As a result, many AI have also begun to learn how to judge a subject's mood by facial expressions. By "viewing" photos of facial expressions and tagging emotions, AI can quite accurately predict the mood of a face that has never appeared in the training data.

Then, back to the original question: does it make sense to identify emotions through facial expressions? Psychologists have debated this problem for nearly a hundred years.

In 2018, psychologist Rachael Jack and her colleagues recruited 80 people to take an interesting test.

The testers were shown hundreds of faces. Some open their eyes wide, others close their lips. Some people have closed eyes, raised cheeks and open mouths. Faced with each face, they have to answer this simple question: is Ta experiencing the best part of the experience or sudden pain?

The test is officially intended to address a long-standing and highly sensitive question: can facial expressions truthfully convey emotion?

In fact, for decades, researchers have been asking people what emotions they see in other people's faces. The study covered adults and children from different countries, as well as indigenous peoples from remote parts of the world.

In the 1960s and 1970s, American psychologist Paul Ekman Ekmen made an influential observation that people around the world can infer emotional states relatively accurately from facial expressions-which seems to mean that emotional expressions are universal.

These ideas were not challenged at the time, but a new generation of psychologists and cognitive scientists have been re-examining the data and questioning their conclusions. Many researchers believe that the real picture is much more complex than described, and that facial expressions vary greatly between contexts and cultures. For example, Jack's study found that although Westerners and East Asians have similar concepts of how facial pain is expressed, they have different views on the expression of happiness.

Researchers are increasingly divided over the validity of Ekmen's conclusions, but the debate has not stopped companies and governments from accepting his claim that facial expressions are predictions of emotion. For example, in many western legal systems, interpreting the emotions of the accused is part of a fair trial. As US Supreme Court Justice Anthony Kennedy wrote in 1992, this is necessary to "understand the heart and mind of criminals".

Decoding emotions is also at the heart of Ekmen's training program for the Transportation Security Administration (TSA). The project, called SPOT (screening passengers through observation technology), aims to teach TSA staff how to monitor passengers to detect dozens of potentially suspicious signs that can indicate stress, deceit or fear. The program was launched in 2007, but it has been widely criticized by scientists, members of Congress and the American Civil Liberties Union (American Civil Liberties Union) and other organizations as inaccurate and racially biased.

Such concerns also apply to the behaviour of technology companies. Some companies have developed software to assess the suitability of job seekers, detect lies, make ads more attractive, and diagnose everything from dementia to depression. The industry is estimated to be worth tens of billions of dollars. Tech giants including Microsoft (MSFT), IBM (MSFT) and Amazon (AMZN), as well as more professional companies, such as affect tiva in Boston, Massachusetts, and NeuroData Lab in Miami, Florida, offer algorithms designed to recognize a person's emotions from the face.

With researchers still debating whether people can truly and effectively express or perceive emotions, many in the industry believe it is too early for computers to express emotions automatically-especially if the technology could have a damaging impact. The AI Now Institute, a research center at New York University, even called for a ban on the use of emotion recognition technology in sensitive situations such as recruitment or law enforcement.

Even for people, facial expressions are extremely difficult to interpret, says Alex Martinez, who studies facial expressions at Ohio State University in Columbus. With that in mind, he said, "we should be extra cautious" about the trend towards automation.

Human expressions may be "empty".

The human face has 43 muscles that can be stretched, lifted and twisted into dozens of expressions. Despite the wide range of human activities, scientists have long believed that certain expressions convey specific emotions.

Darwin was one of the people who promoted this view. His book the Origin of species, published in 1859, is the result of hard field work and a master of observation. His second most influential work is the expression of emotion between people and Animals (1872).

Darwin noticed that primate facial expressions also seemed to have human emotional expressions, such as disgust or fear, and believed that these expressions must have some adaptive function. For example, pursing lips, wrinkling nose and narrowing eyes-an expression associated with disgust-may be designed to protect individuals from harmful pathogens. Only when social behavior begins to develop will these facial expressions have more communicative effects.

There are a lot of posturing expressions in Darwin's emotional monographs, such as these themes try to imitate sadness.

Ekmen's first cross-cultural field research in the 1960s supported this hypothesis. He tested the expression and perception of six major emotions around the world-happiness, sadness, anger, fear, surprise and disgust, even in a remote part of New Guinea.

Ekmen told Nature that he chose the six expressions for practical reasons. Some emotions, such as shame or guilt, cannot be explicitly quantified, he says. "the six emotions I focus on do have expressions, which means they can be studied."

Ekmen says these early studies prove the universality expected by Darwin's theory of evolution. Later studies have also supported the view that some facial expressions may have adaptive advantages.

"people have long thought that facial expression is a necessary action," says Lisa Feldman Barrett, a psychologist who studies emotions at Northeastern University University in Boston. In other words, our faces cannot hide our emotions. An obvious problem with this assumption is that people can fake emotions and experience them without using their faces. Researchers in Ekmen's team acknowledge that there may be considerable differences in the "gold standard" for each emotional expression.

But more and more researchers believe that the difference is so great that the judgment criteria are extended to the critical threshold. This view is supported by a lot of literature. A few years ago, editors of the journal Psychological Science in the Public Interest gathered a group of dissenting authors to review the literature.

"We did our best to put aside prior knowledge, abandoned assumptions and delved into the data," said Barrett, who led the team. "when there are differences, we just expand the scope of the search for evidence." They ended up reading about 1000 papers. Two and a half years later, the team came to a clear conclusion: there was little evidence that people could accurately infer the emotional state of others from a set of facial movements.

The team cited studies as an extreme that found no significant correlation between facial activity and internal emotional state. Carlos Crivelli, a psychologist at the University of de Montfort in Leicester, England, has been studying residents of the Trobrian Islands of Papua New Guinea and found no evidence in his study to support Ekman's conclusions. Crivelli concluded that trying to measure the inner state of mind by external markers is like measuring quality by length.

Another reason for the lack of evidence for universal expressions is that the face does not represent the whole of emotion. Other aspects, including body language, personality, tone and skin color changes, also play an important role in the process of perceiving and expressing emotions. For example, changes in emotional state can affect blood flow, which further affects skin color. Martinez and his colleagues have shown that changes in skin color can be associated with emotions. The visual environment, such as the background scene, can also provide clues to some people's emotional state.

From top left: basketball player Zion Williamson celebrates dunk; Mexican fans celebrate World Cup group stage victory; singer Adele won Grammy album of the year in 2012; Justin Bieber shouts live at a concert in Mexico City.

Mixed human emotions lead to possible overlap of expressions.

Other researchers believe that the debate about Ekman's findings (push-back) is overheated, especially Ekman himself. In 2014, in response to Barrett's criticism, Ekman identified a series of studies that supported his previous conclusions, including the study of natural human facial expressions and the relationship between expressions and the brain and body states beneath them.

These work show that facial expressions not only convey information about individual feelings, but also provide information about neuropsychological activation patterns, he wrote. He said his point of view had not changed.

Jessica Tracy, a psychologist at the University of British Columbia in Vancouver, Canada, believes that researchers who conclude that Ekman's theory of commonality is wrong based on negative examples are exaggerated. In a group of people or a culture, slightly different facial expressions of anger don't overturn the whole theory, she says. Citing an analysis of nearly 100 studies, she found that most people recognize an angry face when they see it. "A great deal of evidence shows that most people in most cultures around the world can see that this expression is common."

Tracy and three other psychologists argue that Barrett's literature review satirizes their position that they create an one-to-one rigid mapping between six emotions and facial activities. "researchers in the field of emotional science I know don't think that's the case," said Disa Sauter of the University of Amsterdam.

Sauter and Tracy believe that understanding facial expressions requires the establishment of an information-rich emotion classification system. Instead of regarding happiness as a single emotion, researchers should break down the categories of emotions. Happiness covers 喜悦 (pleasure), pleasure (satisfaction), compassion (compassion), pride (pride) and so on. The expression of each emotion may be different and may overlap.

At the heart of this debate is how to define "significant". In one study, participants were asked to choose one of the six emotional tags for each face they saw. Some researchers may think that the fact that a tag has been selected more than 20% represents a significant commonality. Other researchers may think that 20% is too low. Jack argues that the threshold for Ekman is too low. She read a paper written by Dr. Ekman. "I've been looking for my mentor to show him these charts from the '60s and' 70s, each of which shows huge differences in cultural cognition," she said. there is still no data to show that emotions can be universally recognized. "

In addition to saliency, researchers also argue about subjectivity. Many studies rely on experimenters putting emotional labels at the beginning of the test so that the final results can be compared. Barrett, Jake and other researchers are trying to find more neutral ways to study emotions. Barrett expects to use physiological indicators to provide approximate representations of anger (anger), fear (fear) or pleasure (喜悦). Jack uses computer-generated facial expressions rather than posted photos to avoid solidifying on six common expressions. Other researchers asked participants to put their faces in as many categories as possible to capture emotions, or to select participants from different cultures and mark the pictures in their own language.

Bioinformatics viewpoint: more data is needed

Technology companies often do not allow their algorithms to have room for free association. A typical emotion recognition AI program needs to enter millions of facial images and hundreds of hours of video clips, each of which is tagged to identify patterns.

Affectiva claims that its software has been trained with more than 7 million faces from 87 countries with an accuracy of more than 90%. The company declined to comment on the underlying theory of its algorithm. Neurodata Lab acknowledges that there are differences in facial expression of emotions, but when a person has an emotional attack, certain facial shapes appear more frequently, and their algorithm takes into account this commonality. However, due to concerns about the data of the training algorithm and the fact that there is still controversy in the academic circle, researchers on both sides of the above debate question this kind of software.

Ekman says he has directly challenged the companies' comments. He wrote to several companies (he did not specify which companies, saying only that they were the largest software companies in the world), asking for evidence of the feasibility of their automation technology, but so far he has not received a reply. "as far as I know, they are advocating things that are not based on facts," he said. "

Martinez acknowledges that automatic emotion testing can only reveal a group of people's general emotional responses. For example, Affectiva sells software to marketing agencies, touting that it helps predict how certain types of customers will react to a product or marketing campaign.

Even if the software makes a mistake, it doesn't matter-it's just that the advertising campaign may be a little worse than expected. But some algorithms are being used in areas that have a significant impact on people's lives, such as job interviews and border checks. Last year, Hungary, Latvia and Greece piloted a system for pre-screening travelers to detect fraud by analyzing microexpressions on their faces.

Solving the debate between emotion and expression requires different types of surveys. Barrett visited Microsoft last month and often asked to show her research to technology companies. She believes that researchers need to do what Darwin did when he wrote the Origin of species: "observe, observe." Observe what people express with their faces and bodies in real life, not just in the lab. Use machines to record and analyze fragments of the real world.

Barrett believes that more data and analytical techniques will help researchers discover new things, rather than reviewing old data and experiments. Technology companies are eager to develop science that she and many increasingly regard as unreliable, so she poses an open challenge to those technology companies.

"We are standing on a cliff," she said. Will artificial intelligence companies continue to adopt these flawed assumptions or do what needs to be done? "

Related reports:

Https://www.nature.com/articles/d41586-020-00507-5

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.