Shulou(Shulou.com)11/24 Report--

Photo: after a thrilling and exciting run, bilibili finally had a conversation with the creepy faceless man who tried to catch him, as he spoke as he slid across the synchronized subtitles at the bottom of the picture. It was supposed to be a scene in a horror movie, but it was actually his dream. Yes, he matched his dream with visible subtitles.

Write an article | not very well

Revision | clefable

Imagine the line "I am a magical on-screen barrage!" Floating in the air. If you have tried, you may find that this is not easy to do. Most people need to imagine the "face" of these words hard, and then take great pains to carve them carefully, otherwise the words will easily turn into a vague block of words that are difficult to read.

Source: no, but if you can easily imagine it, and you often see a continuous line of subtitles scrolling in your mind even in your life, you are likely to have the same rare subtitle synesthesia (synesthesia) as the above-mentioned people who have subtitled dreams. Usually, when we receive a sensory information, we experience only one perception, and the so-called synesthesia means that different senses are connected to each other across the senses, such as hearing colors or tasting shapes.

As early as the 19th century, scientists were surprised to find that some people thought that letters or numbers had a specific color. But at that time, most people saw synesthesia as a fantasy, or they were particularly good at metaphors rather than real sensory experiences. It was not until the 20th century that scientists confirmed the authenticity of synesthesia through systematic experiments.

Source: Wikipedia is less common in a variety of fantasy synesthesia experiences brought about by the intersection of many senses-such as morphemes and colors, words and tastes, music and pictures. In 1883, Francis Galton, a cousin of Charles Charles Darwin, observed that some people visualize what their interlocutors say in their heads. "some people will see every word spoken by the interlocutors rolling over a long piece of paper, just like a note unfolded on an automatic receiver." Therefore, subtitle synesthesia is often called automatic receiver synesthesia (ticker tape synesthesia,TTS).

It turns out that I am special all the time, there is not much empirical research on TTS. It is easily classified under the broad label of synesthesia and is not as bizarre and conspicuous as most synesthesia. "but the phenomenon of TTS goes far beyond anecdotes and helps us better understand the neural mechanisms by which the brain processes written language," says Fabien Hauw, a doctoral student at the brain Research Institute (Paris Brain Institute) in Paris, France.

Off and the team of neuropsychologist Laurent Cohen surveyed 26 volunteers with TTS for the first time systematically collecting data on the timing of visual and subtitle appearance, triggering mechanisms, whether they can actively control and how to deal with intrusive speech messages in TTS. The study was published in the journal Cortex in December 2022.

Because TTS, or most synesthesia, is a subjective experience, finding synaesthetes from many volunteers is challenging because sometimes they don't even realize they are special. More interestingly, "several of the volunteers who were eventually recruited for the study realized through the recruited poster messages that not everyone had their own built-in subtitles," Coeng said with a smile. With regard to the probability of TTS in the population, due to sampling deviation, different studies have different conclusions, so far there is no final conclusion, but the probability is estimated to be about a few percent. (the researchers say previous studies have concluded that the probability of TTS in people is 7% of people-must be overestimated. )

What kind of movie is the world? To learn more about TTS, Koeng's team set up a detailed questionnaire for volunteers to collect their subjective feelings from multiple dimensions. First, they need to determine when TTS will appear, when and how volunteers realize that it is unusual. In a rough concept, TTS is triggered by external voice, so can it be triggered by its own language? Do auditory parameters such as emotion and gender affect the perception of TTS? How much self-control do synaesthetes have over this phenomenon?

Image source: the essence of UnsplashTTS is a vivid psychological visual image. So the researchers also wanted to explore whether volunteers perceive subtitles in their own minds, or do they project subtitles to the outside world, like images of virtual reality. Is the position of the subtitle fixed? They often use uppercase or lowercase letters (the subjects of this study are volunteers whose mother tongue is French), whether they have color, and so on.

At the same time, whether there is a delay between auditory speech perception and the appearance of subtitles, and the duration of subtitles after speech stops, whether TTS is triggered by foreign languages, new words, and different types of non-verbal sounds (such as sneezing, meowing or ringtones), and the effects of TTS on daily life are also questions that researchers are trying to figure out.

For the trigger mechanism of TTS, the researchers used additional auditory stimuli to assist in the evaluation. They selected many types of sounds from an open speech library, such as homonyms (similar to magazines and impurities), mispronounced words (similar to magazines and our journals), false words (words that correspond to sounds but not semantic as opposed to real words, such as refrigerators and ice fairies), as well as non-verbal sounds such as animal calls and car engines, and played them to volunteers in turn. And ask them to write down any subtitles that may be induced.

Anytime, anywhere, unable to control the content collected, researchers found that 19 volunteers (73%) reported having TTS for as long as they can remember (about 8 years old) or literacy (about 11 years old), indicating that preliminary literacy is a prerequisite for the emergence of TTS. For synaesthetes, TTS has a clear advantage-it helps them learn to spell words correctly more quickly. A previous study also showed that 10-to 11-year-old TTS synaesthetes are better at expressing and learning new words than other children of the same age, which shows the advantages of TTS in language learning. In contrast, in a crowded and noisy environment, it is difficult for them to focus on a single interlocutor. About half of the volunteers agreed that TTS brought both advantages and obstacles.

Eighteen volunteers said TTS was an automatic phenomenon that they could hardly control. All volunteers trigger TTS when they hear and watch others talking. In most cases, TTS occurs even if the speaker is out of sight, or the volunteers speak openly, or even a hidden inner monologue. There is no real acoustic process in internal speech, but it can activate TTS as well as external speech process, which may reveal the co-activation mode of brain regions in TTS, but it remains to be further explored.

However, some volunteers reported that the voice of someone who spoke or was out of sight did not trigger TTS. In this regard, the researchers believe that both situations are due to the fact that the volunteers' attention is not focused on the input voice: the speaker in the field is usually insignificant and should be filtered out, while when talking to people, they focus more on the intention of communication than on their own voice content. This is likely to result in a lower probability of triggering the TTS.

Another explanation is that this phenomenon is closely related to the obstruction of lip reading. Neither of these situations can visually assist volunteers in reading words. As a result, some researchers believe that the generation of TTS may be related not only to auditory phonetics, but also to visual perception of the face.

When studying the triggering mechanism of TTS, the most common mental dictionary-the vocabulary stored in memory is like a dictionary stored in the human brain-the real words (with pronunciation and semantics) naturally trigger TTS. But the researchers also found that pseudo words (only phonetics but no semantics) and even nonverbal sounds can trigger TTS.

A volunteer tells of a bird that often sings in his garden early in the morning. A few days later, he translated the song into onomatopoeia. From then on, every time the bird sang, a string of onomatopoeia subtitles would be triggered in his mind. This situation is also similar to the stimulation of TTS by a foreign language. Although these sounds may lose their original semantics, they can still be included in the "lexicon" of their minds through speech coding.

In fact, this process also reveals the process of language learning. Linguistically speaking, pronunciation (phonology, words are encoded in abstract phonetic forms in mental dictionaries) and orthography (orthography, which standardize the form and rules of use of words, so there are the concepts of true words and false words, orthographic words and wrong words, and most languages, including Chinese, will contain some information about phonetic structure in orthographic method) can be combined with semantic understanding to improve reading and writing ability.

Photo source: Unsplash researchers can clearly identify the coding path of speech conversion to orthography in the subtitles fed back by volunteers stimulated by different non-verbal phonemes. for example, in all volunteers, familiar cat calls trigger the onomatopoeic word "meow" ("MIAOU" in French), but for unfamiliar rooster calls, only a small number of people trigger "HUHUHU". Some sounds trigger not onomatopoeia, but the name of the object that makes the noise. For example, the sound of gunfire makes volunteers think more of "GUN" or "GUNSHOT".

In a 2022 TTS magnetic resonance imaging (MRI) study, they found that when synaesthetes listened to a monologue, the areas of the left hemisphere responsible for speech analysis and specific areas involved in spelling, the visual lexical processing area (Visual Word Form Area,VFWA), were more strongly activated, the same as the brain regions activated during reading. In contrast to ordinary people who translate written text into speech, TTS synaesthetes automatically convert speech into written text, that is, TTS is more like an upside-down form of reading.

What's interesting about the subtitles is that TTS is also involved to some extent while volunteers are reading or watching movies. When reading in a quiet place, about half of the volunteers said they would only see the printed text, while the other half would see the same version of synesthesia on top of the text. More painfully, if someone was talking on one side, 88% of the volunteers said it would cause the speaker's voice subtitles to invade the written text and "pollute" it.

Similarly, when watching a movie with subtitles, half of the volunteers were able to see both the movie subtitles and the second-level synesthesia subtitles of the actors when they were talking. A small number of volunteers also reported that they could see the subtitles in their dreams. In a way, it adds a movie dimension to their dreams.

In the volunteers' descriptions, synesthesia subtitles are also extremely rich. Eighteen volunteers saw the subtitles in their minds, two of them projected the subtitles into the outside world, and the rest said they were free to switch. The position of their subtitles will also be different, some subtitles will move left or right according to the direction of the sound source, some subtitles will be as close to the mouth of the speaker as the bubble of comics, and others will be similar to the movie subtitles. Scrolling like a teleprompter.

Source: different bilibili volunteers see different colors, fonts and font sizes of the subtitles. Some volunteers said, "when I am moved, the letters will blur or wobble", "when the emotion is intense, the words will become louder", "the louder the sound, the bigger the words". When hearing the lyrics in the music, "the words seem to rise and fall with the melody" and so on. In fact, some of them not only have TTS synesthesia, but also have more than one synesthesia experience, such as space-time synesthesia, sound color, absolute pitch and so on.

TTS synesthesia is so special that the speech and the words and images that trigger it are linguistic representations, so it provides a very special perspective for us to understand the human reading mechanism.

However, synesthesia itself is not that strange. When many people hear a song or smell a certain smell, they will feel as if they have returned to a certain memory, which is essentially a kind of synesthesia. When the strong human imagination is applied in literary creation, there is the so-called synaesthesia rhetoric.

In the Book of Music in the Book of Rites, "the singers are as resistant as the top and the lower as the team, and the songs are like folds, as long as the wood, the middle moments of the sentences, the hooks in the sentences, and the tiring ends are like beads." it tells that music is not only pleasing to the ear, but also can communicate with people's sense of touch, vision and emotion through hearing.

Two paintings in "Picture Exhibition"the rich" and "the poor" if you also want to experience synaesthesia with a more sense of substitution, you might as well listen to the piano suite of "Picture Exhibition", which depicts pictures in music.

Reference link:

Https://doi.org/10.1016/j.cortex.2022.11.005

Https://www.tandfonline.com/doi/full/10.1080/17588928.2015.1048209

Https://www.eurekalert.org/news-releases/978603

This article comes from the official account of Wechat: global Science (ID:huanqiukexue)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.