An international research team led by Carnegie Mellon University has found that when the brain “reads” or decodes a sentence in English or Portuguese, its neural activation patterns are the same.
Published in NeuroImage, the study is the first to show that different languages have similar neural signatures for describing events and scenes. The research team discovered this by using a machine learning algorithm that essentially learns the relationship between sentence meaning and brain activation patterns in English and then recognizes sentence meaning based on activation patterns in Portuguese. The findings can be used to improve machine translation, brain decoding across languages, and potentially second-language instruction.
“This tells us that, for the most part, the language we happen to learn to speak does not change the organization of the brain,” said Marcel Just, Carnegie Mellon’s D.O. Hebb professor of psychology, a pioneer in using brain imaging and machine learning techniques to identify how the brain deciphers thoughts and concepts. “Semantic information is represented in the same place in the brain and the same pattern of intensities for everyone. Knowing this means that brain-to-brain or brain-to-computer interfaces can probably be the same for speakers of all languages,” Just said.
For the study, 15 native Portuguese speakers—eight were bilingual in Portuguese and English—read 60 sentences in Portuguese while in a functional magnetic resonance imaging (fMRI) scanner. A CMU-developed computational model was able to predict which sentences the participants were reading in Portuguese, based only on activation patterns.
The computational model uses a set of 42 concept-level semantic features and six markers of the concepts’ roles in the sentence, such as agent or action, to identify brain activation patterns in English.
With 67% accuracy, the model predicted which sentences were read in Portuguese. The resulting brain images showed that the activation patterns for the 60 sentences were in the same brain locations and at similar intensity levels for both English and Portuguese sentences.
The results also revealed that the activation patterns could be grouped into four semantic categories, depending on the sentence’s focus: people, places, actions, and feelings. The groupings were very similar across languages, reinforcing that the organization of information in the brain is the same regardless of the language in which it is expressed.
“The cross-language prediction model captured the concept gist of the described event or state in the sentences, rather than depending on particular language idiosyncrasies. It demonstrated a meta-language prediction capability from neural signals across people, languages, and bilingual status,” said Ying Yang, a postdoctoral associate in psychology at CMU and first author of the study.
In addition to Just and Yang, the research team consisted of CMU’s Jing Wang and Vladimir Cherkassky and the Federal University of Santa Catarina in Brazil’s Cyntia Bailer.
The Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the U.S. Air Force Research Laboratory (AFRL), funded this research.
