In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-21 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > IT Information >
Share
Shulou(Shulou.com)11/24 Report--
Is it wrong to pursue the big model blindly?
Will the most influential developments in the future come from industry or academia?
Does the language model understand the language or not?
Is the research I have done valuable or not?
.
A recent survey of the NLP community reflects NLPer's views on all these important issues. The research team from the University of Washington, New York University and Johns Hopkins University solicited the opinions of a large number of researchers on some controversial issues in the field of NLP, including their views on large models, AGI (General artificial Intelligence), language understanding, future direction, and so on.
Paper address:
Https://nlpsurvey.net/ nlp-metasurvey-results.pdf
The results of the survey were so shocking that as many as 67% of NLP researchers were skeptical about the scientific value of their research.
Some netizens complain that even tarot cards are more reliable than NLP.
Other interviewees agree with a fairly high percentage of views:
The most frequently cited research results will come from industry rather than academia, which now has too much influence in leading the development process in the field of NLP
Oppose the assumption that maximizing scale can solve all problems
"NLP Winter" will come in the next 30 years.
NLP researchers should pay attention to AGI
NLP researchers should give more consideration to integrating insights from neighboring disciplines (such as linguistics, cognitive science).
Most of the studies published over the past five years to build interpretable models have gone in the wrong direction; and so on.
First of all, let's give a brief introduction to the respondents in this survey:
A total of 480 NLPer participated in the survey. 327 of them, who had published at least two ACL papers in the past three years, were the target group of the survey.
In terms of regional distribution, 58% of the respondents were from the United States, 23% from Europe and 8% from Asia. Among them, 3% are NLP researchers from China.
Seventy-three percent of the respondents came from academia, 22 percent from industry, and 4 percent worked in non-profit organizations or government. Among them, 41% of teachers and senior managers are junior researchers (including postdoctoral students), 33% are doctoral students, and 2% are postgraduates or undergraduates.
In addition, the proportion of male to female respondents was 67% and 25% respectively.
Let's take a look at the detailed results and analysis of this survey:
1 the overall situation in the NLP field
▲ Chart Note: for each issue, the bottom of the chart shows the proportion of consent, weak consent, weak disagreement, and disagreement. The vertical green line (green number) indicates the total percentage of agreement or weak agreement with the statement.
The domain influence of industry Q1-1: the private sector has too much influence in guiding the development process in this field.
Q1-2: the most frequently cited papers in the next 10 years are more likely to come from industry than academia.
The survey results show that the vast majority of people (86 per cent) believe that the most cited research results in the next 10 years will come from industry rather than academia, but 77 per cent also believe that today's private enterprises have too much influence in leading the development of the NLP field.
In the view of many, the number of citations of a job is not a good representation of its value or importance, and the continued dominance of industry in the field will have a negative impact, such as absolute control of basic systems, such as GPT-3 and PaLM.
However, respondents' answers to the two questions varied widely, with 82 per cent of academics who thought industry was too influential, compared with 58 per cent in industry.
When will the cold winter of NLP come? Respondents were asked if they expected a "NLP winter" in the near future, in which capital and employment opportunities fell by at least 50 per cent from their peak.
Q1-3: I expect a "NLP winter" in the next 10 years.
Q1-4: I expect a "NLP winter" in the next 30 years.
30 per cent of people agreed or weakly agreed that the cold winter would come in the next 10 years, of which only 7 per cent chose to "agree", while far more believed that winter would come in the next 30 years, accounting for 62 per cent.
Although 30% is not a large number, it also reflects the belief of this group of NLP researchers that NLP research will change significantly in the near future (at least in terms of who provides funding and how much). Why are their attitudes relatively pessimistic? There are many possible reasons, such as the stagnation of innovation caused by the excessive influence of industry, the industry will monopolize the industry with a small number of well-resourced laboratories, and the boundaries between NLP and other AI subdomains will disappear.
The scientific value of NLP Q1-5: most of the work published in the field of NLP is questionable in scientific value.
Sixty-seven per cent of NLP researchers reflect on the value of research in this field, arguing that most NLP work is scientifically dubious.
Respondents may have a variety of definitions of "suspicious", including work that is not complete at all, inappropriate questions studied, meaningless findings, or findings that are unimportant and unreliable.
The necessity of author anonymity Q1-6: during the review, author anonymity can be guaranteed to limit the dissemination of research.
The anonymity policy of ACL meetings is much stricter than that of many other meetings, such as NeurIPS, ICLR, and ICML. The survey shows that, despite the controversy, the NLP community generally supports this policy, with 63% of people believing that anonymity can guarantee to limit the spread of preprints. The issue also shows significant gender differences, with 77 per cent of women agreeing and only 58 per cent of men agreeing.
(2) scale, inductive bias and inspiration from adjacent fields
Is scale maximization the ultimate solution? Q2-1: scaling can actually solve any important problem.
Richard Sutton, the father of reinforcement learning, published a well-known view in his article "The Bitter Lesson": the general method of calculation is ultimately the most effective, and the efficiency will be greatly improved. From this point of view, the model is getting bigger and bigger, and the researchers expect that with sufficient training data and model capacity, compared with the inductive bias of introducing language structure or expert design, the use of fewer and more general principle learning mechanisms is a better solution.
However, the results of this survey show that NLP researchers actually agree with this view of Sutton much less than expected. Only 17 per cent agreed or weakly agreed that, given the likely development of math and data in this century, expanding the implementation of existing technologies would be sufficient to solve all the important problems of NLP. At a time when everyone seems to be flocking to big models, this number is extremely low.
The value of language theory and inductive bias Q2-2: the discrete representation of language structure (such as word meaning, syntax or semantic graph) based on language theory is necessary to solve some important real-world problems or applications in NLP.
Q2-3: strong inductive biases designed by experts (such as general grammar, symbolic systems, or cognitive-inspired computational primitives) are necessary to solve important real-world problems or applications in NLP.
Q2-4: by 2030, at least one of the five most cited systems is likely to draw clear inspiration from the results of linguistic or cognitive science research over the past 50 years.
In contrast to the large-scale point of view, the respondents' support for language theory and inductive bias is actually not low. 50% believe that language structure is necessary to solve the NLP problem, and 51% believe that the inductive bias of expert design is also important. Many NLP researchers seem to believe that the current trend of using neural network architecture with low inductive bias for end-to-end modeling will be reversed.
In addition, 61% of respondents said that the five most cited systems in 2030 are likely to draw inspiration from linguistic or cognitive science research over the past 50 years. In fact, the current system's reference to cognitive science is only a rough explanation of neurons, attention and token.
3 AGI and its risk
AGI controversy Q3-1: understanding the potential development of AGI and its benefits / risks should be an important concern for NLP researchers.
Q3-2: the latest progress in large-scale machine learning modeling (such as language modeling and reinforcement learning) is an important manifestation of the development of AGI.
The versatility and amazing language output of large pre-training models such as GPT-3 and PaLM have caused a great deal of controversy about general artificial intelligence (AGI), including predicting when AGI will arrive, whether we are really moving towards AGI, and what the consequences of AGI will be.
On the issue of AGI, the opinions of the respondents are evenly distributed, 58% believe that AGI should be an important concern for NLP researchers, and 57% believe that recent research has significantly promoted our development towards AGI. There is a high positive correlation between these two views.
What will AGI bring? Q3-3: in this century, labor automation caused by the progress of AI / ML may lead to economic restructuring and social changes on the scale of the industrial revolution.
Q3-4: in this century, decisions made by AI / ML systems could lead to a catastrophe on the scale of an all-out nuclear war.
Seventy-three percent of respondents believe that the automation of AI may soon revolutionize society, which is why so many people think that AGI is an important issue. It is worth noting that while 23% of people agree with this change, they do not agree with the importance of AGI, so it may not be necessary for the discussion about NLP to be involved in the debate about AGI.
In addition, about 1/3 (36 per cent) believe that AI decisions could lead to a nuclear war-level disaster. This shows that a considerable number of researchers are worried about AGI.
4 language comprehension
Does the language model (LM) understand the language? Q4-1: for generation models that are trained only on text, natural language can be understood as long as there are sufficient data and computing resources.
Q4-2: for multimodal generation models (such as a model that is trained to access images, sensors, actuator data, etc.), natural language can be understood as long as there are sufficient data and computing resources.
Half (51%) agreed that LM understood the language, and 67% agreed if the model also had access to multimodal data (images, etc.).
Q4-3: in principle, we can evaluate the model's understanding of natural language by tracking the model's performance on pure text classification or language generation benchmarks.
By contrast, only 36% of people believe that plain text assessment can measure language comprehension. This shows that in the view of many people, assessment is an independent problem, and understanding may be learnable but not measurable.
5 existing problems and future direction
Focus too much on scale and benchmark Q5-1: the current field of NLP focuses too much on increasing machine learning models.
Q5-2. The current NLP world is too focused on optimizing benchmark performance.
72% and 88% of NLP researchers believe that too much attention has been paid to scaling and optimizing benchmark performance.
Is NLP going in the wrong direction? Q5-3: most of the model architecture studies published in the past five years have gone in the wrong direction.
Q5-4: most of the open language generation task studies published in the past five years have gone in the wrong direction.
Q5-5: most of the studies published in the past five years to build interpretable models have gone in the wrong direction.
Q5-6: most of the explanatory black box model studies published in the past five years have gone in the wrong direction.
In the four specific research directions of model architecture, language generation, interpretable model and black box interpretability, the consent rates of the interviewed NLP researchers on the above issues are 37%, 41%, 50% and 42%, respectively. On the issue of interpretable model, the critical attitude of community members is more significant.
Interdisciplinary value Q5-7:NLP researchers should pay more attention to integrating the insights and methods of science in related fields (such as sociolinguistics, cognitive science, human-computer interaction).
Up to 82% of people think that NLP research needs to learn from more related fields of science. The problem is that although so many people emphasize this point, they are not doing very well, and the real problem may not be that NLP researchers do not realize the importance of interdiscipline, but that we still lack the knowledge and tools to promote implementation.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 234
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.