Shulou(Shulou.com)11/24 Report--

GPT-4 passed the MIT test and the storm broke out again. Just now, the co-author of MIT personally clarified that it was a disaster caused by the use of unauthorized data sets by "pig teammates".

The author of the paper "the official fight against counterfeiting" is here!

Some time ago, GPT-4 passed the MIT Mathematics undergraduate examination, and even got nearly full marks to attract many netizens to watch.

However, as soon as this paper was released, it was revealed that there was something wrong with the "data set" by the students of the same school, and the result was not accurate.

Unexpectedly, as soon as the revelation came out, AI bosses LeCun, Marcus and so on came out one after another to make a sound.

Today, the author of the paper from MIT gives a formal explanation.

What is surprising is that Iddo Drori, one of the authors, rushed to distribute the paper without the permission of others.

Some co-authors even said that it was only after traveling over the weekend that they learned that the paper had been published.

Moreover, Iddo is said to have not only "concealed" its actual method, but also been told before publication that there were still problems in the paper that had not been corrected.

The full text of the statement was published on arXiv by Iddo Drori on June 15, a paper related to data on dozens of MIT course exams and assignments.

However, he did not get the consent of many co-authors, although he was told that some problems should be corrected before publication. And some of us didn't know that the paper had been published until Sunday, June 18, after a weekend trip.

In solving this problem, we found that, contrary to what Iddo Drori conveyed to us and the students who collected the data, Iddo did not have the permission of all mentors to collect the data sets of assignments and exam questions that make up the topic of the paper.

When the papers appeared on social media and Iddo posted data samples online without anyone's permission, some course mentors became aware of the existence of the data set and that their course materials were included.

These are serious issues that are being addressed through institutional channels, so we have not rashly stated this in public, but we think it is important to explain why this paper should not be published and must be withdrawn.

We have asked Iddo to withdraw the paper from arXiv and contacted arXiv directly to explain the situation.

We would like to emphasize that all the student authors work very hard in this paper, which could have been very interesting and valuable if the data had been collected with consent. Many of the problems in published papers are not the fault of the students.

What's more, GPT-4 can't get a degree from MIT.

Netizens: I'm afraid it's not a pot bar. LeCun retweeted and commented on this statement, "Thank you for clarification."

Raunak Chowdhuri, which has pointed out the problem, has also put the update at the top.

However, some netizens pointed out that the problem of this paper lies not in whether it is "agreed" to be published, but in the "method" itself.

Now it seems that the authors want their names to appear in the potentially popular paper, but do not want to take responsibility for their mistakes.

If the paper had not been "cracked down", then there would not be this so-called "public statement"-forcing some of the authors to separate from the paper.

Obviously, as a co-author of the paper, you must be responsible for the quality of the work you signed.

Another netizen said: "this is the worst scapegoat I have ever seen in my life." "

Interestingly, except for the rush to throw off the pot after the "fraud" of the paper was caught-although I signed it, this question has nothing to do with me. Prior to this, there was a similar scene on the top meeting IJCAI 2016-crazily pulling people after the paper was accepted.

"author X was actually involved, but we didn't have time to write it down. "

Article address: http://ijcai-16-pc.blogspot.com/ 2016 Compact 04 / the-increasing-practice-of-expanding-co.html the day after the receipt list was sent, we found that someone was trying to add additional collaborators to their accepted papers.

I understand that sometimes after the submission of a paper, we may get very important help from colleagues, and our own research team does this occasionally. But all of a sudden, more than 50 papers are needed, which is a little strange.

More surprisingly, many of them found that they had not only one forgotten collaborator, but also "multiple" (sometimes as many as four) forgotten collaborators.

Obviously, the proverb "success has many parents, but no one cares about failure" is fully reflected here.

However, we back up screenshots every week during the review, so we know the original authors of all the papers. (this is what finally appears on the receiving list).

GPT-4 broke the MIT examination GPT-4 opened in the MIT exam as soon as the result was announced, it attracted a lot of attention.

In the same test, GPT-3.5 got 1/3, while GPT-4 won it all.

This chart becomes the brightest part of the paper.

On June 15th, a team of researchers from MIT, Boston University and Cornell University published a new paper demonstrating GPT-4 's ability to take the MIT exam.

Paper address: in the https://arxiv.org/ pdf / 2306.08997.pdf paper, the researchers created a data set covering 4550 problems and solutions.

These include problem sets for undergraduate degree courses, midterm exams, and final exams for MIT Mathematics and EECS students.

The details are as follows:

The researchers randomly generated 228 questions from the data set, not involving existing images and solutions.

Then, five of the most advanced language model models were taken together: GPT-4, GPT-3.5, StableVicuna-13B, LLaMA-30B, and LLaMA-60B.

In the end, it was found that the tuned GPT-4 got a score of 100%. The original version of GPT-4, without any tuning, scored 90 per cent.

The specific tuning process, as shown in the result diagram, includes Few-shot+CoT+Self-critique+Experts.

With each additional tuning step, the ability of GPT-4 jumps by one step.

What was controversial about the study at the time was that GPT-4 was asked to rate itself.

The team tuned GPT-4 on the dataset, given question Q, benchmark solution S, and answer An of LLM, and automatically graded the model response using GPT-4.

It is doubtful that GPT-4 gave himself a full score.

The visiting professor is accused of "scrambling to send" the paper Iddo Drori.

Iddo Drori is an associate professor of computer science practice at Boston University, a visiting associate professor at MIT, and an adjunct associate professor at Columbia University.

Previously, he was a lecturer at MIT EECS, a visiting associate professor of operations research and information engineering at Cornell University, and a research scientist and adjunct professor at New York University's data Science Center, Courant Institute, and NYU Tandon.

He holds a doctorate in computer science and has done postdoctoral research in statistics at Stanford University. He also holds an MBA degree in organizational behavior and entrepreneurial management and has ten years of industrial research and leadership experience.

Iddo Drori's main research areas are machine learning, artificial intelligence and computer vision. It has published 70 papers, been cited more than 5200 times, and taught 35 computer science courses.

He is the author of the textbook Science of Deep Learning published by Cambridge University Press. He has won a number of competitions at computer vision conferences and won many best paper awards at machine learning conferences.

Just now, some netizens keenly discovered: "Iddo has not only removed the title of 'MIT visiting Professor' from the LinkedIn home page, but his guest position seems to be coming to an end this month." "

Three co-authors Armando Solar-Lezama

Armando Solar-Lezama is a professor of Electrical Engineering and computer Science (EECS) at the Massachusetts Institute of Technology and deputy director and chief operating officer of the computer Science and artificial Intelligence Laboratory (CSAIL).

He is the chief project leader of the Expeditions project understand the World through Code, funded by the National Science Foundation (NSF), and is the founder of playskript, an online platform for creating interactive presentations.

His research focuses on program synthesis. This is an exciting research area. On the one hand, program synthesis involves the use of automatic reasoning and learning to help introduce more automation into the programming process. On the other hand, code provides a unique modeling mechanism, so program synthesis can play an important role in building more predictable and robust learning systems.

Tonio Buonassisi

Tonio Buonassisi is a professor of mechanical engineering at MIT. His research is mainly focused on solar photovoltaic and techno-economic analysis, and has played an important role in the technological development of many companies. As a result, he has won the Presidential early Scientist and engineer Award (PECASE), the National Science Foundation Professional Award (CAREER Award) and the Google teacher Award.

At MIT,Tonio Buonassisi, he is the head of the Sustainable Development accelerated Materials Lab and leads the research work on sustainable material development. He also served as the founding director of the accelerated material Manufacturing Program in Singapore. In addition, he co-founded the startup Xinterra and the nonprofit Fraunhofer Center for Sustainable Energy Systems.

Tonio Buonassisi has shown great enthusiasm and talent in education. He has won the MIT Everett Moore Baker Outstanding undergraduate Teaching Award, and his teaching influence is not limited to the classroom, but has also been viewed more than 179000 times through his OpenCourseware / YouTube photovoltaic lecture series. He also recently produced a series of YouTube videos called "accelerating material Manufacturing", focusing on the application of artificial intelligence in material research.

Yoon Kim

Yoon Kim is an assistant professor at the Massachusetts Institute of Technology (EECS / CSAIL). Previously, he received a doctorate in computer science from Harvard University, and his mentor was Alexander Rush.

His research interests include the efficient training and deployment of large-scale models, the ability and limitations of understanding large language models, the use of symbolic mechanisms to control and enhance the links between neural networks, computing and human language processing.

Now, it can be said that GPT-4 has become a new "benchmark" in the field of LLM.

On the one hand, this trend forces researchers to compare their work with it, on the other hand, it has spawned a considerable number of studies that only follow the trend and hype.

Not only that, the "black box" approach pioneered by OpenAI in the GPT-4 technical report has also been emulated by others.

In the discussion of HackerNews, one user said that machine learning is no longer a scientific field, but has become like social science, based on another kind of research that cannot be falsified and can not be reproduced.

Some media said that this incident undoubtedly set a bad precedent in the field of artificial intelligence, causing people to question the authenticity of the research-how many papers on the Internet are actually problematic?

As the benchmark GPT-4 begins to get involved in the writing phase, the quality of the paper is expected to decline further.

Reference:

Https://people.csail.mit.edu/asolar/CoursesPaperStatement.pdf

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.