"Save the AI field, save the children! "this is an initiative of AI practitioners. 03/13 Update SLTechnology News&Howtos

"Save the AI field, save the children! "this is an initiative of AI practitioners.

2025-03-13 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Shulou(Shulou.com)06/02 Report--

2020-04-10 12:46:43

Machine Heart report

Participation: Zhang Qian, mayonnaise

Yesterday, a "Save AI field, save children" appeared on Zhihu! The post has aroused widespread concern in the community. What's wrong with AI? How to save it? This article will focus on these issues.

Original link: https://zhuanlan.zhihu.com/p/127085043

The poster believes that the current "irrigation" and fake chaos in the field of AI have reached a point that can not be ignored, so they call on everyone to expose the problems found in the paper or code. To this end, she created a column called "online questioning of AI papers", hoping that you can feedback the problems found, discuss them together, and use your efforts to purify the AI community.

This initiative is supported by Liu Zhiyuan, an associate professor at Tsinghua University, and other researchers in the circle. After all, academic fraud and irrigation are very disadvantageous to the long-term development of the AI field, especially the former. As the poster said, scientific research is an iterative process. "if the top papers of the predecessors are fake, the articles of follow behind them are like castles in the air built on garbage, which is in jeopardy." "

At present, many top organizing committees and journal editors seem to have noticed this problem, so they pay more and more attention to the reproducibility of papers, requiring authors to submit codes, encouraging repetitive research, and so on. However, there are a large number of contributions, and the official power is limited, so we also need to rely on the "folk" forces.

Problems exposed by CVPR 2019

As a long-standing problem, the attention of the domestic community to the problems of "irrigation" and "fraud" in the field of AI seems to have reached the best part when CVPR 2019 was released.

CVPR 2019 finally received 5165 valid contributions, an increase of more than 50 per cent over the previous year. At that time, due to the substantial increase in the number of contributions, overleaf, an online collaborative editing tool for papers, collapsed because of too many editors.

Professor Tan Mingkui of South China Institute of Technology commented: "when Moore's Law in the hardware field gradually failed because of the hardware limit, the number of paper publications took over." "

Since then, more and more people have realized that today's summit meeting is no longer the original summit meeting.

People's dissatisfaction with CVPR and other problems at the top club finally broke out. In Zhihu's question, "what are the bad papers on CVPR in 2019?" below, a number of articles were pulled out and analyzed and criticized one by one, including oral papers, which "makes people think that CVPR is no more than that."

Such "doubts" also exist in various top meetings, with the popularity of artificial intelligence, the surge in the number of practitioners and the sharp increase in the number of papers submitted, which makes the top of the former halo look "bloated".

NeurIPS 2019

Tens of thousands of contributions, 1428 papers received, 57 workshop,13000 attendees, seen from afar, the conference looked like a concert.

ICLR 2020

The number of contributions has increased year by year, reaching more than 2600 in 2018, 1500 in 2019 and more than 2600 in 2020.

CVPR 2020

The number of papers submitted by ID has exceeded 10,000, and 6656 articles have been effectively contributed, which is double that of CVPR 2018.

People who have been in the AI field have more or less a strange peer pressure: after voting for NIPS, they are thinking about what to vote for AAAI, and after casting AAAI, they are thinking about what to vote for CVPR. Why are there so many jobs to invest in? Do not rule out that there are a small number of bosses and tycoons who can continue to produce impressive idea and can thoroughly verify idea from theory to practice in a short period of time, but what about people other than bosses? Nothing else, pure irrigation ear. (quoted from @ running dog under the information door)

How to save it?

There are a lot of problems, how to solve them? From the current top meetings, periodical trends and personal suggestions, it can be divided into two ideas. First, starting with meetings and periodicals, formulate more stringent contribution policies (such as requiring the submission of codes), and at the same time actively receive repetitive studies and studies that yield invalid results; second, the community's spontaneous "anti-counterfeiting" campaign, make full use of community resources, "let's find fault". In response to these two ideas, all parties seem to have begun to try.

NeurIPS 2020: "strongly recommended" to submit code

Since NeurIPS 2019, the organizing committee has launched a strategy (non-mandatory) to encourage the submission of paper codes. This strategy has had a certain effect. During the final submission phase of NeurIPS 2019, 75 per cent of received papers were accompanied by code, up from 50 per cent in 2018.

This year, the organizing committee continued to strengthen this strategy, from "encouraging" to "strongly recommended" (still not mandatory), and provided guidelines and templates for submitting code.

This rule is based on feedback from reviewers in 2019. According to the survey, after each review, 13% of reviewers said they had read the code, 21% said they had not read it, and the rest said "not applicable". This may be because many papers do not have code. Without providing the code, 21% of the reviewers said they wanted the code to be attached to the paper. And the organizing committee found that the quality of the code submitted with the paper was highly related to the score given by the reviewer.

Nature: encourage repetitive research and invalid results research

In an editorial in February this year, Nature proposed an initiative, pointing out that research institutions and journals should encourage repetitive research and research with invalid results (NULL results). The article emphasizes that researchers who publish invalid results can help communities invest a lot of money in more productive areas, and the wider dissemination of ineffective results will eventually prompt communities to change their theories and make them more applicable to the real world.

In response, this editorial article puts forward two strategies.

First of all, research institutions should encourage such behavior through words and actions. Last year, the Berlin Institute of Health issued a written initiative encouraging its researchers to publish their own repetitive and ineffective results to combat a recurrent crisis. Researchers who respond to the initiative will receive a reward of 1000 euros. In addition, the Institute has an APP and consultant to help researchers determine which journals, preprint platforms or other channels should be contacted to publish repetitive research and data. The APP will provide information about expected publication costs, submission deadline, paper format, peer review requirements, and so on.

Second, the article points out that more journals should emphasize to the community the importance of publishing repetitive research and invalid results, and Nature makes it clear that "in Nature, repetitive research is placed at the same level as other studies. Nature welcomes the submission of research that provides insights into previously published work. "

Create AI realm pubpeer?

Although periodicals and conferences have a wide range of influence, they are faced with a large number of contributions, and their strength is also limited, so they have the initiative mentioned at the beginning of the article. In response, Professor Liu Zhiyuan replied: "you can consider setting up an AI version of PubPeer."

Founded in 2012, PubPeer is a website that encourages researchers to comment anonymously on published papers. It is equivalent to a scientific forum, as long as scholars have a PubPeer account, they can comment on published papers, such as criticism, questioning, improvement suggestions and so on. Many of the previous research discussions at the center of the storm were fermented on this site, and some papers were even withdrawn as a result.

The existence of this kind of "anti-counterfeiting" websites can deter researchers with academic misconduct to a certain extent. If there are similar websites in the field of AI, I believe it will be helpful to improve the research atmosphere of the whole community.