Shulou(Shulou.com)11/24 Report--

The title of the original text: "the top reviewer of AI is an undergraduate?" The female physicist tweeted and complained, and the big guys in the circle liked it one after another. "

Recently, Lenka Zdeborova, a female physicist from the Lobsang Federal Institute of Technology (É cole polytechnique f é d é rale de Lausanne (EPFL)), complained on Twitter about the screening mechanism for reviewers at the top meeting, which resonated with many leaders in academic circles.

Do you think the reviewers of top academic conferences and journals are all Daniel from their respective fields?

In fact, the reviewers of the research papers you have worked so hard for months are likely to be a group of undergraduate or graduate students?

Recently, Lenka Zdeborova, a female physicist from the Lobsang Federal Institute of Technology (É cole polytechnique f é d é rale de Lausanne (EPFL)), complained on Twitter about the reviewer mechanism of the top meeting.

Graduate school applicants already have the experience of reviewing manuscripts? And there are a lot of them.

I am reviewing the application of EPFL Graduate School of computer Science, and I see that many applicants have experience as top ML conference reviewers. I wonder why we bother to conduct peer review if a large part of peer review is done by undergraduates.

Lenka Zdeborov á is a professor of physics and computer science at EPFL. She leads the statistical physics department at the computer School.

She is the editorial board of Journal of Physics A, Physical Review E, Physical Review X, SIMODS, Machine Learning: Science and Technology, Information and Inference and many other academic journals.

Lenka specializes in applying statistical physics concepts (such as advanced mean field methods, replication methods, and related messaging algorithms) to machine learning, signal processing, reasoning, and optimization problems.

Some people in the circle also agree that first-year graduate students can already review manuscripts at the top meeting, and no one thinks this is a problem.

You think the reviewers are these academic leaders, but in fact they are undergraduate or graduate students.

Zhihu respondent "Kaka Karabi" believes that this phenomenon is already commonplace, from ICLR to AAAI, the level of reviewers is uneven, and your paper will fall into the hands of an undergraduate.

After complaining about the level of reviewers, the key is to think about how to write a make sense that allows everyone to taste and most people review the manuscript.

Source: Kaka Kakabi, another respondent, "hysterical I" also believes that reviewers seldom understand your field, so it is your problem to present the results of your paper clearly and clearly so that others can't understand it.

Source: hysterical I how to review the manuscript at the top?

Peer review (Peer-review) is the cornerstone of modern science, and almost all machine learning (ML) summit meetings (such as NeurIPS, ICML, AAAI) rely on it to determine whether the submitted papers are relevant to the community and whether they are original enough to be published there.

Unfortunately, with the exponential growth in the number of articles submitted over the past decade, the quality of manuscripts has declined at an equally rapid rate.

If you have ever submitted a paper to one of the meetings, after months of hard work on what you think is a great idea, what you get is bad and useless. And (worse) ironic comments mean that you will have to go through the submission process again without any hints about what is wrong with your paper.

Turing Award winner Geoffrey Hinton gave one of the reasons for this in an interview with Wired magazine in 2018:

Now if you submit a paper with a new idea, it has no chance of being accepted at all, because these young junior reviewers simply cannot understand it.

Or it will meet a senior reviewer who reviews too many papers and doesn't understand them for the first time and thinks it must be nonsense. I think it's really bad.

Zhihu answered the main electro-optical phantom alchemy to make a hypothesis about being rejected. If you find that your paper has been given a low score because of some conceptual problems that the reviewer does not understand, there are three possibilities:

(1) the reviewer is an undergraduate / junior doctoral student who does not understand these concepts.

(2) the reviewer is a super boss, he attaches great importance to basic concepts, and he has not worked in this field for many years.

(3) the reviewer was a super boss, but he had no time, so he handed it over to his junior doctoral students to review the manuscript. Therefore, the academic circle is a reincarnation, and the performance of undergraduates is the most similar to that of super bosses.

What do you think of the ideal reviewer:

Open the paper on Monday, look at the abstract, read it quickly, and evaluate the novelty. Look at the methods carefully on Tuesday and find fault. Take a look at the experiment on Wednesday, look at the relevant papers of baseline, and check the details of the experiment. On Thursday, we will start to write comments on manuscript review and decide on the score. Add some comments on the review on Friday and submit the review score.

Actual reviewers:

I opened the manuscript review paper at eight o'clock on Monday morning and read the abstract. I felt that there was no novelty. After looking at the picture, I feel that I can't understand it without looking at the text. I turned to the back and looked at the experimental figures, and I felt that the improvement was not great. Look at the method. I can't finish a lot of four pages. I have something to do this afternoon, so give me a 5 and go. The review of the manuscript ended at 11:30 on Monday.

Let's take a look at ICML2022's manuscript review rules:

Significance of the topic:

The purpose of this part of the review is to show MR (Meta-reviewer) and the author how much you understand the paper and what you think of it.

Innovation, relevance and importance of writing:

Are they relevant to our community? Are they new? If the answer is no (or partially no, for example, citing precise results from an earlier paper), a precise reason is needed so that the author knows how to repair the paper.

Integrity:

Ideally, the proposition put forward in a paper should be fully supported by theoretical arguments or experimental results.

Writing quality:

Is this paper well organized and clearly written? Does it explain the novelty and results well? Does the paper contain sufficient information to support its claim?

References:

Is this paper properly placed in contemporary literature? If not, please specify what is missing. Please note that because the paper has a page limit, it is usually necessary to judge whether the results should be mentioned.

These belong to the audit standards of the first stage reviewers, and those who meet the audit standards can enter the second stage.

Through the first stage of the paper, it will be reviewed by the original reviewer (MR) and the project chairman, and then there will be additional reviewers to ensure the quality of the paper review.

And why the current level of top reviewers is uneven, and even a group of undergraduate and graduate students have emerged, mainly because the industry is developing too fast, and the rate of increase in the number of contributions is much higher than that of qualified reviewers.

This leads not only to fewer people with the ability to review manuscripts, but also to less time for each article.

Reference:

Https://towardsdatascience.com/reviewing-for-machine-learning-conferences-explained-f73bc037babc

Https://twitter.com/zdeborova/status/1612841482192388098

Https://www.zhihu.com/question/461564257/answer/1908243034 、

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.