Shulou(Shulou.com)11/24 Report--

The AI detector has a spectrum, and the new algorithm achieves 99% accuracy.

Before that, many people have developed ChatGPT detectors, but none of them can really identify them effectively.

Researchers from the University of Kansas have introduced a new algorithm that can detect whether or not to cheat with ChatGPT with an accuracy of more than 99%.

The latest study was published in the journal Cell Reports Physical Science on June 7.

Heather Desaire, a professor of chemistry at the University of Kansas and one of the authors of the paper, said

"We try to create an easy-to-use method so that even high school students can build an artificial intelligence detector for different types of writing with little guidance. "

Four major features, 99% recognition rate, as the researchers say, 90% accuracy is often not enough. But to achieve higher accuracy, the trade-off is often universal.

In the study, the researchers selected 64 articles written by human authors in the journal Science, covering subjects ranging from biology to physics.

Then, the data is fed to ChatGPT and used to generate a dataset of 128 artificial intelligence articles.

This set of training data contains 1276 sample paragraphs of chatbots.

The researchers used these data to construct ChatGPT detection algorithms.

After the model was fully developed and optimized, they also generated two test sets. Each test set has 30 real articles and 60 articles written by ChatGPT (a total of 1210 paragraphs) to form a new dataset to test the latest algorithms.

The experimental results show that the new algorithm can detect the whole article written by ChatGPT at 100%.

At the paragraph level, it is less accurate, but still impressive: the algorithm found 92% of the paragraphs generated by artificial intelligence.

It is worth mentioning that, according to the paper, you can find out from some details what content was created by ChatGPT.

Through the manual comparison of many examples in the training set, the researchers identified four types of features. These features help to distinguish between human writing and chat robots.

(1) paragraph complexity, (2) sentence length diversity, (3) punctuation, and (4) buzzwords or numbers

In general, human writers write longer paragraphs, use a larger vocabulary, and contain more punctuation.

Moreover, they tend to modify their statements with words such as "however", "but" and "although". ChatGPT is less specific in quoting numbers and mentioning other scientists.

In the following table classification, humans are much better at content than ChatGPT.

Of these four types of features, two (1 and 3) are ways in which ChatGPT produces less complex content than humans. The biggest difference is the number of sentences and the total number of words in each paragraph.

In both cases, the average value of ChatGPT was significantly lower than that of humans.

The researchers also found that humans are more likely to change sentence structures. Human beings change the sentence length more times than ChatGPT. Humans also use longer sentences (35 words or more) and shorter sentences (10 words or less) more frequently.

The remaining two types of distinguishing features can be more described as "stylistic" choices.

On the one hand, human scientists use question marks, dashes, parentheses, semicolons and colons more frequently, while ChatGPT uses more single quotes.

Humans also use more proper nouns and / or acronyms, as well as numbers.

The model, established by Desaire, doesn't work for teachers who want to punish high school students for cheating.

The algorithm is established for academic writing, especially the kind of academic writing that people read in scientific journals.

In theory, the company says, you can use the same technology to build a model to test other types of writing.

However, one has to consider the fact that a person can easily make minor adjustments to the writing of a chatbot and make it more difficult to detect cheating, which makes things more complicated.

Nevertheless, the researchers described the study as a "proof of concept" and said a more stable and accurate tool and larger data sets could be developed in the future.

If artificial intelligence continues to develop at a very rapid rate, no one can guarantee that such a detection method can still be effective.

Because the closer a large language model is to the ability to copy human language, the more difficult it is to recognize traces of robot language.

AI testing why it is so difficult has been used by students and teachers in many colleges and universities in daily homework and teaching since the advent of ChatGPT.

However, if not restricted, ChatGPT will become the most powerful cheating tool in history, helping students to do homework or even finish exam papers.

In order to counter reconnaissance, an easy-to-use detector has become something the teacher expects. Edward Tian, a 22-year-old Princeton student, developed a self-developed detector, the GPTZero.

Even OpenAI officials announced the launch of a new tool called AI Text Classifier document detector.

However, the performance of these detectors is not satisfactory.

Detecting the content created by AI sounds simple. But when we give you a handwritten email and an ChatGPT-generated email, we can hardly tell it apart.

Eric Wang, vice president of artificial intelligence at Turnitin, says testing AI writing with software involves statistics. From a statistical point of view, artificial intelligence differs from human beings in that it is extremely stable at the average level.

To put it bluntly, the level of AI is stable. In fact, however, this is not the case.

A system like ChatGPT is like an advanced version of auto-completion, looking for the next most likely word to write. This is actually why it reads so naturally. AI writing is the most likely subset of human writing. "

Reference:

Http://today.ku.edu/2023/05/19/digital-tool-spots-academic-text-spawned-chatgpt-99-percent-accuracy

Https://gizmodo.com/chatgpt-detector-ai-kansas-research-paper-99-accuracy-1850519081

This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.