Shulou(Shulou.com)11/24 Report--

CTOnews.com, April 25 (Xinhua) Last month, OpenAI launched the latest AI chat robot, GPT-4, and claimed that the robot performed exceptionally well in various exams, such as the bar exam and the graduate entrance exam. However, according to a study led by Brigham Young University in the United States, ChatGPT lags far behind humans in accounting exams and even talks nonsense to defend the wrong answer.

Wood, an accounting professor at Brigham Young University in the United States, and his team decided to take a test on ChatGPT to see if it could answer questions for college accounting courses. They launched a call on social media that attracted 327 partners from 186 educational institutions in 14 countries. They provided a total of 25181 accounting examination questions, covering areas such as accounting information systems, auditing, financial accounting, management accounting and taxation. They also invited BYU undergraduates, including Wood's daughter Jessica, to input 2268 textbook questions into ChatGPT.

The results show that although the performance of ChatGPT has some bright spots, it is still lower than the level of students as a whole. The average score of the students is 76.7%, while the average score of the ChatGPT is only 44.4%. On 11.3% of the questions, the ChatGPT score was higher than the average student score, especially in accounting information systems and auditing. However, ChatGPT appears to be inadequate in taxation, finance and management, perhaps because these areas require more mathematical computing power. In terms of question types, ChatGPT is good at judging questions (68.7% correct rate) and multiple choice questions (59.5% correct rate), but it is very difficult to answer short questions (the correct rate is between 28.7% and 39.1%). Generally speaking, ChatGPT is more difficult to answer questions that require a higher level of thinking. Sometimes, ChatGPT even provides an authoritative explanation for the wrong answer, or gives different answers to the same question.

"it's not perfect. You can't expect it to do everything." Jessica said, "it's stupid to rely solely on ChatGPT to learn."

CTOnews.com noted that the researchers also found some other interesting phenomena:

ChatGPT sometimes does not know that he is doing math problems and makes some ridiculous mistakes, such as adding two numbers to the subtraction problem or miscalculation of division.

ChatGPT often provides explanations for its answers, even if they are wrong. Sometimes, ChatGPT's description is correct, but it chooses the wrong option.

ChatGPT sometimes makes up facts. For example, when providing a reference, it produces a reference that appears to be real, but it is actually completely fictional, and sometimes even the author does not exist.

Nevertheless, the researchers believe that in future versions, ChatGPT will make progress on accounting issues and solve the above problems. They think the most promising thing is that the chat robot can help improve teaching and learning, such as designing and testing assignments, or for drafting parts of the project.

"this is an opportunity for reflection, and we need to see if we are teaching valuable information." "this is subversion and we need to assess what we should do next," said Larson, an accounting professor at BYU, one of the co-authors of the study. "of course, I will still have teaching assistants, but the content of their work will be different."

The study was published in the journal Accounting Education.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.