Shulou(Shulou.com)11/24 Report--

The main issue of Science publishes a survey report on ChatGPT, "Experimental evidence of the impact of generative artificial intelligence on productivity".

For some paperwork tasks, ChatGPT not only improves performance by 18 per cent while saving 40 per cent of time, but also reduces performance differences between different people.

MIT officials also reported the results of the study.

The control group will also use the experiment to invite a total of 453 subjects, divided into the experimental group and the control group.

The subjects were highly educated professionals from different fields. The salary level, occupation, and other information of the people in the two groups are shown in the following table:

During the one-month experiment, they were asked to complete career-related writing tasks.

These tasks include writing press releases, short reports, analytical plans and e-mails for about 20 to 30 minutes. The evaluation indicators include two parts: efficiency and quality, and each result is blindly evaluated by three professionals in related fields (1-7 points).

The results showed that the time spent in the experimental group decreased by 0.75 standard deviation (about 11 minutes), while the score increased by 0.41 standard deviation.

If converted into a percentage, the proportion of time reduction and quality improvement is 40% and 18%, respectively.

Specifically, the experimental group was first assigned to complete a task, and then registered for ChatGPT (version 3.5). They are told that if they think ChatGPT works, they should use it in the second task.

The control group was asked to register a LaTex editor (registration complexity is comparable to that of ChatGPT), but there was no similar notification.

As a result, 80 per cent of the experimental group used ChatGPT in the second task, while less than 5 per cent of the control group used the LaTex tool.

For the second task, the experimental group took an average of 17 minutes, 10 minutes less than the control group. In terms of quality, the average score of the experimental group was 4.53, while that of the control group was 3.802.

In addition to improving quality and efficiency, ChatGPT also reduces the difference in the quality of work between different people.

According to the completion of the first task, the researchers linearly fitted the scores of these people for the second task. The results showed that the fitting slope of the experimental group was 0.414, which was 0.272 lower than that of the control group.

So how do these people use ChatGPT? Do you want to use the output directly or just fine-tune it, or just make a draft?

The tracking results showed that 33% of the subjects submitted authentic ChatGPT results, while 53% modified them. But these changes may be minor, such as just adjusting the format, and so on.

The reason for this conclusion is that their average activity after ChatGPT generated responses is only 3.3 minutes, and most of them are between 0 and 2 minutes.

It is important to note that before the start of the experiment, 70% of the subjects had heard of the use of ChatGPT,32%. So, in the course of the experiment, the researchers found that 10-20% of the people in the control group also used ChatGPT.

This phenomenon has a certain impact on the experimental results, and the researchers used statistical methods to correct it.

At the end of the experiment, the researchers also conducted two return visits to the subjects, and 92% and 83% respectively responded.

The first return visit was two weeks after the end of the experiment. 34% of the people in the experimental group and 18% in the control group were still using ChatGPT. Six weeks later, two months after the end of the experiment, the numbers rose to 42% and 27%, respectively.

To sum up, the researchers believe that ChatGPT does play a significant role in improving the quality and efficiency of work, but it is not clear whether it can replace humans.

The author briefly introduced that the study was jointly completed by Whitney Zhang, a doctoral student in Chinese economics at MIT, and Shakked Noy, a classmate.

Zhang has been studying at MIT since 2017, and his current research areas include branches of economics such as behavioral economics. Zhang is also the mentor of the support program for beginners at Boston Chinese Community College.

Noy, from New Zealand, joined MIT in 2022 to study for a PhD in economics.

Paper address:

Https://www.science.org/doi/10.1126/science.adh2586

Reference link:

[1] https://news.mit.edu/2023/study-finds-chatgpt-boosts-worker-productivity-writing-0714

[2] https://economics.mit.edu/people/phd-students/whitney-zhang

[3] https://economics.mit.edu/people/phd-students/shakked-noy

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.