Shulou(Shulou.com)06/03 Report--

As far as I am concerned, as soon as I enter the media circle, my career will encounter a very serious AI threat.

Because GPT-3 is here, and he is better at writing articles and making up stories than the previous generation.

In May this year, OpenAI, which has been enhanced by Microsoft's Azure computing power, released GPT-3, a giant NLP model monster with 175 billion parameters, which is 10 times larger than Turing NLG, the world's largest deep learning model just launched by Microsoft in February, and 100 times the parameters of its predecessor GPT-2.

We can use a chart to intuitively feel the position of GPT-3, isn't it a bit cold at the top?

At the same time, the training data set used by GPT-3 is also very large, based on nearly 1 trillion words of CommonCrawl data sets, network text, data, Wikipedia and other data, the amount of data has reached 45TB. The cost of training has also reached a staggering $12 million, which is already a training scale and cost that individual developers and small AI development teams cannot easily get their hands on.

In a large number of recent introduction articles about GPT-3, many people have noticed the amazing size of the model and the text generation ability of all kinds of ideas, not only writing articles, making up stories, doing translation, but also multiple rounds of dialogue, writing code, doing mathematical operations, matching texts, making tables, generating icons, etc., almost doing whatever they want in the text.

Some people exclaim that "the real AI has come" and "GPT-3 can change the world", while others say that "GPT-3 is an image project" and "a naked show of wealth".

No matter what the evaluation, people don't pay much attention to the fact that one of the main reasons why OpenAI is now releasing GPT-3 's API interface is to promote the commercialization of this technology. Now, among the fields in which the GPT-3 model has been widely used, which areas are better to try to commercialize, and which areas are still unsatisfactory, these may be more worthy of our discussion.

How good is GPT-3?

Compared with the previous GPT-2, what are the obvious improvements in this GPT-3?

As far as the training method is concerned, GPT-3 is no different from the previous version, but the training data and parameters have been improved by several orders of magnitude. But from the actual effect, GPT-3 's attempt at least validates one thing, that is, by increasing a deep neural network, it can indeed become smarter.

Compared with the current BERT model, GPT-3 can mainly solve two problems, one is to avoid over-reliance on labeled data in various fields, and the other is to avoid over-fitting of data distribution in various fields, so as to train a more general and generalized NLP model. The main goal of GPT-3 is to use less domain data and to remove fine-tuning steps to solve problems.

(photo Source: Li Hongyi's "Deep Learning of Human language processing")

To understand intuitively, as shown in the figure, GPT-3 is to remove the Fine-tune (fine-tuning) this link, and also get the sample materials of Task-Specific to directly answer specific domain questions.

Based on this, the researchers use GPT-3 to test the reasoning effect in different forms, including Zero-shot, One-shot and Few-shot, but these three forms do not need to go through Fine-tuning. Because GPT-3 chooses one-way transformer, it encodes the previous token when it predicts a new examples.

So, what are the test results?

Judging from the average performance of 42 benchmarks in various fields, as the number of parameters continues to increase, the correct rate continues to improve (of course, some people will question that the model has increased by 10 times the number of parameters, so the correct rate has increased less than twice as much), of which Few Shot is the best.

In the closed Trivia QA Q & A, the performance of GPT-3 's Few-Shot has been better than that of Fine-tuned SOTA. In addition, the SuperGLUE test can also achieve better performance than the current SOTA, and generate very realistic articles, even to the extent that it is difficult for humans to tell whether it is a machine or a human collaboration.

So, after people currently call OpenAI's open API interface, we can already see a series of interesting cases of GPT-3.

GPT-3 is now able to perform excellent translation, question and answer and cloze tasks, and can perform two-or three-digit mathematical addition and subtraction operations. You can also generate code and websites based on text descriptions.

(GPT-3 uses natural language to generate code and graphic buttons.)

Different styles can be changed for the text, such as turning colloquial into written language and everyday language into legal documents. Or turn the thriving legal language into everyday language, such as those long "user agreements".

(GPT-3 translates everyday language into legal documents)

Of course, the main business of GPT-3 is to generate text content, such as jokes, news, novels, and even give themes and key words, can be modeled to produce a complete paper.

(GPT-3 completed the paper just by giving the title and beginning.)

GPT-3 has also performed quite well in many rounds of conversations with humans. For example, the following programmer named Kosmopol and GPT-3 have a "mysterious" discussion about the existential relationship between human beings, AI and God.

(at the end of the conversation, the programmer said, "I don't have any questions now.")

Judging from the performance of GPT-3 published on the Internet, GPT-3 seems to be able to play a role in any field related to text generation.

So what is the future of GPT-3 in commercialization?

What are the commercial prospects of GPT-3?

We remember that when GPT-2 was released, OpenAI was not willing to release the GPT2 model in its entirety at once, but chose to release the full version bit by bit like toothpaste, on the grounds that GPT- 2 was too dangerous and would be used to create fake news, e-mail fraud and other bad things. Of course, the terrible consequence did not happen, perhaps the technical ability of the bad guys is not enough, the main reason is that the cost threshold of the application is too high.

This time, OpenAI chose to publish the API interface to invite testing, rather than a direct open source model, and also had this consideration. If the model is open source, once someone develops dangerous applications on this basis, it will be difficult for officials to stop. People's abuse of technology can be well dealt with through API.

At the same time, because GPT-3 is such a large basic model, few institutions and individuals can develop and deploy it, except for a few large companies, and it will be extremely expensive to run.

In fact, a more important point is that OpenAI hopes to promote the commercialization of GPT-3 technology through API, and develop relevant AI products on the basis of safety, reliability and policy compliance in the future, and achieve commercial profits.

According to the current OpenAI, preliminary commercial tests have been carried out with more than a dozen companies before providing API. In terms of specific open functions, GPT-3 can be used commercially in semantic search, chat robots, productivity tools, text generation, content understanding, machine translation and so on.

For example, Algolia, a start-up, is using GPT-3 to conduct complex natural language searches, as shown in its ability to reduce prediction time to about 100ms and answer complex natural language questions four times faster than BERT.

In terms of productivity tools, GPT-3 's API can provide more diversified functions, such as breaking text into charts, tables, and e-mail summaries, and extending content from project points. For programming work, programmers can talk to computers through natural language, and they can get the basic code they want without having to remember all kinds of complex commands.

In addition, commercial applications can be realized in aspects such as spelling suggestions in document writing, grammar error correction, case indexing in related work of legal institutions and law firms, legal research, pattern-based litigation application writing, auxiliary search and examples of teaching materials in educational and teaching institutions, and chatbots in online customer service.

From this point of view, it seems that the emergence of GPT-3 not only makes media editors (Microsoft fired a group of manual editors not long ago) directly encounter a career crisis, but even seems to be in danger of being laid off by basic clerks, online customer service, and even programmers in many organizations.

However, judging from the examples publicly displayed by GPT-3 so far, there is still some reason for such concern. Directly speaking, as a productivity tool of enterprises, GPT-3 will play a more auxiliary role in improving efficiency. GPT-3 can be used as an auxiliary tool for any need for text generation, data retrieval, and enlightening content production.

For example, writers can use keywords to get creative ideas provided by GPT-3 for inspiration. Company staff and agency clerks can use meeting minutes to generate professional reports, e-mails and professional documents.

In this process, it is impossible for us to use and publish directly without human review and revision. Obviously, no organization or individual will let the AI model take responsibility for publishing its content. Of course, when some people are better able to collaborate more efficiently with artificial intelligence tools such as GPT-3 to improve the productivity of business organizations, it will be followed by a reduction in the number of basic jobs. In this sense, the role of GPT-3 as a job killer will appear indirectly.

But is the current GPT-3 up to the task? Judging from the feedback from some developers after testing and the comments of some experts, GPT-3 is still a long way from being truly commercialized, and some of these problems must be solved.

The difficult problem of commercialization of GPT-3

While there was more praise for GPT-3 's performance, OpenAI co-founder Sam Altman stepped forward on Twitter and said, "GPT-3 is being touted too much."

In fact, this statement is indeed very practical and realistic. At present, GPT-3 performs well in common sense Q & An and factual text production, but once it is in the Q & An of counterfactual or contradictory questions, GPT-3 will show an infantile tendency to "pretend to understand".

For example, in the above counterfactual questions or meaningless language repetition, GPT-3 turns on the "chat" mode. In the words of Julian Togelius, an associate professor at New York University, "GPT-3 often behaves like a smart student who hasn't finished reading, babbling and trying to muddle through on exams. Some well-known facts, some half-truths, and some direct lies look like smooth narratives at first glance."

GPT-3 can also make some biased low-level errors on some output. For example, when someone was talking to the virtual Jobs via GPT-3, when asked where Master Qiao was now, GPT-3 gave the answer as "Apple headquarters" and gave a place name. However, everyone knows that this answer is not correct, and the answer is that Master Qiao is now living in our hearts and is more reliable than the answers retrieved above.

In addition, OpenAI is more cautious about the output of GPT-3 biased content because GPT-2 has a precedent for generating an article with discriminatory descriptions that offend black women. This may stem from all kinds of discrimination contained in the training data itself. However, if you take into account the elimination of these contents in the collation of data, it requires a lot of labor costs, which is actually neither operational nor necessary. In the end, we can only optimize and improve the results of GPT-3 on the output side.

These low-level, misleading and biased mistakes will still make the business application of enterprises have a lot of worries. Once fully handed over to AI to perform work communication, customer service and other work, it will inevitably cause the loss of interests of the enterprise, or increase operating costs such as audit.

A more important difficulty in commercializing GPT-3 is the ratio of performance to price. If some of the automated text generation tasks that can be achieved by GPT-3 can be done by cheaper but more professional AI software, then the commercial value of GPT-3 will be greatly reduced. That is, if people try to replace Google's keyword search with GPT-3, but can't get more comprehensive information, why not go back to Google and Wikipedia for free?

Although the versatility of GPT-3 shows the good characteristic of "doing miracles", there is still a long way to go for OPenAI to exert its commercial value more effectively, which requires "slow workers to do fine work" in the following model optimization.

Now, after accepting a $1 billion investment from Microsoft, the commercialization of OpenAI has raised a more urgent agenda. The AI model, which relies on huge computing resources to run, must be commercialized.

Therefore, GPT-3 bears the brunt.

Generally speaking, the commercial opening of GPT-3 has a very positive significance. As such a huge model training is generally small businesses and individual users simply can not afford, then the opening of API can enable these users to pay for the use of AI functions at less cost. However, from another point of view, the scientific research monopoly in the field of AI is also forming. At that time, the giant who formed a monopoly in the operating system, search engine and other fields now occupies the basic mining right of data-rich mines in the field of AI through computing hegemony.

We see that the initial commercialization of GPT- 3 will not be so smooth. However, such a basic project, regardless of its own results, the technical experience and technical capabilities acquired by OpenAI in the process of completing the project is actually a more important asset. The point is that GPT-3 is still the right way for AI to move forward.

At that time, the United States spent countless manpower and wealth on the Apollo moon landing program, and the result was only to win the first prize in Star Wars with the Soviet Union at that time. However, some of the by-products of these massive projects, such as space communications, material science, automatic control, integrated circuits, and computer science, have so far benefited American technology and commerce.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.