In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > IT Information >
Share
Shulou(Shulou.com)11/24 Report--
Because open source, AI arms race, Google and OpenAI are all losers?
Heavy!
An internal Google document titled "We don't have a moat and OpenAI doesn't" seems to have been leaked.
This morning, foreign media SemiAnalysis released this blockbuster leaked document.
It is reported that the document was shared by an anonymous person on the Discord server, the document from an internal Google researcher, the authenticity has been confirmed.
SemiAnalysis specifically stressed that this document only represents the opinions of Google employees, not those of the company as a whole.
Internal files, we don't have a moat, and neither does OpenAI.
We've been keeping a close eye on OpenAI. Who will cross the next milestone? What will be the next step?
But now, the disturbing truth is that we cannot win this arms race, and neither can OpenAI.
While our two sides are at war, the third party is quietly eating the benefits that belong to us.
Yes, I'm talking about open source. To put it bluntly, they have overtaken us. What we think of as the "major open problem" has now been solved and is in the hands of all users. A few simple examples:
LLMs on the phone: on the Pixel 6, you can run the basic model at a speed of 5 token per second.
Scalable personal AI: you can fine-tune a personalized AI on your laptop in just one night.
Responsible release: this point is not "solved", it would be more appropriate to say "avoid". Now the Internet is full of websites full of various art models, without any restrictions, and the open source big language model is not far behind.
Multimodal: the current multimodal ScienceQA SOTA can be trained in only an hour.
Although our model still has a slight advantage in quality, the gap is narrowing at an alarming rate.
These open source models are faster, more customizable, more private, and more powerful.
They can do what we can do with $10 million and 540B with only $100B and 13B parameters. They finished it in weeks, not months.
The quality of Vicuna-13B reaches 90% of OpenAI ChatGPT and Google Bard *, which has a huge impact on us:
We don't have an exclusive secret weapon. The best hope is to learn what other people are doing and work with them. We should give priority to allowing third-party integration.
When these free, unrestricted open source alternatives are of exactly the same quality, users will no longer pay for restricted models. We should consider where our real value-added lies.
Giant models are slowing us down. In the long run, the best models are those that can iterate quickly. Now that we know what the model will be like when the parameters are less than 20 billion, we should pay more attention to the small model.
What happened to https://lmsys.org/blog/2023-03-30-vicuna/ in early March, when Meta's LLaMA was leaked to the public, the open source community got the first truly powerful basic model. It has no instructions or dialog adjustments, nor does it have a RLHF.
Nonetheless, the open source community immediately understood how important what they got was.
Subsequently, a large number of innovative open source replacement models continue to emerge. Every few days, there is a great progress.
In just one month, there are variants of instruction adjustment, quantification, quality improvement, manual evaluation, multimodal, and RLHF, many of which are based on each other.
Most importantly, they have solved the problem of scale, and now anyone can participate in it.
Nowadays, many new ideas come from ordinary people. The threshold for training and experimentation has been greatly lowered. Once upon a time, a large research institution was needed to work together. Now, all it takes is a powerful laptop that can be done by one person in one night.
We could have seen it coming. It wasn't a surprise to anyone. The renaissance in the field of image generation was followed by a renaissance in open source LLM.
Many people say that this is the "Stable Diffusion" moment of the big language model.
In both areas, public participation at low cost is achieved through low-rank adaptation (LoRA). It greatly reduces the cost of the fine-tuning mechanism.
A major breakthrough in the scale of the model has also been achieved. (for example, Chinchilla of image synthesis Latent Diffusion,LLM)
After obtaining a sufficiently high-quality model, individuals and institutions around the world have begun a series of innovation and iteration of the model. And these innovations quickly surpassed the big technology companies.
In the field of image generation, these contributions are crucial and put Stable Diffusion on a completely different path from Dall-E.
The open source of Stable Diffuision has led to product integration, marketing, and user interface innovation, but in Dall-E, this has not happened.
The consequences of this are obvious, Stable Diffusion quickly took the mainstream, compared with OpenAI's solution has become irrelevant.
Will the same thing happen in the LLM world? It is still unknown, but there are too many similarities between the two things.
What did we miss? Many of the recent successful innovations in the open source community have directly solved many of the problems that we have not yet solved.
Paying more attention to their work can help us avoid rebuilding wheels.
LoRA is a very powerful technology, and we should probably pay more attention to it.
Https://arxiv.org/ pdf / 2106.09685.pdfLoRA works by expressing the model update as a low-rank decomposition, which reduces the size of the update matrix by thousands of times.
This greatly reduces the time and cost of fine-tuning the model.
It would be significant if a personalized language model could be fine-tuned on consumer hardware within a few hours. In particular, it can also integrate a lot of up-to-date and diversified knowledge in real time.
But this technology has not received enough attention within Google, although it has a direct impact on our most hopeful projects.
Retraining the model from scratch is a difficult path. Part of the reason LoRA is so effective is that, like other forms of fine-tuning, it is stackable.
You can apply instructions to adjust the improved model so that it can be used directly when other contributors add conversations, reasoning, or tools.
Although individual fine-tuning is low-rank, their summation is not needed, so full-rank updates to the model can be accumulated over time.
This means that as long as new and better datasets and tasks emerge, the model can be kept up-to-date at a low cost without paying the cost of a full operation.
By contrast, training giant models from scratch loses not only the process of pre-training, but also any iterative improvements made at the top.
In the open source world, these improvements will soon dominate, making it extremely expensive to fully retrain the model.
We should consider whether every new application or idea really needs a brand new model.
If we really have significant architectural improvements that make it impossible to reuse model weights directly, then we should invest in more active forms of distillation to retain as much functionality as possible from the previous generation of models.
If we can iterate quickly on small models, then in the long run, big models are not strong enough to be omnipotent.
LoRA (low Rank adaptation of large language models) is a novel technology proposed by Microsoft, which aims to solve the problem of fine tuning large language models.
Its update is very cheap for the most popular model size (about $100), which means that almost anyone with an idea can generate one and distribute it.
In the future, it is a normal thing to train a model in one day.
At this rate, it won't be long before the cumulative effect of these fine-tuning will soon make up for the disadvantage of the original model size.
In fact, these models are improving much faster than we can do with the largest model, and the best model is largely indistinguishable from ChatGPT.
Focusing on some big models puts us at a disadvantage.
Data quality, not data scale. Many projects save time by training on small, selected data sets. This shows that the law of data expansion has a certain flexibility.
The existence of such data sets stems from the idea in the article "Data Doesn't Do What You Think", and they are rapidly becoming the standard way to train outside Google.
These datasets are built by compositing methods (for example, filtering the best responses from existing models) and collecting from other projects. Google is not dominant in either.
Fortunately, these high-quality data sets are open source, so they can be used for free.
Direct competition with open source is a failed proposition. New developments in AI have a direct and immediate impact on Google's business strategy. If there is a free, high-quality, unlimited alternative, who will pay for Google's products?
And we should not expect to catch up. There is a reason why the modern Internet relies on open source. Open source has some significant advantages that we cannot replicate.
We need them more than they need us. The secrecy of our technology has always been a fragile proposition.
Google researchers are leaving regularly for other companies. So we can assume that they know everything we know. And as long as the channel is open, they will continue to do so.
However, due to the low cost of cutting-edge research on LLM, it is more difficult to maintain the competitive advantage in the field of technology.
Research institutions around the world are learning from each other to explore solution space far beyond our own capabilities in a breadth-first manner.
We can try to hold on to our secrets, and external innovation weakens its value, or we can try to learn from each other.
Compared with companies, individuals are less restricted by licensing, and most of the model innovation is carried out after the weight of Meta's LLaMA model is leaked.
While this will certainly change as the real open source model gets better, the key is that they don't have to wait.
The legal protection provided by "personal use" and the impracticality of prosecuting individuals mean that individuals have access to these technologies when they are hot.
Being your own customer means that you understand that use case browsing models created by people in the field of image generation, from animators to HDR landscapes, creativity continues to emerge.
These models are used and created by people who go deep into specific subtypes, giving us a depth of knowledge and empathy beyond our reach.
Owning an ecosystem: the paradox of letting open source work for us is that behind the competition among big companies, the winner is Meta.
Because the leaked model LLaMA is theirs, it is equivalent to their effective access to free labor of the value of the entire planet.
Since most open source innovations are based on LLaMA, there is nothing to stop them from incorporating them directly into their products.
With the value of the ecosystem, the future will be inestimable. Google has successfully used this paradigm in its open source products such as Chrome and Android.
By having a platform for innovation, Google has consolidated its position as a thought leader and direction setter.
The tighter we control the model, the more attractive open source alternatives become.
Both Google and OpenAI tend to strictly control the use of models, initiating a defensive response.
But this control is fictional, because anyone who tries to use LLMs for unauthorized purposes can choose a freely available model.
Google should establish itself as a leader in the open source community and play a leading role through cooperation.
This may mean taking some disturbing steps, such as publishing model weights for small ULM variants. This must mean giving up some control over our model.
But such a compromise is inevitable. We cannot promote and control innovation at the same time.
Conclusion: what about OpenAI? Given OpenAI's current closed policy, all the talk about open source may seem unfair.
If they don't want to, why should we share? But the truth is, we are sharing everything with them through a steady stream of senior researchers being poached.
There is no point in keeping secrets until we stop this trend.
Finally, OpenAI is not important.
They are making the same mistake as open source, and their ability to maintain an edge is bound to be called into question.
Unless they change their position, open source alternatives can and will eventually surpass them. At least in this respect, we can take the lead.
Open source timeline 23 February 24, LLAMA released Meta released LLaMA, open source code, but did not publish the weight. At this point, LLaMA has not yet done instruction or dialog tuning.
Like many current models, it is a relatively small model (with parameters of 7B, 13B, 33B and 65B, respectively). After relatively long training, it has a strong ability compared with its size.
Less than a week after the inevitable happened on March 3, 23, LLAMA was leaked to the public. Meta's existing license prohibits the use of LLAMA for commercial purposes.
All of a sudden, anyone can experiment. In the whole community, there is a tsunami of model innovation.
On March 12, 23, after running the language model on the oven for more than a week, Artem Andreenko successfully ran the model on raspberry pie. At that time, the model was very slow because the weights had to be paged in memory, which was not very practical.
Nevertheless, this lays the foundation for a series of efforts to reduce the size of the model.
On March 13, 23, the day after fine-tuning on laptops, Stanford released Alpaca, which added instruction tuning to LLaMA.
Important, however, is Eric Wang's alpaca-lora repository, which uses LoRA to do this training in a few hours on a single RTX 4090.
From then on, all of a sudden, anyone can fine-tune the model, sparking a competition for low-cost fine-tuning models.
There have been numerous reports that the xxx model cost a total of several hundred dollars.
More importantly, low-rank updates can be easily distributed separately from the original weights, freeing them from the constraints of Meta's original license. Anyone can share and apply them.
March 18, 23, became faster when GeorgiGerganov used 4-bit quantization to run LLaMA on MacBookCPU.
This is the first "no GPU" solution, fast enough and practical.
On March 19, 23, a model 13B achieved a "balance" with Bard. The next day, an interuniversity collaboration released Vicuna and used GPT-4-driven evaluations to qualitatively compare the model output. Although the evaluation method is questionable, the model is essentially better than the earlier variants.
Most importantly, only $300 was spent on training.
It is worth noting that they are able to use data from ChatGPT while circumventing their API restrictions
All they have to do is get impressive samples of ChatGPT conversations from sites like ShareGPT.
On March 25, 23, he chose his own model Nomic to create GPT4All, which is both a model and, more importantly, an ecosystem.
For the first time, everyone saw the models (including Vicuna) converging in one place. Training cost: $100.
On March 28, 23, the open source version of GPT-3Cerebras used the best computing plan implied by Chinchilla and the best scaling (optimal scaling) implied by μ parameterization to train the GPT-3 architecture.
This has a great advantage over the existing GPT-3 clones and represents the first use of μ parameterization in practical applications. These models are trained from scratch, which means that the community is no longer dependent on LLaMA.
On March 28, 23, the one-hour multimodal training LLaMA-Adapter adopted a new parameter effective fine-tuning (PEFT) technique, introducing instruction tuning and multimode into the one-hour training.
Impressively, they use only 1.2 million learnable parameters. The model refreshes SOTA on multimodal ScienceQA.
On April 3, 23, people couldn't tell the difference between the 13B open source model and ChatGPT Berkeley's release of Koala, a conversation model trained entirely with free data.
They took the key step of measuring real human preferences between Koala and ChatGPT.
Although ChatGPT still has the upper hand, users either prefer Koala or don't care more than 50% of the time. Training cost: $100.
On April 15, 23, ChatGPT-level open source RLHFOpen Assistant released a model and, more importantly, a dataset for alignment through RLHF.
This model is close to ChatGPT in terms of human preferences (48.3% 51.7%).
In addition to LLaMA, they also showed that this dataset can be applied to Pythia-12B, providing an option for people to run the model using a fully open stack.
In addition, because the dataset is publicly available, it makes RLHF from unachievable cheap and easy for small experimenters.
Google built a wall and slapped open source in the face, although open source is a victory, but now Google has put up a wall and refused to open source.
In February, Jeff Dean, long-time head of Google's artificial intelligence division, announced a shocking policy shift:
Postpone sharing internal work with the outside world.
For many years, Dean has managed the department as a university, encouraging researchers to publish a large number of academic papers. According to Google Research, they have promoted nearly 500 studies since 2019.
Since the birth of ChatGPT, all the way to unboiled water, obviously let Google panic for a moment, and this must be changed.
Dean said that Google's findings in the field of artificial intelligence will not be shared until they are translated into products.
Jeff Dean said at the quarterly meeting of Google's research department that the San Francisco-based startup OpenAI kept pace with Google through the papers of its learning team.
As we all know, the T in ChatGPT refers to Google's Transformer architecture, which is a large language model with Transformer architecture as its core.
Paper: https://arxiv.org/ pdf / 1706.03762.pdf Google opened the defense mode in order to catch up with ChatGPT. This is a major shift for Google.
This policy is first to defend against a group of strong AI competitors, but also to protect its core search business, as well as its possible future.
However, as internal document leaks say, Google is not the winner, nor is OpenAI, the real winner is Meta.
Google has tasted the victory of open source. Now, it's time to make some changes.
Reference:
Https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
Https://www.washingtonpost.com/technology/2023/05/04/google-ai-stop-sharing-research/?utm_source=reddit.com
This article comes from the official account of Wechat: Xin Zhiyuan (ID:AI_era)
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 234
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.