Shulou(Shulou.com)11/24 Report--

CTOnews.com, September 27 (Xinhua) when DeepMind researchers evaluated the compression ability of large language models (LLM) a few days ago, they found that the "compression ability" of these models is quite amazing. In addition to regular text data, they can also compress images and audio, which has been posted on ArXiv.

It is reported that in the study, DeepMind used a model called "Chinchilla 70B". Although this model mainly uses text training, the researchers found that the model can also be used to compress ImageNet images, compressing files to 43.3% of their original size, and even compressing LibriSpeech voice samples to 16.4% of their original size.

DeepMind's research proves that there is an "equivalence" between the "prediction" ability and the "compression" ability of the model, so researchers can use any compression algorithm to build a more powerful conditional generation model.

CTOnews.com Note: "Compression" is essentially a coding process, the goal is to represent more data with less content, so when the model reaches a certain predictive ability, it actually means that the model has learned a way of coding, which can be used to compress files, because the model already understands the characteristics and patterns in the corresponding files (that is, if a model can achieve accurate prediction. It will also be able to capture the essential characteristics and structure of the file, thus effectively compressing the data file.

DeepMind believes that at a time when the current language model is fruitful, anyone can take the language model and use it in compression without additional training costs.

At the same time, the research also shows that even the "basic model which mainly uses words for training" can become a "general-purpose compressor" because of its context learning ability.

The study also found that if the model is to be used for compression, the larger the model, the better, and the larger the model may have a negative impact on the compression capacity, because the parameters of the model itself need to be taken into account in the output. when there is a model with very many parameters, although it can effectively compress data, the huge parameters themselves will become a burden. And the Tokenization method commonly used in natural language processing (cutting a string of words into smaller and easier steps) will not improve the compression efficiency at the compression level, but will increase the volume of the model.

Referenc

Language Modeling Is Compression

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.