Shulou(Shulou.com)11/24 Report--

CTOnews.com May 2 message, most AI chatbots currently need to be connected to the cloud for processing, and even those that can be run locally are extremely demanding to configure. So are there lightweight chatbots that don't need to be connected to the Internet?

A new open source project called MLC LLM has been launched in GitHub. It runs completely locally without networking, and even Jixian old computers and Apple iPhone phones can run.

According to the MLC LLM project introduction, "MLC LLM is a general-purpose solution that allows any language model to be deployed locally on a different set of hardware backends and native applications, as well as an efficient framework for everyone to further optimize the model performance of their own use cases. Everything runs locally, without server support, and is accelerated by local GPU on mobile phones and laptops. Our mission is to enable everyone to develop, optimize, and deploy AI models locally on the device. "

The official demonstration of the ▲ MLC LLM project, CTOnews.com query GitHub page, found that the developers of the project came from the Catalyst project of Carnegie Mellon University, the SAMPL machine learning research group, as well as the University of Washington, Shanghai Jiaotong University and OctoML. They also have a related project called Web LLM, which runs AI chatbots entirely in web browsers.

▲ MLC is the abbreviation of Machine Learning Compilation. MLC LLM uses Vicuna-7B-V1.1, which is a lightweight LLM based on Meta LLaMA. Although it is not as effective as GPT3.5 or GPT4, it has an advantage in size.

Currently, MLC LLM is available for Windows, Linux, macOS, and iOS platforms, and there is currently no version for Android.

According to the test of foreign media tomshardware, Apple iPhone 14 Pro Max and iPhone 12 Pro Max phones with 6GB memory successfully run MLC LLM with the installation size of 3GB. Apple iPhone 11 Pro Max with 4GB memory does not run MLC LLM.

▲ Picture Source tomshardware in addition, the ThinkPad X1 Carbon (Gen 6) also tested MLC LLM, a laptop with i7-8550U processor, no independent graphics card, and Intel UHD 620 GPU. MLC LLM needs to be run from the command line on the PC platform, and the performance of foreign media tests is mediocre, with a response time of nearly 30 seconds and almost no continuous conversation ability. I hope it can be improved in subsequent versions.

▲ drawing source GitHub page of tomshardwareMLC LLM: click here to view

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.