Shulou(Shulou.com)12/24 Report--

Fully compatible with Stable Diffusion ecosystem, LCM model successfully achieved 5-10 times faster generation speed, real-time AI art era is coming, what you want is what you get!

Latent Consistency Models is an image generation architecture that focuses on speed of generation.

Unlike traditional diffusion models that require multiple iterations (such as Stable Diffusion), LCM can achieve the effect of traditional models in only 1 - 4 steps.

Invented by Luo Simian and Tan Yiqin, graduate students of Tsinghua University's Cross-information Research Institute, LCM has increased the generation speed of Wensheng Map by 5-10 times, and the world has since entered the era of real-time generative AI.

LCM-LoRA: https://huggingface.co/papers/2311.05556

Project homepage: latent-consistency-models.github.io/

Stable Diffusion Killer: Prior to LCM, various teams explored a wide variety of SD1.5 and SDXL alternatives in various directions.

Each of these projects has its own characteristics, but there are hard flaws that are incompatible with LoRA and not fully compatible with the Stable Diffusion ecosystem. In chronological order, the most important projects are:

At this time, LCM-LoRA appeared: SD1.5, SSD1B, SDXL distillation into LCM LoRA, will generate 5 times faster generation capacity on all SDXL models and compatible with all existing LoRA, while sacrificing a small part of the quality of generation; the project quickly obtained Stable Diffusion ecosystem A large number of plugins, distribution support.

LCM has also released training scripts that support training its own LCM large models (such as LCM-SDXL) or LCM-LoRA, giving consideration to both quality and speed. With just one training session, you can speed up by five times while maintaining the quality of production.

At this point, the LCM ecosystem has a complete replacement for SD rudiments.

As of November 22, 2023, open source projects that support LCM:

Stable Diffusion Release

WebUI (native support LCM-LoRA, LCM plugin support LCM-SDXL), ComfyUI, Foocus (LCM-LoRA), DrawThings

small model

LCM-LoRA compatible with other LoRAs, ControlNet

AnimateDiff WebUI Plugin

Projects supported in the plan:

WebUI main support

Training Script Kohya SS

ControlNet for LCM-SDXL and LCM-DreamShaper

LCM-AnimateDiff

As the ecosystem evolves, LCM has the potential to replace Stable Diffusion completely as a next-generation image generation substrate.

Future Outlook Since the release of Stable Diffusion, generation costs have been slowly optimized, while the emergence of LCM has directly reduced image generation costs by an order of magnitude. Every time a revolutionary technology comes along, there are plenty of opportunities to reshape industry. LCM can bring significant changes to the industrial pattern in at least three aspects: image generation cost disappearance, video generation and real-time generation.

1. Image generation costs disappear

To C product end, free replacement fee. Due to the high GPU computing cost, a large number of graphic services represented by Midjourney choose freemium as their business model. LCM makes it possible for mobile phone clients, PC CPUs, browsers (WebAssembly), and more flexible CPU computing power to meet the computing power needs of image generation in the future. Simple, fee-based services such as Midjourney are replaced by high-quality free services.

To B server, the reduced generation computing power demand will be replaced by the increased training computing power demand.

AI image-generation service demand for computing power fluctuates greatly between peaks and valleys, and the idle time of the purchase server usually exceeds 50%. This feature has promoted the vigorous development of a large number of functional computing GPUs (serverless GPUs) such as Replicate in the United States and Alibaba Cloud in China.

In terms of hardware virtualization, such as Ruiyun and Tencent Cloud in China, virtual desktop products related to image model training have also been launched in Inspur. With the generation of computing power decentralized to the edge, client or CPU computing power that is easier to expand, AI graphics will be popularized in various application scenarios, and the demand for fine tuning of image models will rise sharply. In the image domain, professional, easy-to-use, vertical model training services will become the main consumers of cloud GPU computing power in the next stage.

2. Wensheng Video

At present, the extremely high production cost of Wensheng video restricts the development and popularization of technology, and consumer graphics cards can only render frame by frame at a slow speed. A batch of projects represented by AnimateDiff WebUI plugin support LCM first, so that more people can participate in the open source projects of Wensheng Video. A lower threshold will inevitably accelerate the popularity and development of literary videos.

3 min fast rendering: AnimateDiff Vid2Vid + LCM3. real-time rendering

The increase in speed has spawned a plethora of new apps that expand everyone's imagination.

RT-LCM and AR

With RealTime LCM as the forerunner, real-time video generation at about 10 frames per second is realized on consumer-grade GPUs for the first time, which is bound to have a profound impact in the AR field.

At present, high-definition and low-delay capture and redrawing of the whole scene in the line of sight requires extremely high computing power, so in the past AR applications mainly focused on adding new objects and redrawing some objects in low resolution after extracting features. LCM makes it possible to redraw entire scenes in real time, with unlimited imagination in games, interactive movies, social and more.

Future game scenes don't need to be newly created, wear AR glasses, and the streets you are in will immediately transform into neon flashing cyberpunk futuristic styles for players to explore; wear AR glasses when watching future interactive horror movies, everything familiar at home can be seamlessly integrated into the scene, and scary things are hidden behind the bedroom door. Virtual and real will merge seamlessly, and reality and dream will become increasingly difficult to distinguish. And all of this at the bottom there may be LCM figure.

RT-LCM Video Rendering Interaction-What you imagine is what you get

The real-time image editing UI, first commercialized by Krea.ai and ilumine.ai, once again lowers the threshold of creation, expands the boundaries of creativity, and allows more people to get real-time feedback on the final painting on the basis of fine control.

Krea.ai Real-time image editing

Real-time image editing modeling software + LCM explores new directions in 3D modeling, allowing 3D modelers to go beyond WYSIWYG and gain WYWYG capabilities.

LCM Real-Time Spatial Modeling Rendering The hand is the most useless thing for humans, because the hand can never keep up with the speed of the brain. What you see is what you get is too slow, and What you imagine is what you get will become the mainstream of future creative work.

For the first time, LCM allowed the presentation to keep pace with the speed at which ideas were generated. New ways of interacting continue to emerge, and the end point of the AIGC revolution is to lower the cost and technical barriers to creativity to infinitely close to zero. Across industries, good ideas will go from scarce to surplus. LCM takes us one step further into the future.

References:

https://latent-consistency-models.github.io/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.