In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-04-03 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > IT Information >
Share
Shulou(Shulou.com)11/24 Report--
CTOnews.com June 27 news, Microsoft researchers have launched a new technology called ZeRO++, used to optimize the training of large AI models, easy to encounter data transmission costs and bandwidth constraints, can significantly reduce the training time and cost of large models.
ZeRO++ builds on existing ZeRO transmission technology and provides enhanced communication strategies that improve training efficiency while reducing training time and costs.
In order to reduce parameter traffic, ZeRO++ quantizes the weights, using a block-based quantization method to maintain training accuracy, which is faster and more accurate than the original Zero transmission technology. To minimize communication overhead, ZeRO++ trades GPU memory for communication bandwidth by maintaining a complete copy of the model on each machine. In gradient communication, ZeRO++ introduces a new quantized gradient communication method called qgZ, which can reduce cross-node traffic and delay.
These improved communication technologies have greatly reduced traffic, and Microsoft researchers say ZeRO++ reduces traffic by up to four times compared to ZeRO, improving training throughput and efficiency. When small batch sizes are used on each GPU, ZeRO++ achieves throughput improvements of 28 to 36 percent over ZeRO-3 in high-bandwidth clusters. In low-bandwidth clusters, ZeRO++ achieves an average of 2x speedup compared to ZeRO-3, making large model training more feasible on a wider variety of clusters.
CTOnews.com Note: CTOnews.com notes that large models such as Turing-NLG, ChatGPT, and GPT-4 require significant memory and compute resources to train across multiple GPU devices, while ZeRO++ introduces communication optimization strategies to overcome bandwidth limitations of the original ZeRO transport technology when trained on low-bandwidth clusters. Microsoft has released relevant technical documentation, and researchers can use ZeRO++ to train models more effectively and explore new possibilities in the field of AI.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.