Shulou(Shulou.com)06/02 Report--

[IT168 comment] Today, big data and artificial intelligence have become very popular. Big data platforms like Hadoop and Spark have almost become the most common platforms for storing, processing and analyzing large-scale data in the industry. However, the data science teams of many organizations and enterprises still have to face many system and platform challenges in the actual development process for AI.

Intel has a wide layout in the AI field and has been committed to providing full-stack technical solutions. In addition to hardware for different scenarios, Intel also supports the development of the AI industry by constantly improving the software ecology. At the Intel artificial Intelligence Conference held earlier, Dai Jinquan, senior chief engineer of Intel Software and Services Division and global CTO of big data Technology, shared two major software "sharp tools"-BigDL and Analytics Zoo.

Help users to build a unified big data analysis and AI platform

According to reports, BigDL is an open source and Apache Spark-based distributed deep learning framework developed by Intel. It can help users directly develop deep learning work applications on existing Hadoop/Spark clusters without deliberately copying data from big data clusters to independent deep learning clusters for training. Deep learning can also be used for analysis on the same cluster where the data is stored, and existing big data tools (such as Spark workflow) can be reused to build large-scale deep learning applications.

Analytics Zoo can be said to be an extension of BigDL, which can seamlessly integrate different software frameworks, such as Spark, TensorFlow, Keras, into an end-to-end data processing pipeline, run distributed on a large-scale cluster, and be transparent to users, helping users to build a unified big data analysis and AI platform.

Senior Chief engineer of Intel Software and Services Division, big data Technology Global CTO Dai Jinquan

Dai Jinquan said: "(with the help of BigDL and Analytics Zoo) users can run deep learning and AI applications directly where the data is stored on the existing big data cluster and big data platform, and can also integrate the whole life data processing process-- from data collection, processing and analysis, to machine learning, deep learning, etc., into a unified workflow and data pipeline." This has greatly improved the efficiency of development, operation and deployment. "

Recognized by industry users, the practical application effect is remarkable.

Baoxin Software is a listed software company controlled by Baosteel, which was formerly the automation department of Baosteel. The company is committed to promoting the integration and development of a new generation of information technology and manufacturing technology, and helping manufacturing enterprises move forward from information and automation to intelligent manufacturing. In order to promote the development of intelligent manufacturing, Baoxin Software launched big data platform software xInsight with independent intellectual property rights to explore how industrial big data creates value in practical application scenarios.

Baoxin software R & D team cooperated with Intel team to build an unsupervised deep learning system using Analytics Zoo solution, and developed a verification model for automatic equipment fault prediction. Dong Wensheng, general manager of Baoxin Software Research and Development, said: "the current cooperation between Baoxin and Intel has integrated Analytics Zoo and BigDL with the xInsight big data platform, which can provide relevant solutions on our big data platform. In the field of smart manufacturing, we will also work with Intel to further explore and provide more solutions."

In addition, Dai Jinquan also shared more cases. Midea and Intel team built a highly optimized defect detection solution based on Analytics Zoo to automatically detect product defects on the assembly line through computer vision. With Analytics Zoo, the R & D team seamlessly integrates Spark, BigDL and TensorFlow programs into an integrated pipeline.

The World Bank uses Intel's Analytics Zoo and AWS Databricks platform to build an image classification model. Analytics Zoo simplifies the pipeline of complex data processing, analysis and model training, helping R & D teams easily train data sets composed of 69 categories and nearly 1 million pictures, achieving high accuracy, and near-linear expansion of some data sets.

CRAY, a supercomputing company, worked with the Intel team to successfully build a short-term precipitation prediction model for their TB-level training data while ensuring excellent scalability. With the scalable deep learning function based on Apache Spark in Analytics Zoo, data scientists easily build complex workflows and make full use of CRAY supercomputers.

Last

Today, in the field of deep learning and AI, there are numerous excellent algorithms and frameworks, and the data used for training can often meet the requirements. However, with the increase of application scenarios, the development of AI applications becomes more and more difficult, and the project process becomes more and more complex. BigDL and Analytics Zoo launched by Intel, to some extent, fill the gap in AI development process optimization and efficiency improvement, and can help developers focus more on model optimization and promote application landing.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.