Shulou(Shulou.com)06/02 Report--

Introduction: with the rapid development of the Internet of things, more and more users want to sink AI capabilities to the edge layer, so that edge devices can automatically process some data they care about, and then report the results to the cloud platform. This not only saves resources, but also improves the efficiency of operation.

With the progress of technology, mobile devices such as mobile phones have become a very important carrier of local deep learning. However, the increasingly heterogeneous hardware platform and complex terminal-side usage make the application ability of AI technology in the end-side quite challenged. The reasoning of end-to-side model is often faced with the limitation of computing power and memory. In order to fully support many hardware architectures and optimize the performance of artificial intelligence applications on these hardware, many technology companies have started in-depth research. Technology media InfoQ last week interviewed Zhang Zhiqiang, product manager of JD.com Zhaopin Cloud Vision Research and Development Department, to discuss JD.com 's application of AI technology at the end-to-side.

(the following is the arrangement of the interview manuscript)

What problems can AI technology solve at the end-to-side?

Before we discuss this topic, we need to clarify the end-to-side definition. The end side is actually relative to the cloud center side. In fact, the cloud center side is a centralized service, and all the video and image data collected and perceived are transmitted to the cloud center side through the network for subsequent processing. The resources on the cloud side are highly concentrated and versatile, but with the exponential explosive growth of IoT devices and data, the centralized computing model on the cloud side has gradually exposed many shortcomings, such as real-time data processing, network constraints, data security and so on.

AI technology is used in end-side computing, which we often call edge computing. Zhang Zhiqiang said that this model can better support AIoT scenarios and has the following advantages: first, AI technology used in end-side can process the collected data at the first time, and does not need to be uploaded to the cloud-side processing center through the network, which greatly speeds up the system response and reduces the system processing delay. The popularity of 5G technology also provides a guarantee for end-side processing. Second, end-to-side computing can more efficiently process valuable critical data (about 20%), and the rest (about 80%) is only temporary. Combining AI capabilities on the end-side can not only process data in a more timely manner, but also reduce the limitation of network bandwidth and ease the pressure on center-side data storage. Third, the end-side AI technology can efficiently process the user's source data, clean and protect some sensitive data, and the end-side equipment can only report the results of AI processing.

Generally speaking, the execution of AI on the edge side can carry out real-time data processing and real-time response, effectively reducing the delay caused by data transfer to the cloud computing center.

At present, the main application is the edge computing of the Internet of things. JD.com has some landing cases both internally and externally, for example, unmanned vending machines and intelligent checkout desks use AI's commodity detection technology, including SKU of goods selected by customers, mainly using end-to-side AI capabilities. As well as the basic human face structure, human body structure, vehicle structure, perimeter security detection based on video analysis, object legacy and other scenarios in the wisdom park. Some scenarios not only combine the AI capabilities of edge computing, but also use the cloud AI capabilities for secondary analysis, in the form of cloud collaboration. In some more complex scenarios, such as smart parks, smart communities, and smart cities, most of the projects are in the form of end-cloud collaborative applications.

Research and Development difficulties of end-side AI

As we all know, the end-to-side technical difficulty is how to meet the performance requirements of the business in the hardware environment with limited resources, whether storage, memory, or computing resources are relatively limited. The end-side puts forward higher requirements for the AI model, such as model life cycle management, release, rollback, grayscale, version management and so on. How to facilitate the upgrade and update of the model needs to establish a set of perfect general mechanism and platform. In addition, there are many kinds of hardware devices for edge computing, and the initial learning, adaptation and model migration of different hardware architectures will take a long time. For example, many previously defined model algorithm operators are not supported, and algorithms are needed to focus on new design and development, which virtually increases the complexity of end-to-side application of AI.

For this reason, after early accumulation, JD.com also abstracted and platform the popular computing hardware products at different edges in the market, shielded the underlying differences, and uniformly built a series of CI/CD mechanisms from algorithm development, model adaptation, local testing, deployment and online, to achieve rapid delivery.

In the past, many developers have encountered the problem of inconsistent results between offline training and online reasoning. In this regard, JD.com found that the results of training and reasoning will not be exactly the same, but basically the same. It can be considered from the following three aspects: one is to lighten the weight of the model design as much as possible. On the premise of ensuring the performance of the model, the network parameters are reduced without losing the network performance, and the complexity and computation of the model are reduced as much as possible; the second is to weigh the performance and accuracy to find the balance point; the third is to detect whether there is a difference between pre-processing and post-processing, which can be predicted and compared before and after network processing.

Technical practice of JD.com end-to-side AI

In 2019, JD.com Cloud and AI Visual Research and Development Department began to plan AI video analysis platform mVCG-Air in end-to-side scenarios, which corresponds to the version of mVCG-Pro deployed in cloud scenarios, which are collectively referred to as mVCG (mega-Video Computing Grid), that is, a very large-scale video computing and analysis network.

[overall architecture diagram of mVCG-Air technology]

MVCG-Air is a typical product of AI video analysis and processing capabilities in end-to-side applications. After rapid iteration, mVCG-Air has landed in different scenarios. Zhang Zhiqiang said that the platform solves the problem that there are many devices connected to the landing scene, at the same time, the timeliness of video analysis is very high, and cloud processing is affected by the lag of network bandwidth.

In addition, mVCG-Air combines the current high-performance end-to-side computing devices, while all internal AI core technologies developed by JD.com Zhaopinyun are used. All algorithm models are fully optimized and accelerated for end-side computing devices, so developers can easily achieve customization and fast delivery.

It is necessary to unify the model development process

Integrating the whole practice process, it is not difficult to find that it is necessary to have a unified research and development process of all models, so that it is very necessary to make unified planning from feature engineering at the beginning of model development to model training, and then to reasoning online. For example, JD.com 's same model may need to be deployed in different scenarios in the future, some based on edge computing mVCG-Air deployment, some based on cloud-centric mVCG-Pro deployment, then a unified model training and transformation mechanism is required. At the same time, the same model may be deployed on different edge hardware, and subsequent consistency needs to be considered at the beginning of the model design. JD.com has deposited a complete set of model development and testing process, so that a model can easily land on different heterogeneous hardware devices at the same time.

Resource scheduling design

With regard to resource scheduling, it actually includes the scheduling of edge-side mVCG-Air and center-side mVCG-Pro, as well as the scheduling between devices managed by mVCG-Air. Zhang Zhiqiang said that there is a very important module component in mVCG, which is the model repository, which is associated with computational scheduling, task scheduling and so on.

[algorithm warehouse architecture diagram]

Algorithm warehouse architecture makes intelligent analysis capabilities no longer solidified in mVCG products, and realizes "one platform, multiple algorithms", which has the advantages of high system availability, high resource utilization, flexible and open algorithms and so on. Based on the algorithm warehouse, all algorithm models of mVCG can be managed uniformly and efficiently, including model management, model gray deployment, model release, model upgrade and model algorithm version management. Each model has its own portrait Model Profile. Through distributed components, the model computing power is flexibly allocated and scheduled according to the model portrait and the current business load. All model algorithms are based on plug-in mechanism to achieve flexible deployment, automatic allocation, fast loading, efficient operation and unaware version update.

Hardware selection

In fact, there is no end to customization and optimization at the software level, which includes model accelerated reasoning, model quantitative pruning, making the model smaller in size, taking up less resources without losing accuracy. Of course, the optimization at the software level also depends on the openness of the edge-side devices.

In addition to the ability at the software level, the super computing power required by AI must be matched by the right hardware. Zhang Zhiqiang said that mVCG-Air of JD.com Cloud and AI Visual Research and Development Department used CPU+NPU based on ARM architecture in the research and development process, and now there are constantly newer and faster hardware acceleration cards or end-to-side products, selecting the most cost-effective and the best for business scenarios.

In the future, mVCG-Air will also continue to endow sensing devices with intelligence, and integrate data from the Internet of things and information network in the cloud to realize holographic perception of people (face, human body), cars, objects, scenes and behavior, while deeply mining the potential value of multi-dimensional data. The focus of the application includes many important scenes inside JD.com, as well as external scenes, such as smart park, smart city, smart station, smart policing and so on.

Future planning

As an important supplement to traditional cloud computing, edge computing plays its own strengths. With the substantial improvement of the processing capacity of end-to-side devices, intelligent devices based on AI are playing a more and more important role.

The combination of AI and end-to-side is actually a part of AIoT, because AIoT is a concept that sprang up in 2018. At present, all kinds of AI companies and chip companies are in the stage of contending among a hundred schools of thought, and the market space is very large, including the prospect of end-to-side artificial intelligence in different industries, such as architecture, human settlements, industrial manufacturing, smart cities, and so on.

A trend that is taking place in AIoT in 2020 is the accelerated integration of artificial intelligence and devices. In most cases, end-to-side artificial intelligence computing and cloud-side computing will coexist for a long time, or even adopt a hybrid approach, some of which will be performed in time by the device's own AI capabilities, while the other will be completed through a unified cloud center. Now, the problems of hardware and software selection, dynamic scheduling and vertical landing are more discussed in the industry for end-to-side artificial intelligence. As a sinking computing model of cloud computing, the development of end-to-side artificial intelligence still needs some time and patience.

Welcome to click "more" to learn more about JD.com Zhaopinyun!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.