Shulou(Shulou.com)06/01 Report--

Editor to share with you can explain what AI is, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

1. Why do you need interpretable AI?

In human history, the development of technological progress, the logic of production relations and ethical laws and regulations is a dynamic evolution. When a new technology makes a breakthrough in the laboratory, the change in the way of value generation will have an impact on the commodity form and production relations in turn, and at the same time, when the value promotion brought by the new technology is recognized, in the process of spontaneous adjustment, the organizational form of business logic will also put forward demands for the path, content and even speed of technological development, and adapt to new ethical regulations when the demands are met. In this interaction, the technological system and the social system will resonate to complete the evolution, which is called the technological revolution.

In the past 10 years, with the performance-to-price ratio of computing power and data scale breaking through the critical point, the connectionist model architecture represented by deep neural network and statistical learning paradigm (hereafter referred to as deep learning) have made a breakthrough in the ability of feature representation, which has greatly promoted the development of artificial intelligence and achieved incredible results in many scenarios. For example, the accuracy of face recognition is more than 97%; the correct answer rate of Google Intelligent Voice Assistant is 92.9% in the 2019 test. In these typical scenarios, the performance of deep learning in intelligent performance has exceeded that of ordinary human beings (and even experts), thus reaching the critical point of prying technology replacement. In the past few years, in some areas where business logic is technology-friendly or ethical regulations are temporarily scarce, such as security, real-time scheduling, process optimization, competitive games, information flow distribution, etc., artificial intelligence (and deep learning) has made rapid breakthroughs in technology and business.

Knowing the taste, every field is willing to let go of the sweetness of technological development. When the commercial application of deep learning comes to some areas that are sensitive to technology and closely related to human survival or security, such as autopilot, finance, medical treatment, justice and other high-risk application scenarios, the original business logic will encounter resistance in the process of technology replacement, which leads to the slowdown or even failure of commercialization (and realization). The reason is that one of the centers of business logic and ethical regulations behind the above scenarios is stable and traceable responsibility clarity and distribution, while the model obtained by deep learning is a black box. it is impossible to obtain any information about the behavior of the model from the structure or weight of the model, so that the centers of responsibility tracking and distribution in these scenarios cannot be reused, resulting in technical and structural difficulties for artificial intelligence in business applications.

Give two specific examples:

In the financial risk control scenario, through the deep learning model to identify a small number of users suspected of fraud, but the business unit dare not directly use this result to deal with, because it is difficult to understand how the result is obtained, so it is impossible to judge whether the result is accurate, worried about handling errors; and there is no clear basis, and if handled, it cannot be explained to the regulatory authorities.

Example 2, in the medical field, the deep learning model judges that the patient has tuberculosis according to the patient's test data, but the doctor does not know where the diagnosis comes from and does not dare to use it directly, but carefully examines the relevant test data according to his own experience, and then gives his own judgment. From these two examples, we can see that the black box model seriously affects the application and promotion of the model in the actual scene.

In order to solve these problems of the model, it is necessary to open the black box model and transparent the model construction process and reasoning mechanism. Then interpretable AI is an effective technology to achieve model transparency.

two。 What is an interpretable AI?

Interpretable AI (eXplainable AI (XAI)) does not have a unified definition in both academia and industry. Here are three typical definitions for your reference and discussion:

① interpretability is to seek a direct understanding of the working mechanism of the model and break the black box of artificial intelligence.

② interpretable AI provides a human-readable and understandable explanation for decisions made by the AI algorithm.

③ 's interpretable AI is a set of ways to ensure that humans can easily understand and trust decisions made by artificial intelligence agents.

It can be seen that the focus is on the understanding of the model, the white box of the black box model and the trust of the model.

According to their own practical experience and understanding, the MindSpore team defines interpretable AI as a set of technology collections for machine learning (mainly deep neural networks), including visualization, data mining, logical reasoning, knowledge graph, etc., in order to make deep neural networks understandable through this technology collection. In order to meet the information demands of relevant users on the model and application services (such as causal or background information), so as to establish cognitive trust for users on artificial intelligence services.

3. Can you explain what problem AI solves?

According to DARPA (Advanced Research projects Agency of the U.S. Department of Defense), the purpose of AI is to solve the problem that users encounter with the black box of the model, so as to achieve:

① users know why the AI system does this, and why the AI system does not.

② users know when the AI system will succeed and when the AI system will fail.

③ users know when to trust the AI system.

④ users know why the AI system went wrong.

The interpretable AI plan implemented by MindSpore mainly addresses the following two types of problems:

① solves the problem of opaque technical details under the deep learning mechanism, so that developers can obtain the detailed information in the development process through XAI tools, and through the obtained understandable information, help developers to troubleshoot models or improve performance.

② provides a friendly cognitive interface (cognitive-friendly interface) between deep learning technology and service objects. Through this interface, service providers based on deep learning can have appropriate tools to commercialize and logically operate deep learning services. Consumers of artificial intelligence services based on deep learning can obtain the necessary doubt resolution and build trust, and can also provide ability support for the audit of AI systems.

4. Methods of implementing interpretable AI

From the previous introduction, we can feel that the interpretable AI is very high-end, so how to implement it? Combined with the research results of the industry, we believe that there are three main methods to implement interpretable AI:

The first kind is interpretable based on data, which is often called depth model interpretation, which is the easiest method to think of, and it is also a kind of technology often involved in many papers; it is mainly based on data analysis and visualization technology to realize depth model visualization and intuitive display to get the key basis of model results.

The second type is model-based interpretability, which is also known as interpretable model method, which mainly makes the model itself explainable by building an interpretable model, and outputs the reason of the result at the same time.

The third kind is interpretable based on results, which is also called model induction method. The idea is to take the existing model as a black box, observe the behavior of the model according to a given batch of inputs and corresponding outputs, and infer the reason for the corresponding results. the advantage of this kind of method is that it has nothing to do with the model and any model can be used.

5. Interpretable AI capabilities of the MindSpore1.1 version

The open source version of MindSpore1.1 integrates interpretable AI capabilities in MindInsight parts: saliency map visualization (Saliency Map Visualization), also known as key feature area visualization. This part belongs to the first kind of interpretable methods based on data. In the future, we will open up more interpretable methods. In addition to the first category, we will also include the second type of model-based interpretable methods and the third kind of results-based interpretable methods.

At present, saliency map visualization is mainly model interpretation in CV domain. In version 1.1, we support six visualization interpretation methods: Gradient, Deconvolution, GuidedBackprop, GradCAM, RISE, Occlusion.

Among them, Gradient, Deconvolution, GuidedBackprop and GradCAM belong to gradient-based interpretation methods. This type of interpretation method, mainly using the gradient calculation of the model, to highlight the key features, relatively high efficiency, the following four methods are briefly introduced:

Gradient is the simplest and most direct interpretation method. By calculating the gradient of the output to the input, the "contribution" value of the input to the final output is obtained, while Deconvolution and GuidedBackprop are the extension and optimization of Gradient.

Deconvolution, the ReLU in the original network is modified to be a gradient ReLU, thus filtering the negative gradient, focusing only on the characteristics that make a positive contribution to the output.

GuidedBackprop, which filters negative gradients on the basis of the original network ReLU, only focuses on the features that make a positive contribution to the output and is activated, and can reduce significant graph noise.

GradCAM calculates the category weight for the intermediate activation layer and generates a significant graph that is sensitive to the category, which can get the explanation of the category.

The other two methods, Occlusion and RISE, belong to disturbance-based interpretation methods. The advantage of this type of method is that it can be model-independent by using only the input and output of the model.

RISE, using Monte Carlo method, weighted the random mask (the weight is the output of the masked model) to get the final saliency map.

Occlusion, by masking the input of a specific location, calculating the changes of the model output to get the "contribution" of that location, traversing all the inputs to get a significant graph.

So for specific scenarios, how to choose the appropriate interpretation method to explain the corresponding model?

In order to meet this demand, the MindSpore1.1 version provides a measurement framework that can explain AI, as well as four measurement methods: Faithfulness, Localization, Class sensitivity and Robustness, to measure the interpretation effect of different interpretation methods to help developers or users choose the most appropriate interpretation method. Briefly describe these four metrics:

Faithfulness, credibility. Remove features from large to small according to importance, and record the reduction of the probability of a particular label. After traversing all the features, the distribution of confidence reduction and the distribution similarity of feature importance are compared. The more consistent the interpretation with the black box model, the more similar the two distributions should be, and the better credibility of the interpretation method.

Localization, locality. The interpretation based on the salient map has the ability to locate the target (that is, given the tag of interest, highlighting the part related to the tag in the salient map), localization uses the target detection data set, for the same picture with the same tag, through the coincidence of the highlighted part of the salient map and the Ground Truth to measure the positioning ability of the salient map.

Class sensitivity, classification sensitivity, the highlight of the objects in the map corresponding to different classification should be significantly different. By comparing the saliency maps of the maximum and minimum labels, the greater the difference between the saliency maps of the two labels, the better the classification sensitivity of the interpretation method.

Robustness, robustness. The index reflects the anti-disturbance ability of the interpretation method in the local range. The smaller the Lipschitz value is, the smaller the influence of the interpretation disturbance is, and the stronger the robustness is.

Combined with three interpretation methods of saliency map visualization methods supported in the MindSpore1.1 version: Gradient, GradCAM and RISE, this paper introduces how to use the interpretable AI ability to better understand the prediction results of the image classification model and obtain the key feature regions that are the basis for classification prediction, so as to judge the rationality and correctness of the classification results and accelerate model tuning.

6. Introduction of visual interpretation methods for common salient maps 6.1 Gradient interpretation method

Gradient is the most simple and direct interpretation method based on gradient. By calculating the gradient of the output to the input, the "contribution" value of the input to the final output is obtained, which is used to explain the key feature basis of the prediction results.

For the depth neural network, the gradient calculation can be obtained by the back propagation algorithm, as shown in the following figure:

Source:Understanding models via visualizations and attribution

Note: Gradient interpretation method will encounter the problem of gradient saturation, that is, once the contribution of a feature reaches saturation, because the feature no longer affects the change of the result, it will cause the gradient of the feature to be 0 and make an error.

The interpretation effect of Gradient is shown in the following figure:

Note: all explanatory effect pictures are from MindInsight screenshots.

As can be seen from the above two pictures, the interpretation result of Gradient is not understandable, the highlight area of the significant map is scattered, and there is no clear feature region location, so it is difficult to obtain the specific feature basis of the prediction results.

6.2 GradCAM interpretation method

GradCAM, the abbreviation of Gradient-weighted Class Activation Mapping, is generally translated as: weighted gradient class activation mapping, is a gradient-based interpretation method; this interpretation method weighs each channel activation graph of a certain layer network (the weight is calculated from the gradient) to obtain the key feature regions that affect the prediction classification results.

GradCAM provides an overview of the process of explaining the result, as shown in the following figure:

Source:Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization

Here are two practical examples to see the specific interpretation effect of GradCAM:

It can be seen that the interpretation results of GradCAM are well positioned and understandable, and the highlighted areas focus on the specific features, and users can judge the features related to the predicted results through the highlighted areas. For the label "road sign", the road sign in the image is highlighted, and for the label "aircraft", the aircraft in the image is highlighted, that is, GradCAM believes that the road sign area and the aircraft area are the main basis of the two classification results.

6.3 RISE interpretation method

RISE is the abbreviation of Randomized Input Sampling for Explanation, that is, the interpretation based on random input sampling, is a disturbance-based interpretation, independent of the model; the main principle: using Monte Carlo sampling to generate multiple masks, and then the random mask is weighted (the weight is the output of the masked model) to average the final significant map.

An overview of the interpretation process of the RISE method is as follows:

Source:RISE: Randomized Input Sampling for Explanation of Black-box Models

Two examples are also given to show the effect of the explanation:

RISE uses the masking method to get the feature areas related to the classification results, and the interpretation results are understandable and localized well. Similar to GradCAM, RISE accurately highlights the road sign area and the aircraft area.

7. How does the MindSpore interpretation method explain the picture classification model?

In practical application, what is the interpretation effect of the three interpretation methods introduced above? According to the prediction results and the effectiveness of the interpretation, we divide the interpretation results into three categories, and find several typical examples to see the actual effect.

Description: the explanation results in all the sample diagrams below are shown by the model interpretation features in the MindInsight part of MindSpore.

7.1 examples of correct prediction results and reasonable key features

Example 7.1.1: the prediction label in the above picture is "bird". The key feature given on the right is in the bird, indicating that this classification basis is reasonable.

Example 7.1.2: the prediction label above is "bicycle". When explained on the right, the bicycle wheel is highlighted, which is also a reasonable key feature.

7.2 the prediction results are correct, but the key features given may be based on false examples.

Example 7.2.1: in the original image, there is someone in the prediction tag, there is a tag is "person", this result is correct; but select the "person" tag, when you look at the explanation on the right, you can see the highlighted area on the horse's head, then this key feature basis is likely to be wrong.

Example 7.2.2: in the original picture, there is a bird whose prediction result "bird" is correct, but when explained on the right, the highlight is on the branches and leaves, not on the bird, which is probably wrong.

Example 7.2.3: in the picture above, there is a small boat with a label of "boat". This is true. However, in the interpretation of the label "boat" on the right, the highlight area is on the person, which is a bit of a deviation.

Careful analysis of the above three examples, this highlight marks as the basis for the classification of the key features, the occurrence of errors, generally appear in the scene where there are multiple targets in the image. According to the tuning experience, it is often because these features often appear together with the target object in the training process, resulting in the model mistakenly identifying these features as key features in the learning process.

7.3 the prediction result is wrong, combined with the key features of the basis to analyze the examples of error causes.

Example 7.3.1: in the above image, there is a prediction label "boat", but there is no ship in the original image. Through the interpretation of the label "boat" on the right side of the image, we can see that the model takes the water surface as the key basis for classification, and the prediction result "boat" is wrong. Through the analysis of the data subset labeled "boat" in the training data set, it is found that there is water surface in most of the pictures labeled "boat", which may lead to the misuse of water surface as the key basis of "boat" type in model training. Based on this, the picture collection with ships without water surface is supplemented proportionally, thus the probability of misjudging key features in model learning is greatly reduced.

Example 7.3.2: there is a classified prediction result of "aeroplane" in the figure, but there is no aircraft in the picture. From the interpretation of the label "aeroplane", the highlight area is on the eagle. Open the subset of the training data related to the aircraft, and find that in many cases the plane in the training picture is a distant target, which is very similar to the flight of an eagle. It is speculated that this may be the reason why an eagle is mistaken for a plane when model reasoning is made. When optimizing the model, we can consider to increase the picture proportion of eagle gliding, improve the discrimination and discrimination ability of the model, and improve the classification accuracy.

Example 7.3.3: in this example, there is a prediction tag "person", and there is no one in the picture. According to the explanation of the label "person", the highlight area is in the front of the motorcycle; in the original picture, at first glance, it looks like a person lying on his stomach; guessing is that when reasoning from the classification model, this part is mistaken for a person.

8. How to deploy and use the MindSpore interpretation method?

In the model interpretation tutorial on MindSpore's official website, it describes in detail how to deploy and use the interpretation methods provided by MindSpore. For links, please see:

Https://www.mindspore.cn/tutorial/training/zh-CN/r1.1/advanced_use/model_explaination.html

The following is a brief introduction to deployment and usage:

First, use the script to call the python API provided by MindSpore to explain the generation and collection of the results, and then start MindInsight to display these results. The overall process is as follows:

The specific steps are as follows:

8.1 prepare script

By calling the interpretation method Python API to interpret the prediction results of the model, the provided interpretation method can be obtained in the mindspore.explainer.explanation package. The user prepares the black box model and data to be interpreted, instantiates the interpretation method as needed in the script, and calls API to collect the interpretation results.

MindSpore also provides a mindspore.explainer.ImageClassificationRunner interface that supports automated operation of all interpretation methods. By registering the instantiated interpretation method in advance, the user can automatically run the interpretation method after the interface is called, and generate and save the interpretation log containing the interpretation results.

Taking ResNet50 as an example, this paper describes how to initialize the interpretation method in explanation and call ImageClassificationRunner for interpretation. The sample code is as follows:

````import mindspore.nn as nnfrom mindspore.explainer.explanation import GradCAM, GuidedBackpropfrom mindspore.explainer import ImageClassificationRunner from user_defined import load_resnet50, get_dataset, get_class_names # load user defined classification network and datanetwork = load_resnet50 () dataset = get_dataset () classes = get_class_names () data= (dataset, classes) runner = ImageClassificationRunner (summary_dir='./summary_dir', network=network, activation_fn=nn.Sigmoid (), data=data) # register explainersexplainers = [GradCAM (network, layer='layer4') GuidedBackprop (network)] runner.register_saliency (explainers) # run and generate summaryrunner.run () ``8.2 use MindInsight to display the results 8.2.1 start MindInsight

Start the MindInsight system and select to enter the Model interpretation module at the top. You can see all the interpretation log paths, and when the log meets the conditions, the operation column will have a functional entry for "saliency visualization".

8.2.2 introduction of various functions

Saliency map visualization is used to show the areas of the picture that have the most significant impact on the prediction results of the model. usually, the highlighted part can be regarded as the key feature that the picture is marked as the target classification.

Enter the visual interface of the salient image, such as the above image, which will show:

The target dataset set by the user through the Python API interface of Dataset.

The real label, the prediction label, and the prediction probability of the model to the corresponding label. According to the specific situation, the system will add the flag of TP and FP,FN (for meaning, see the prompt on the interface) in the upper left corner of the corresponding label.

A salient diagram given by the selected interpretation method.

Introduction to interface operation:

1. Check the required interpretation method through the interpretation method at the top of the interface

two。 By switching the "overlay on the original image" button at the top right of the interface, you can choose to overlay the significant image on the original image.

3. Click on different tags to display the significant map analysis results of different tags. For different classification results, the key feature areas are usually based on different features.

4. Filter out the specified tag image through the tag filtering function at the top of the interface

5. Change the order in which pictures are displayed by sorting the pictures in the upper right corner of the interface

6. Click on the picture to see an enlarged picture.

These are all the contents of the article "what can be explained by AI". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.