Zebras are recognized as dogs, and the reason why AI made mistakes was found by Stanford – Programmer Sought

It’s obviously a zebra, why does AI say it’s a dog?

Classification models sometimes misjudge categories when classifying images.

The AI ​​that has been learned will also mess up some predictions, and there must be a flaw in one of the links.

In a paper by two Stanford PhD students and professor James Zou, we explore why classification models make mistakes.

Subsequently, the paper proposes a method, Conceptual Counterfactual Explanations, and evaluates its effectiveness.

In this way, we can redefine the taxonomy of models to explain why AI makes mistakes.

Let’s take a look together.

Why does AI make mistakes?

To improve the accuracy of the subsequent predictions of the classification model, you must first understand where the model went wrong.

Generally speaking, AI messes up predictions for the following reasons:

The first is that in the actual prediction process of the model, the classification criteria may be somewhat deviated from the pre-training, which makes the training model ineffective in the actual prediction process, thereby reducing the accuracy of the prediction results.

For example, a pathologist uses a pre-trained model to classify histopathology images, but the results are not very good on his images.

This may be due to the different tones in his images than in the original training data.

Second, in the process of model learning, something called “pseudo-correlation” may be learned, which is to associate some seemingly unrelated features with the recognized image.

See this example:

During the training of this model, all photos of dogs in the sample had snow in them, which caused the model to associate snow with dogs and incorrectly predict that a dog without snow is not a dog.

This may be the datasets used, all collected in the same scene, hindering the generalization of the model.

In addition, there may also be some artificial biases when training the model.

For example, one dermatologist used trained AI to classify skin diseases in images, but other colleagues used it less well.

This may be because in the training samples, the skin color is single and the age distribution is narrow.

Now that we understand why AI “makes mistakes”, how can we accurately determine where the model is wrong?

AI makes mistakes, it explains

In his paper, James Zou proposes a method called Counterfactual Conceptual Explanation (CCE).

Specifically, this method is used to explore the connection between the input data and the predicted results, and finally find the error of the model.

So how does CCE explain it?

Define concept base

The first thing to do is to set up and refine a concept library C, that is, to make a classification standard.

Specifically, each concept in the concept library C can be used to classify images, such as installation c1 (street, snow, etc.), picture quality c2 (clear, blurred, etc.)・・・・・・

In this way, a set of interpretable concept bases C = {c1, c2, …} can be obtained.

Then, you need to find corresponding training data for each concept in it.

Specifically, it is to collect examples that are consistent (Pci) and non-conforming (Nci), generally speaking, the number should be the same (Pci=Nci=100).

For each concept, CCE learns their taxonomy and ‘way of thinking’.

By two methods:

One is by learning support vector machines (SVMs) to find algorithms (linear classifiers) that can distinguish the optimal way of two things.

The other is to learn the corresponding concept activation vector (CAV), which can be used to explain the specific reasons why images are misclassified.

Just like the picture below, they are all images of zebras, but the reasons for the misclassification are different.

This step only needs to be done once for each model you want to evaluate, after which CAV can be used to account for any number of misclassifications.

Given error classification criteria

We can adjust the classification criteria of different concepts by changing the weight of different concepts in the model. These adjustments must meet the following principles:

1. Correctness: A classification criterion is considered correct if it achieves the expected results.

2. Validity: Classification criteria cannot violate basic human cognition.

3. Sparsity: The ultimate goal is to convey the error of the model to the user. Too many variables are not conducive to effectively conveying information.

Our goal is to make the predictions as close as possible to the training results, i.e. minimize the cross-entropy loss.

Therefore, it is necessary to continuously optimize the standard of model prediction, and finally achieve the effect of correcting misclassification by adjusting the standard to be modified and weighting it.

After understanding, let’s see through a practical example, how to use CCE to “detect” the error of the classification model.

Here, the classification model incorrectly identified the zebra image as an African hound.

So, we first generate a set of criteria for the model to recognize zebras as dogs.

These criteria are then scored, and if the score is positive, it means that adding this concept to the image will improve the probability of correct classification, and vice versa.

In this example, if the concept of stripes (stripes) is added, the probability of identifying it as a zebra will be higher.

In figure c), it can also be seen intuitively through the CCE analysis that “Polka Dots” (spots) and “Dog” (dogs) are responsible for the wrong predictions of the model.

What is the effect of CCE?

Seeing this, everyone must have a preliminary understanding of the principle of CCE.

Then it judges whether it is accurate or not, and what is the effect?

The purpose of CCE is to reveal the “pseudo-correlations” learned by the model during training, which can be used to capture other “irrelevant elements” that appear in the image and pseudo-correlations of the image.

Testing found that, in most cases, the model identified spurious correlations in more than 90% of the misclassified test samples.

Looking at this table, using CCE has the highest probability of identifying spurious correlations in samples compared to other methods.

CCE can accurately identify false correlations in samples, see this example:

Changing the color of the apple image (making the image gray) increases the probability that the classification model will misidentify (black line), and the higher the CCE’s score for identifying “green” as a false correlation (green line).

In addition to this, CCE also has the potential to be useful in the medical field.

CCE is used by Abubakar Abid et al., which has been tested in dermatology (classification of skin conditions) and cardiology (classification of pneumothorax) in chest radiographs.

CCE uses learned bias and image quality conditions to explain model errors, and has also been confirmed by professional dermatologists—factors that indeed largely contribute to the difficulty in classifying skin images.

In addition, the speed of CCE is also very fast.

The concept base only needs to be learned once using a simple SVM, and each test example takes less than 0.3s on a single CPU.

Importantly, it can be easily applied to any deep network to detect why the model is making mistakes without the need for training data.

If you are interested in this method and want to try it yourself, you can click the link below to view it.

about the author

James Zou, the corresponding author of the paper, is an assistant professor in the Department of Biomedical Data Science and an assistant professor in the Department of Computer Science and Electrical Engineering at Stanford University.

He received his PhD from Harvard University in 2014 and was a member of Microsoft Research, a Gates Scholar at Cambridge University, and a Simons Fellow at UC Berkeley.

His research has been supported by Sloan Fellowship, NSF CAREER Award, and Google, Amazon, and Tencent AI awards.

Abubakar Abid (front) and Mert Yuksekgonul (back) are the first authors of the paper, both PhD students at Stanford University.

Reference link:

1,https://arxiv.org/pdf/2106.12723.pdf

2,https://github.com/mertyg/debug-mistakes-cce

3.https://twitter.com/james_y_zou/status/1541452062344417280