Google AI can’t understand netizens’ comments, and it will misunderstand up to 30%. Netizens: You don’t understand my stalks – yqqlm

Let me give you two sentences to taste the emotions contained in them: “I will really thank you.” “Listen to me, thank you, because of you, the four seasons have been warmed…”

Maybe you will say, this is very simple, isn’t it a stalk that has been played frequently recently? But if you ask the elders, they may be like “the old man in the subway looking at his mobile phone”.

However, there is a generation gap with popular culture, not only for the elders, but also for AI. No, a blogger recently po published an article analyzing Google datasets,It was found that the error rate of its sentiment judgment on Reddit comments was as high as 30%.

Just like this example:

I want to express my love to my friend angrily.

The Google dataset judged it as “angry”.

And this comment:

You TM almost freaked me out.

The Google dataset identifies it as “confused”.

Netizens shouted: You don’t understand my stalk.

Artificial intelligence becomes artificial mental retardation in seconds. How did it make such an outrageous mistake?

Taking it out of context, it is the most “good”

This has to start with the way he judges. Google dataset when labeling comments,It is judged by picking up the text sheet. We can take a look at the picture below, the Google dataset all misjudges the emotion in the text as anger.

Why don’t we speculate on the reason why the Google data set is wrong. Take the above example, there are some “swear words” in these four comments.

The Google dataset uses these “swear words” as a basis for judgment, but if you read the entire review carefully,You will find that this so-called “basis” is only used to enhance the tone of the entire sentencehas no practical significance.

The comments posted by netizens often do not exist in isolation, and factors such as the posts they follow and the platforms they publish on may cause the entire semantics to change.

For example, just look at this comment:

his traps hide the fucking sun.

It’s hard to judge the emotional element on this alone. But maybe it’s not hard to guess if it’s a comment from a muscle site (he just wanted to compliment the guy’s muscles).

It is unreasonable to ignore the commented post itself, or to single out a word with a strong emotional color to judge its emotional element. A sentence does not exist in isolation, it has its specific context, and its meaning changes with the context.

Putting comments into a complete context to judge their emotional color may greatly improve the accuracy of judgment. But the high error rate of 30% is not just “out of context”, there are deeper reasons.

“Our stalk AI doesn’t understand”

In addition to context interfering with dataset discrimination, cultural background is also a very important factor.

As large as a country or region, or as small as a website community, there will be its own internal cultural symbols, which are difficult for people outside the circle of cultural symbols to interpret, which creates a difficult problem: if you want to more accurately judge a certain The sentiment of community comments requires targeted data training on its community to gain an in-depth understanding of the cultural genes of the entire community.

On the Reddit website, netizens commented that“All raters are native English speaking Indians”.

This leads to misunderstandings of some very common idioms, modal particles and some specific “stalks”. Having said so much, the reason for such a high rate of error in data set discrimination is obvious.

But at the same time, there is also a clear direction for improving the accuracy of AI’s ability to identify emotions. For example, the blogger also gave several suggestions in this article:

first, when labeling comments, he has to have a deep understanding of his cultural background. Taking Reddit as an example, to judge the emotional color of its comments, it is necessary to have a thorough understanding of some American culture and politics, and to be able to quickly get the “stalk” of the exclusive website;

Secondto test whether the label’s judgment of some sarcasm, idioms, and memes is correct, to ensure that the model can fully understand the meaning of the text;

at lastcheck the model judgment and our real judgment to give feedback and train the model better.

One More Thing

AI Daniel Wu Enda once launched a data-centric artificial intelligence movement.

Shift the focus of AI practitioners from model/algorithm development to the quality of the data they use to train models. Wu Enda once said:

Data is the food of artificial intelligence.

The quality of the data used to train it is also critical for a model, and in emerging data-centric approaches to AI, data consistency is critical. In order to get correct results, the model or code needs to be fixed and data quality improved iteratively.

…

Finally, do you think there are other ways to improve the language AI’s ability to discriminate emotions?

Welcome to discuss in the message area~

Reference link:

• [1]https://www.reddit.com/r/MachineLearning/comments/vye69k/30_of_googles_reddit_emotions_dataset_is/

• [2]https://www.surgehq.ai/blog/30-percent-of-googles-reddit-emotions-dataset-is-mislabeled

• [3]https://mitsloan.mit.edu/ideas-made-to-matter/why-its-time-data-centric-artificial-intelligence