もっと詳しく

After being challenged to write Chinese essays, AI is now focusing on English for the college entrance examination.

As a result, good guy, this year’s college entrance examination English paper (National A paper) got 134 points as soon as I got started.

And it’s not an accidental super performance.

In 2018-2021 10 sets of real test,AI scores are above 125 points, the highest record is 138.5 pointslistening and reading comprehension also got full marks.

This is proposed by CMU scholars,College Entrance Examination AI System Qin.

Its parameter quantity is only 1/16 of GPT-3,Average score is 15 points higher than GPT-3.

The secret behind it is called reStructured Pre-trainingis a new learning paradigm proposed by the author.

Specifically,It is to re-extract and reconstruct the information of Wikipedia, YouTube and other platformsand then feed it to AI for training, so that AI has stronger generalization ability.

The two scholars explained this new paradigm in depth with a paper of more than 100 pages.

So what exactly does this paradigm say?

Let’s take a deep dive~

What is Reconstruction Pretraining?

The title of the paper is very simple, it is called reStructured Pre-training (reconstruction pre-training, RST).

The core point of view is just one sentence, pay attention to data!

The authors argue that valuable information is everywhere in the world, and that current AI systems are not taking full advantage of the information in the data.

For example, like Wikipedia, Github, which contains various signals that can be learned by the model: entities, relations, text summaries, text topics, etc. None of these signals have been considered before due to technical bottlenecks.

Therefore, the author proposes a method in this paper,Data containing various types of information can be uniformly stored and accessed using neural networks.

They represent data in units of signals and structure, much like in data science we often structure data into tables or JSON format, and then retrieve the required information through specialized languages ​​such as SQL.

Specifically, the signal here actually refers to the useful information in the data.

For example, in the sentence “Mozart was born in Salzburg”,“Mozart” and “Salzburg” are signals.

Then, it is necessary to mine data and extract signals on various platforms. The author compares this process to treasure hunting in a mine.

Next, using the prompt method, these signals from different places can be unified into one form.

at last,Then integrate and store these restructured data into the language model.

In this way, the study can unify 26 different types of signals from 10 data sources, giving the model a strong generalization ability.

The results show that in multiple datasets, the performance of RST-T and RST-A zero-shot learning,Both outperform the few-shot learning performance of GPT-3.

In order to further test the performance of the new method,The author also thought of a way to let AI do the college entrance examination questions.

They said that many working methods now follow the idea of ​​​​Chinese GPT-3, and follow OpenAI and DeepMind in the application scenarios of evaluation.

Such as GLUE benchmarks, protein folding scores, etc.

Based on the observation of the development of the current AI model, the author believes that a new track can be opened up to try, so I thought of using the college entrance examination to practice AI.

They found a total of 10 sets of test papers in the previous and previous years to mark, and asked high school teachers to score.

For topics such as listening/image comprehension, scholars in the fields of machine vision and speech recognition are also invited to help.

finally,After refining this set of college entrance examination English AI model, you can also call her Qin.

As can be seen from the test results,Qin is definitely a top student, and all 10 sets of papers are higher than T0pp and GPT-3.

In addition, the author also proposed a benchmark for the college entrance examination.

They feel that the tasks of many evaluation benchmarks are very single, and most of them have no practical value, and it is difficult to compare with the human situation.

The college entrance examination questions not only cover a variety of knowledge points, but also have human scores for comparison, which can be said to kill two birds with one stone.

Fifth normal form of NLP?

On a deeper level, the author argues that,Refactoring pre-training may become a new paradigm for NLPthat is, treat the pre-training/fine-tuning process as a data storage/access process.

Previously, the author summarized the development of NLP into 4 paradigms:

  • P1. Fully Supervised Learning in the Non-Neural Network Era (Fully Supervised Learning, Non-Neural Network)

  • P2. Fully Supervised Learning Based on Neural Network (Fully Supervised Learning, Neural Network)

  • P3. Pre-training, fine-tune paradigm (Pre-train, Fine-tune)

  • P4. Pre-training, prompt, prediction paradigm (Pre-train, Prompt, Predict)

But based on their current observations of NLP developments, they think it may be possible to look at things in a data-centric way in the future.

That is, the differentiation of concepts such as pre-training / fine-tuning, few-shot / zero-shot will be more blurred, and the core only focuses on one point –

How much valuable information is there and how much can be used.

In addition, they also proposed an NLP evolution hypothesis.

The core idea here is that technology is always going in this direction — doing less for better, more general systems.

The author believes that NLP has experienced feature engineering, architecture engineering, target engineering, and prompt engineering, and is currently developing in the direction of data engineering.

Created by Fudan University alumni

One of the papers in this paper is Weizhe Yuan.

She graduated from Wuhan University with a bachelor’s degree, and then went to Carnegie Mellon University to study data science.

Research interests focus on text generation and evaluation for NLP tasks.

Last year, she was accepted for a paper by AAAI 2022 and NeurIPS 2021, and she also won the ACL 2021 Best Demo Paper Award.

The corresponding author of the paper is Pengfei Liu, a postdoctoral researcher at Carnegie Mellon University’s Language Technology Institute (LTI).

He received his Ph.D. from the Department of Computer Science of Fudan University in 2019, under the tutelage of Professor Qiu Xipeng and Professor Huang Xuanjing.

Research interests include NLP model interpretability, transfer learning, task learning, etc.

During his Ph.D., he swept various scholarships in the computer field, including IBM PhD Scholarship, Microsoft Scholar Scholarship, Tencent Artificial Intelligence Scholarship, and Baidu Scholarship.

One More Thing

It is worth mentioning that when Liu Pengfei introduced this work to us, he bluntly said, “We didn’t plan to submit it at first.”

This is because theyDon’t want the format of the conference paper to limit the imagination of conceiving the paper.

We decided to tell this paper as a story and give the “reader” a movie-watching experience.

This is why we set up a panorama of “viewing mode” on the third page.

It is to take everyone to understand the history of NLP development and what the future we look forward to, so that every researcher can have a certain sense of substitution and feel that he is leading the pre-trained language models (PLMs) through A process of mine treasure hunting towards a better tomorrow.

At the end of the paper, there are also some surprise eggs hidden.

For example, PLMs theme emoticons:

And the ending illustration:

Looking at it this way, it will not be tiring to read a paper with more than 100 pages~

Paper address:

https://arxiv.org/abs/2206.11147

.
[related_posts_by_tax taxonomies=”post_tag”]

The post This year’s college entrance examination English AI score is 134. The research of Fudan Wuhan University alumni is interesting. appeared first on Gamingsym.