25,000 people managed to beat an AI in chess together

A few weeks ago Mehdi Moussaïd asked himself a question: what could a crowd of beginner chess players do against a grandmaster? From this starting postulate, this researcher in cognitive sciences at the Max Planck University in Berlin, decided to conduct his own experiment, unprecedented in the world of research.

Passionate about multi-agent systems (systems with individuals very similar to each other such as companies of ants or crowds of humans), Mehdi Moussaïd spent his doctorate on the movement of pedestrians in 2007. After this first study , he decided to take a closer interest in the study of crowds. A discipline which, at the beginning of the 2010s, was then nascent. “ A lot of things still needed to be discovered, maybe that’s what I liked. »

Studying crowds: their movement, but also their intelligence

If the research is quite new in this field, the concrete applications are already numerous. With regard to the question of the movement of crowds, many people are very interested in the simulations made by Mehdi Moussaïd and his colleagues. “ As well in the organization of sporting events, as for concerts or just the circulation of the flow of travelers in a metro or an airport “.

Another very fashionable area in the world of research is the question of the dissemination of information, particularly on social networks. If here the concrete applications are still rare, it is easy to imagine how such discoveries could be used.

With regard to the intelligence of crowds, which interests us today, Mehdi Moussaïd rather speaks of “ fondamental research ». According to him, experiments prove that the crowd can be useful in solving complex problems, but the way in which this collective intelligence should be used is still to be found.

5 days to do better than the best scientists in the world

For example, the scientist David Becker set up an experiment a few years ago, which today serves as a reference when it comes to crowd intelligence. While the greatest biochemists had been stuck for years on a question, they had the good idea to ask it to the internet, and let the crowd answer. In just 5 days, more than 200,000 people thought together and found a viable solution.

This experiment shed light on the intelligence of the crowd, but also allowed the scientific community to understand the movement and the complex mechanisms of a protein which plays a key role in the birth of AIDS. “ We know that the crowd can be very intelligent, we have to understand when and how we can reach our full potential “.

25 000 people against an AI around a chessboard

So, Mehdi Moussaïd looked for the best way to measure the intelligence of a crowd. He finally settled on chess, a game “symbol of intelligence” which also has the great advantage of being able to easily assess the level of a player according to his choices. “ With the ELO system (notation of chess players), we can precisely calculate the level of a single player, but also of a crowd playing together. »

To carry out his experiment, Mehdi Moussaïd therefore brought together nearly 25,000 people on a website created for the occasion. “ Each new player had to rate their level on a scale of 1 to 5 and then they had 5 different games to play, with three moves per day at a fixed time. “. Facing the players, the researcher has placed an artificial intelligence, Maïa. She has the specificity of playing “like humans” and does not always seek the best theoretical move. Set up specifically for the experiment, she had a different level in each of the five parts. “ It ranged from very weak, to really really hard to beat “.

In addition to these five parts which built the scientific experiment, Mehdi Moussaïd had the idea of ​​creating a reference part. He then contacted Kévin Bordi, streamer and former professional player to serve as an opponent to the crowd. In a game broadcast live, this National Master quite easily beat the crowd.

And here is the complete part of @Blitz_Stream (in black) against the crowd (in white).

Come join the rematch here: https://t.co/ro0OVJRbUk pic.twitter.com/5XlcxBEuFq

— Mehdi Moussaïd (@Mehdi_Moussaid) March 29, 2022

A result that would tend to discourage the researcher when analyzing the data of the relayed parts against the AI, but fortunately things are quite different against the machine. “ The win percentage against the AI ​​was very good, and far from what I expected. It is estimated that the ELO level of the crowd was 1700-1800 points, which corresponds to a very serious player “. When we look at the graph below we can see the different win rates for the crowd depending on the level of the AI, as well as the projection of the win rates for a player with an ELO level of 1165, which had been calculated as the individual player average.

“The difference is clear, and the benefit brought by the crowd is beyond doubt” declares the researcher.

From this experience, Mehdi Moussaïd draws two main conclusions. The first concerns a sub-part of his experience. Indeed, of the five games played, some were played with a poll specifying the majority moves, others were played “blind”. According to the information collected during this experiment, the poll affected the results in a beneficial way. “ The crowd’s win rate was 77% with a poll and 64% without. So it has a clear benefit, which I hadn’t anticipated at the start. “.

Surveys that improve the overall level of play

Polls tend to favor ideas that are in the majority first. “ If the first shots proposed are good, they will be put forward by the poll and the majority will play well. But if those first moves are bad, then the crowd gon’ follow that bad idea all the way », explains the researcher.

According to him, the success of the polls can be explained by a bias in his experience that he had not foreseen. “ The best players were very often the ones who played first, propelling “THE” good shot in front from the start of the polls “. This bias, quickly spotted by Mehdi Moussaïd, cannot really be explained.

According to the researcher, the most probable hypothesis would be that “ the best players are also the most passionate and motivated, so they are the ones who play first, and so they put their shots forward “. There would therefore be a link between the level of the players and their involvement in the participation in this experience, explains the researcher.