Google AI Improves 78 Percent in Two Years, 2x Smarter Than Siri According to Chinese Study

A 2016 study of Artificial Intelligence (AI) systems published in June in the Annals of Data Science showed that Google AI has improved 78% in just two years. The research was conducted by Feng Lui, Yong Shi and Ying Liu and includes data from both 2014 and 2016 analyses. The researchers have affiliations with various schools within the Chinese Academy of Sciences and College of Information Science and Technology at the University of Nebraska. Google and Baidu both top the list in the 2014 and 2016 analyses.

Humans Still Have a Big Edge Over AI IQ

The study attempts to determine IQ level and assign grade level intelligence to a variety of AI systems including Google Brain, Baidu Brain, Bing, Siri and others and then compares that the human intelligence. It proposes a six characteristic model with seven levels of intelligence to rank the sophistication of each form of intelligence. The results show Google Brain as the most sophisticated artificial intelligence with an absolute IQ of 47.28 compared to an average six-year-old human with 55.5. Human IQ scales are typically adjusted by age so this is not a clean comparison, but it does provide some perspective.

Apple Siri Has Half a Google Brain

Many people have said that Siri just isn’t as smart as Google Assistant and now they have scientific evidence. Well, sort of. Google Brain is leveraged by Google Assistant, but it isn’t actually Google Assistant. And, the study authors repeatedly refer to AlphaGo which is a DeepMind project. Google Brain and DeepMind are both under Google ownership and direction, but technically are different projects and code bases. Yes, Siri is about half as smart as Google in this study which anecdotally seems about right. Siri is a wonderful application that can execute a number of tasks. However, its domain knowledge is very limited and it fails to maintain even a modicum of context about an interaction or the user.

Baidu Takes Two Spots Behind Google

Baidu also had a good showing in the study coming in second to Google in the 2014 analysis and taking both the second and third spots in 2016. During the period, the Absolute IQ of Baidu Brain rose 40% to 32.92. That is about half the rate of improvement as Google. Displacing Baidu Brain in the number two slot is Baidu’s Duer. Voicebot readers will recall a recent article about DuerOS which is a cloud-based voice assistant service for Mandarin language users.

Also included were Micrsoft’s Bing and Xiaobing. The latter is a chatbot and the former a search engine. Both fall between Google and Siri with Xaiobing only slightly edging out the Apple service. Voicebot has reached out to the researchers to learn if we should expect another study in 2018 and if Amazon Alexa, Microsoft Cortana and Samsung Bixby might be considered for analysis.

Should We Fear the Singularity?

There is a comment in the abstract that says, “The view that artificial intelligence remains a threat is persistent. An open question is that if we can compare the development levels of artificial intelligence products and systems with measured human intelligence quotient (IQ), can we develop a quantitative analysis method to assess the problem of artificial intelligence threat?”

The study goes on to suggest that artificial intelligence advancements can take two different paths in comparison to the slow march of human intelligence gains represented by curve B in the chart. Curve A would represent a super intelligence by computers that outstrips human intelligence and improves at an increasing rate. By contrast, curve C shows a gradual improvement of artificial intelligence that eventually become asymptotic to human intelligence and never surpasses it. The study authors suggest that by building a model and regularly measuring AI advancement compared to human IQ, we can better understand whether curve A or curve C is the likely trajectory and where the difference lies at any given point. This is interesting because it posits that the headline grabbing super intelligence fears are just one potential outcome and we need to establish measurement systems to understand whether that theory has real-world evidence to support it.

The Value of Measuring Relative Changes in Performance

Even average adult human intelligence exhibits more than twice the performance of the leading AI in the study. The number most people will watch is to see if AI exhibits another 78% rise between 2016-2017. That would suggest we might be heading toward Curve A and an AI super intelligence. A more modest rise might assuage fears suggesting Curve C is more likely. With that said, we know very little about the methodology used to calculate the Absolute IQ of these systems. The study likely has value simply by showing relative changes in AI capabilities even if the absolute findings may be more easily contested. We look forward to more studies of this nature.

Baidu DuerOS To Bring Mandarin Voice Assistants to Hundreds of Products