Google Assistant Most Capable Finds Loup Ventures, Cortana the Least

on July 26, 2018 at 9:43 am

The results of Loup Ventures Annual Digital Assistant IQ Test have been announced and again, it is no surprise that Google Assistant is the most capable. The study measured how correctly Google Assistant, Amazon Alexa, Apple Siri and Microsoft Cortana answered a series of 800 questions on smartphones. The assistants were graded on two metrics, the first being did it understand what it was being asked, and second, did it deliver the correct response. One reason for Google’s dominance was that it was able to understand what it was being asked 100% of the time. For the second criteria, Google Assistant was able to answer 86% of the questions correctly, followed by Siri at 79%, then Alexa at 61% and last was Cortana with only 52%.

Different Categories Reflect Different Assistant Strengths

The study also broke down the 800 questions into five different categories: local, commerce, navigation, information and command. This was to better understand the different assistant’s distinct abilities and utility. While Google Assistant was found to be the best overall digital assistant, Apple’s Siri did beat it in the command category:

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

So Apple’s push to be the audiophile’s assistant has made Siri extremely capable when it comes to music commands. Similarly, Google’s assistant also reflects the search company’s strengths as it gave a correct answer 93% of the time in the information category, the highest percentage Loup Ventures has seen in its testing of digital assistants ever. However, this pattern was not true for Amazon Alexa, which one would think would be the best performer in the commerce category, however, it placed third. Another interesting finding from Loup Ventures was that when asked a product related query, it almost always presented an “Amazon’s Choice” product, even if it did not answer the question’s intent:

For example, when asked, “where can I buy a new set of golf clubs?” Alexa responded, “Amazon’s choice for golf clubs is Callaway Women’s Strata complete set, right hand, 12-piece.” This is unhelpful because the question remains unanswered and a purchase of this type will likely require more research.

Voice Assistants Improve Across the Board

Another finding was that each assistant, except Alexa for who Loup Ventures had no past data, has improved. The most improved award goes to Siri, who improved 13 points compared to Google Assistant’s 11. Cortana even lags behind in this category, only improving 3% from the previous year.

Follow @voicebotai