Siri Does Better Than Expected in Voice Assistant Face Off, But Google Still Leads
Apple Siri trails voice assistant competitors in many capabilities categories. That is well documented here and elsewhere. However, in terms of answering questions, a new analysis from Loup Ventures shows Siri performance exceeds expectations. Google Home led in every category. This is not a surprise. Google’s knowledge graph offers distinct advantages in query-based interactions. Siri accessed through HomePod was the weakest overall performer, but was second only to Google in two categories: local and commerce.
Siri Performance 52.3% Correct
The test was based on 782 different queries. Siri answered 52.3% of the queries correctly, while Google Assistant came in at 81%, Alexa at 64% Microsoft Cortana at 57%. Cortana was tested using the Harman Kardon Invoke smart speaker. A Loup Ventures blog post by Gene Munster commented:
Siri understood 99.4% of queries and answered 52.3% of them correctly. This places HomePod at the bottom of the totem pole in terms of AI assistant performance. Siri is particularly strong in Local (where can I find a good cup of coffee around here?) and Commerce (help me buy some new shoes.) queries, handily beating Alexa and Cortana, but still falling short of Google Home in those areas. Overall, Siri performed above our expectations given the limited scope of HomePod’s music focus.
It is also worth noting that Munster reports Siri correctly understood 99.4% of queries. A December study by Loup Ventures showed that Siri on a mobile device also recognized 99% of queries and successfully answered 75.4% of them from the same question set. Some of these are commands that Siri can no doubt complete on a smart phone (e.g. making a phone call) that fail today as a non-supported feature on HomePod.
Siri Benefits from Google Search Integration
Munster points out that Siri has a self-imposed disadvantage due to a limited number of knowledge domains supported (N.B. expanding these domains was Voicebot recommendation #2 for Apple). His hypothesis is that Apple can improve its score quickly by simply expanding domains. That may be true, but Siri uses Google for search so you should expect it to perform better. It is likely that the Google search integration is responsible for Siri outperforming Alexa and Cortana for local and commerce-based queries. For some reason this Google alliance doesn’t carry over into the “Information” queries in the test. For navigation and command, Apple is likely using its own capabilities such as Maps and HomeKit which are simply not as mature as offerings from the other companies.
Alexa and Cortana both use Bing for knowledge graph search and that puts them near parity for “Local,” “Navigation” and “Information.” Alexa is clearly using its own knowledge graph for “Commerce” and “Command” where it handily outperforms Cortana. Munster’s analysis suggests that Siri would match or surpass the overall scores for Alexa and Cortana if it improved in “Navigation” and added features for calling, email and calendar. That truly would be news.
Could iOS Developers Close the Bigger Gap?
This data is very useful, but it comes from a fairly narrow test. It doesn’t attempt to go beyond common queries and commands. The real advantage that Amazon holds today is its nearly 30,000 custom Alexa skills. Thirty to forty percent of device owners use custom skills on a regular basis. These voice apps offer a breadth of capabilities that go well beyond answering a list of common questions. Custom skills also mean that Alexa — and Google Home with its nearly 2,000 Assistant apps and “a million things to try” — can offer more value to users. Apple claims to have 16 million iOS developers today. Could they be incentivized to quickly build thousands of Siri apps?
The Loup Ventures findings are good news for Apple and for HomePod owners. The data show that the gap between Siri performance and other voice assistants may not be as wide as is commonly believed. And, Loup Ventures also has high regard for HomePod’s ease of use, microphone performance and sound quality. If Siri can add some knowledge domains in 2018 and motivate its iOS developer to support HomePod, Apple could quickly become formidable in the smart speaker category.
Apple HomePod Has a Privacy Flaw That No One is Talking About
Apple Should Do 3 Things in 2018 to Make Siri Better for Everyone
