Alexa Skill Voice Ratings Accelerate User Rating Submissions – Analysis of Box of Cats, Tricky Genie, and Heads Up!
The Box of Cats Alexa skill currently has more user ratings than all others. A week ago, it trailed only Ambient Sounds: Thunderstorm Sounds but it added nearly 5,000 user ratings in the past week to vault to the top spot. On October 13th, Box of Cats ranked just inside the top 300 in terms of the total number of ratings with 238 user-submitted ratings for an average of 2.8 stars according to Dashbot.io data. Less than two months later it had 11,375 ratings and was ranked second overall with an average rating of 4.2. Today it stands at 16,173 ratings and is in the number one position. How did this happen?
Voicebot was the first to report on Amazon’s experiment enabling users to rate Alexa skills by voice. Currently, the only way to submit a rating for most skills is to go into the Alexa app or Amazon.com online and add a rating through a visual interface. Since most consumers interact with Alexa primarily through the voice interface, this has led to challenges for developers to secure reviews from users. Amazon’s new capability prompts users to submit ratings by voice after they have used certain skills. This has a dramatic impact on the number of reviews for the skills in a short period of time.
Over 5,000 New Ratings Per Week
Between March 6, 2018, and August 4th Box of Cats was adding about 1.75 new user ratings per week. Between August 4th and October 15th, the skill added no new user ratings. Then, in the following 30 days, Box of Cats started adding an average of 510 user ratings per week. That was followed by 12 days of new user rating accumulating at a rate of 5,200 per week. Box of Cats using ratings rose almost 5,000% in 42 days and 7,000% in 48 days.
Source: Dashbot.io Box of Cats Alexa Skill Ratings, November 2018
Other Skills Show Similar Increases
Box of Cats isn’t alone. The game Heads Up! also has received voice rating enablement around October 23rd. In the preceding 10 weeks, it has added about one new rating per week. The following 34 days, it added 482 new ratings per week. Then, in the last six days, it has added over 600 more placing it in the top 10 Alexa skills in terms of ratings.
Source: Dashbot.io Heads Up! Alexa Skill Ratings, November 2018
Voicebot’s original analysis was of the Alexa skill Tricky Genie around October 23rd. Our analysis showed a 285% rise in new skill ratings over just two weeks. The updated analysis shows Tricky Genie was adding about 70 new ratings per week in the 30 days after receiving the voice rating capability. That rate accelerated again between November 7th and November 26th. Over the entire period of October 8th through November 26th, the rate of new Alexa skill ratings was about 104 per week. That accelerated again in the last week to a rate of over 165.
Source: Dashbot.io Tricky Genie Alexa Skill Ratings, November 2018
Factors Influencing New Alexa Skill Ratings Submission Rate
Overall, we can conclude that voice ratings enablement drives between 100 and 700 new Alexa skill ratings per week for skills with moderate-to-good existing discoverability. The acceleration over the past two weeks may be associated with the significant rise in Echo device sales over what Amazon calls the Turkey 5, the five days from Thanksgiving in the U.S. through Cyber Monday. However, the rise started before that as did Amazon’s Black Friday deals which began several days before Thanksgiving.
Box of Cats and Heads Up! have characteristics that make them stand out. When you say, “Meow” to Alexa, Box of Cats is auto enabled and simply responds to the utterance. That must significantly increase discovery as users randomly test Alexa’s responses to a variety of sounds and inquiries. Heads Up! is an existing game franchise backed by Ellen Degeneres. These factors may explain why the acceleration of user ratings was faster than Tricky Genie which was created by independent skill developer, Tellables. Both Box of Cats and Heads Up! started with 4-5 times more ratings that Tricky Genie and presumably had much larger user bases. As a result, the introduction of voice ratings enabled a faster acceleration of new ratings submitted.
What this means for Developers
It is unclear the impact the number of ratings has today on driving Alexa skill discovery. Average rating is a search parameter users can apply in the Alexa skill store so that is clearly important. There is an expectation that Alexa’s discovery algorithm also takes into account total ratings. If this is the case, any skill that has access to voice ratings would have a clear advantage in discovery and Box of Cats’ developer Sean Simpson would seem to have won the Alexa lottery twice with an auto-enablement based on a keyword and early access to voice ratings.
The data presented above also suggests that there may be more third party skill usage than many believe is taking place. If third-party skill usage was very light and the current ratings reflected actual activity, the introduction of voice ratings would not have had such a large impact so quickly. The ratings reflect actual third party skill usage. What percentage of users are getting the prompt for a voice rating is unclear, but I have yet to receive one for Box of Cats so it is not everyone.
Finally, we can conclude that the hypothesis suggesting the low number of third-party Alexa skill ratings was in part reflective of the incongruity between a voice experience and ratings that can only be made through a separate visual interface. When users can submit a rating in the same context as they use an Alexa skill, the number of ratings rises quickly.
Follow @bretkinsella Follow @voicebotai
61% of Alexa Skills Still Have No Ratings and Only 1% Have More Than 100
Amazon Echo Devices on Backorder Worldwide, Echo Dots Particularly Hard to Find in the U.K.
