Nearly All Google Actions Restored After About 3 Days of Downtime with Danish, French, Norwegian, and Chinese Still with a Sizable Outage
Editor’s Note: New information suggests these activities by Google were prompted by security vulnerabilities in Google Actions discovered by SR Labs. This article was updated on October 21 at 8:55 to reflect this new information.
The ongoing saga of the Great Google Action Disappearance seems to be winding down (see previous stories for the full context including Google Action Disappearance Wider Than Previously Thought with More Than 80% Down in English, Korean, and Dutch; and Thousands of Google Actions Mysteriously Disappeared from Online Directory and are No Longer Available).
Thousands of Google Actions worldwide that suddenly disappeared on Wednesday, October 16th, started returning to the online Google Action directory and becoming accessible through Assistant again yesterday and overnight. The charts below were created by Emiel Langeberg based on a comprehensive analysis of Google Actions by language worldwide. The baseline is an analysis conducted on August 2nd and the updates are for October 18th (first chart) and October 20th (second chart). The percentage decrease or increase illustrates the extent of the impact and restoration respectively. Langeberg commented in a message to Voicebot:
Most Google actions appear to be back online again. A new analysis shows that most languages have roughly the same amount of live actions again, compared to the 2nd of August. The Nordics seem to be still recovering, as well as France and China.
Data on October 18th showing the decrease in available Actions by language compared to August 2nd.
Data on October 20th showing the change in available Google Actions by language since August 2nd and depicting strong contrast with results from two days prior.
Normally, you would expect some growth in each language over an 11-week period, the time between August 2nd and October 18th. The October 18th chart depicts a considerable decline in total Actions by language which is an indicator of how widespread the issue was and corroborates Voicebot’s analysis of decline among English language Actions. The percent growth bars moved back to positive from negative territory between October 18th and 20th and reflects the restoration of the third-party Google Actions that had disappeared.
Most Google actions appear to be back online again. Langeberg’s most recent analysis shows that most languages have roughly the same amount of live Actions again, compared to the 2nd of August with Danish, French, Italian, Japanese, Norwegian, Swedish partially restored but still with a remaining backlog. Google appears to not yet have started restoration for Google Actions in Chinese. Langeberg notes that the spike in Thai is because of the small base of Actions in August and the creation of a few dozen new voice apps in the intervening months.
Growth in Google Actions Also a Story
Hindi is obviously a standout in this chart. It clearly had a dip on the 18th, but a much bigger spike post-restoration. This most likely suggests there was a large influx of new Google Actions in Hindi between August and October. It is not clear why this would have happened, if it is an anomaly, or if there is something else going on with more first-party Google Assistant Actions. Google Assistant added Hindi to Google Home devices in November 2018 and it became available on many Android OS smartphones earlier that year. Further analysis is needed to better understand the scale of Google Action development in Hindi.
Notably, German, English, and Spanish all show strong growth in Google Actions over the August-to-October period suggesting increased third-party developer attention to Google Assistant before the incident occurred. As the restoration of missing Google Actions continues in other languages, we may well see growth in those languages as well. That is an interesting side-note of this incident. It may have exposed much faster growth in new third-party Google Action development than previously acknowledged.
Timeline of the Great Google Action Disappearance
- Sometime in July 2019 – SR Labs informs Google that there are security vulnerabilities in Google Actions that could lead to malicious passive listening by hackers or phishing scams.
- October 16, 2019 – On Wednesday, several Google Action developers noticed that their Actions were no longer functional and their Google Assistant online directory pages were gone. The first reported outage was at 11:00 am CST and it was identified through a user complaint to a third-party Action developer though the Google Action dashboard didn’t indicate any problems.
- October 16-17 – Throughout Wednesday and Thursday, Action developers submitted support tickets to Google about the disappearing Actions, shared comments in Reddit and Slack groups while Vociebot contacted Google for an official explanation.
- October 17 – Late Thursday, Google provided a post in a subreddit frequented by Action developers saying that there was a “comprehensive review” underway related to “developer policies.”
- October 17-18 – Shortly after this, some developers that had submitted support tickets received a nearly identical message by email with one additional line saying, “Note this process may take up to a few weeks.”
- October 18 – Voicebot conducted a partial analysis of the Google Assistant online directory and identified over 2,000 Actions that were available in January were no longer available. This published early morning on the 18th.
- October 18 – Shortly after Voicebot’s article, Emiel Langeberg published a comprehensive global analysis showing the percentage of Actions available in August 2019 that were no longer available as of Friday, October 18th confirming the impact was global and that many thousands of Actions were affected.
- October 19 – Voicebot interviewed Langeberg and published the results of his analysis from the 18th as well as reviewed newer data for the 19th. Voicebot analysis also identified that some Actions that had disappeared as of Friday were newly available again and Langeberg’s updated data confirmed that this seemed to be true mostly for English-language Actions. Most Actions were not back online regardless of language but there was clearly some restoration taking place.
- October 20 – As of the morning of October 20th, Langeberg produced a new data set indicating that most Actions worldwide seems to be fully restored to the online directory except for significant blocks in Danish, French, Italian, Japanese, Norwegian, Swedish, and Chinese. However, each of those languages except for Chinese, showed considerable restoration over the previous 24 hours. Some Actions in a variety of languages remained down with no indication on when they will be restored.
- October 20 – Berlin-based SR Labs publishes blog post about security vulnerabilities common to both Alexa skills and Google Actions and the public statement by Google is very similar to its statement to Google Action developers on Thursday. Both Amazon and Google say the vulnerabilities have been addressed and any voice apps with these types of risks were removed from circulation.
- October 21 – Voicebot learns that SR Labs informed Google about the vulnerabilities “more than three months ago” which would put the disclosure timing around July 2019.
How Will Developers Respond
As this unexpected incident appears close to resolution, the remaining question is how third-party developers will react. Many expressed pointed frustration in online forums and social media. There was clearly a negative reaction to the incident itself, but that seemed to be heightened by the lack of information coming initially from Google and then with the nature of the explanation the company offered. Identifying the issue as linked to a “policy review” suggested that this was known by Google in advance. That would mean the company could have proactively notified developers to prepare for a temporary outage.
However, Voicebot learned that Google’s own monitoring dashboards didn’t initially identify the problem even 24 hours after it began. It appear that it was the flurry of support tickets from developers and maybe an inquiry from Voicebot that finally caused Google to recognize the issue was widespread. The problem may have originated in a “policy review” process but it appears that Google didn’t expect that activity to take down thousands of Google Actions worldwide.
Voicebot also noticed in its analysis that large brands were among the first Google Actions to be restored. There was some clear prioritization taking place in bringing higher profile Actions back online quickly. As we discussed in Voice Insider Issue #60 this week, those big brands are important to Google’s strategy for a robust ecosystem filled with the content and services consumers want access to through Assistant. Google wants all of its third-party Action developers to have positive sentiment toward Assistant, but it likely sees bigger risk in frustrating popular brands than the smaller, independent developers.
Provided the restoration continues at its current pace, this is not likely to have a big impact on support for the Google Assistant ecosystem. Yes, a few developers will leave the ecosystem as some did when Google changed its Action discovery algorithm and significantly reduced user traffic for implicit invocations. However, most will recognize that Google Assistant has a large and growing global user base and will not want to be left out of those prospective consumer interactions. They won’t forget the incident and it may make them warier going forward about supporting the ecosystem, but few will want to abandon Google Assistant entirely. It is likely that Google can greatly reduce developer concerns by offering a more comprehensive explanation of the incident. Whether that will be forthcoming is unclear.
How Will Consumers Respond
Google’s bigger concern would have been a consumer revolt when their favorite Actions were suddenly unavailable. This apparently did not happen at any meaningful scale. Certainly, there were some complaints directed toward Action developers that led to the initial support tickets. However, Voicebot was unable to find any mention of unavailable Actions on Google Home and Assistant forums on Reddit. There were plenty of complaints during the past week, but none about mysteriously unavailable Actions.
It is uncertain who consumers would hold responsible for this incident. If an Action is suddenly unavailable through Assistant and no longer listed in the online directory, would a consumer assume it was caused by a Google problem or that the developer decided to remove the Action? More likely the latter. So, this could explain the lack of complaints online. Another reason may be that most consumers are not using third-party Google Actions frequently enough to have noticed. If true, that is a bigger concern for developers than 3-4 days of unexpected downtime.
Follow @bretkinsella Follow @voicebotai
