The Importance of Conversational Analytics: 3 Metrics to Consider

Editor’s Note: This is a guest post by Lee Boonstra, Applied AI Engineer & Developer Advocate at Google

How can you make sense of all the data you can collect working with conversational platforms? What kinds of data typically lend themselves best to conversational analytics? And why should we use conversational analytics in the first place? You should know why conversational analytics are important, and some of the best ways to monitor them.

We are all familiar with the problem; you are trying to have a conversation with a voice bot, but the virtual agent doesn’t understand what you’re saying. A default fallback message follows or worse, a wrong answer! On most occasions, the chatbot or voicebot is working fine, but it is trained to answer different types of questions, caused by the fact that organizations rarely integrate conversational analytics into their conversational design. This way, organizations don’t understand how your users are using their virtual agents.

Conversational Analytics Matter

An excellent conversational design would of course prevent the problems mentioned above from arising. According to Google’s guidelines for conversational design, the solution lies in informing users what your assistant can and can’t do. For example, an assistant has to welcome users during start-up and explain how it can help them. This way, you guide users to asking the right questions. To accomplish this, you have to have an idea of the problems the assistant is expected to solve for users. That’s why conversational design should always be based on the data that tells you something about these problems. As an organization working with chat or voice, you probably already have this data at your disposal. Is this your first attempt at building a virtual agent? Then consider data deriving from other channels like contact centers, social media, and e-mail, to retrieve all questions, topics, complaints that are relevant to customers that want to communicate with your organization.

When building such a virtual assistant, it is essential to set up analytics directly after going live. Conversational analytics is not just a nice-to-have addition to your current data collection methods, it is crucial to improving the customer experience of your bots. Conversational analytics is what gives you direct feedback about the way customers interact with your bot. However, experience has taught us that organizations often overlook this valuable data. Our advice is: don’t spend a year of endlessly improving conversations when you can learn so much from live traffic data. Even if you only collect data on a small selection of variables, you can use it to grow your bot and quickly make it smart. After that, by gradually adding more topics to your design based on this conversational data, you will be able to build a bot that has answers to any questions.

Measuring Conversational Analytics

While setting up conversational analytics, there are three specific categories of metrics relevant to designing a voice bot: conversation-related metrics, chat session & funnel metrics, and bot health metrics. Conversation-related metrics can help understanding conversations and shining a light on questions like what’s been said, by who, when, and where? To effectively monitor conversation-related metrics, data could be stored in a data warehouse: an enormous database to which several data sources can be connected. Here, you can store as much structured data as you want, whether it’s website data, website logs, login data, advertising data, or Dialogflow chatbot conversations. The more data you gather, the better you can understand and help your customers.

Chat session & funnel metrics

To obtain an idea of the chat funnel and therefore identify the structural course of conversations users have with your bot, it is important to incorporate chat session and funnel metrics into your conversational analytics. These metrics allow you to visualize the conversational route users take when they engage with your bot. Your conversational analytics could monitor the following eight chat session and funnel metrics (we shall explore this type of metrics in greater detail in the whitepaper):

1. Total usage of people using your virtual agent
2. The percentage of users whose queries matched the correct intent and the number of queries the intent was matched to.
3. Completion rate
4. Drop-off rate
5. Drop-off place
6. User retention
7. Endpoint Health, which indicates the extent to which different systems are correctly linked
8. Google Assistant: discovery information on how users came upon your Action

Bot Model Health metrics

Working with well-known bot building tools, it’s likely that your bot uses Natural Language Understanding – a form of machine learning – to understand (text) queries and match them to a specific intent, a process referred to as intent classification. If this process goes smoothly, you can be sure users will get the right answers to their questions. However, to ensure intent classification runs smoothly, there are ten Bot Model Health metrics you could monitor (we shall explore this kind of metrics in greater detail in the whitepaper):

1. A true positive indicates that the bot matches a query to the correct intent.
2. A true negative means that the bot matches a query to the correct fallback message.
3. A false positive indicates that the bot matches a query to the wrong intent and the wrong fallback message.
4. A false negative means that the bot wrongly matches a query to a fallback message, while the correct intent exists.
5. Accuracy: the ratio of correctly predicted observations to the total number of observations.
6. Precision: the ratio of positive prediction values
7. Recall and fallout: the sensitivity ratio and false alarm ratio respectfully
8. F1 score: the weighted average score of precision and recall
9. Confusion Matrix: a table used to describe the performance of a classification model on a set of test data.
10. ROC curve: A graphic representation indicating how well a model can distinguish intents.

Follow @voicebotai