You’re reading this article most likely for one of two reasons: you’re either planning the deployment of your chatbot and trying to stay as informed and knowledgeable every step of the way, or you just got done with launching it, but your chatbot effectiveness does not meet expectations and you’ve come seeking answers to your predicament.

What happened? Can I fix it? Have I just wasted a load of money on a negative ROI? Or, if you’re in the first camp, what should I do to make sure this doesn’t happen to me?

Rest easy, regardless of your reason, in this chapter of the Bot Building Series we’ll cover everything concerning chatbot monitoring, including chatbot evaluation, analysis tools, and the ever-important conversational AI KPIs.

The Importance of Detailed Monitoring of Chatbot Metrics

The simple truth is that a chatbot can’t be left to its own devices after you deploy it – technologically, we’re not at an advanced-enough level (yet) for chatbots to analyse their own performance against a pre-programmed set of metrics, which in turn necessitates constant observation and iteration. Indeed, if you’ve been following the previous articles in our Bot Building Series, you already know how crucial it is to stay on top of things, and even have a dedicated team responsible for measuring the chatbot’s effectiveness. Honestly, all those marvellously-sounding statistics powering the chatbot hype train, like how they could save businesses $8 billion annually, should come with an asterisk underscoring the necessity to keep measuring the chatbot accuracy.

A chatbot that is constantly worked on leads to overall better customer experiences, and a higher chatbot containment rate – the share of conversations handled entirely by the bot without needing to hand the client over to a live agent – means your investment is paying off. Plus, you get a nice bird’s eye view of your customer’s behaviour patterns for future chatbot tweaks, and if there’s a tech issue concerning the chatbot, keeping an eye on it regularly will let you swoop in and deal with it handily. This is particularly important when considering any major security threats or breaches, as the chatbot here could even be seen as an alarm system for those kinds of issues.

Let’s switch sides for a second and pretend you don’t bother checking up on your chatbot metrics. What can go wrong? Actually, quite a lot. A chatbot whose conversation flow was not set up correctly could be running buck wild, consequently squandering any potential returns, even when used in industries that benefit from chatbots the most. One infamous example of such negligence is that of Tay, a chatbot released by Microsoft in 2016, who was quickly indoctrinated and turned into a racist due to users bombarding it with inappropriate tweets. PR crises and controversial topics like this require stalwart monitoring to prevent the chatbot from spiralling out of control. Of course, there are other problems to consider as well, such as the chatbot simply providing customers with factually incorrect information, which, depending on the context, could range from a small problem to a massive catastrophe. The latter is what happened very recently when Google debuted Bard in February of 2023 and subsequently lost $100 billion of market value virtually overnight when the bot fumbled quite badly during its first-ever demo.

Possible Reasons for Your Chatbot’s Underwhelming Performance

Let’s put our detective hats on and try to see why a chatbot might not be doing so hot:

• Poor training: this one should be quite self-explanatory. A poorly trained chatbot, i.e. one whose design was based on insufficiently thought-out conversation flows and/or inappropriate language models, lacks the proper capacity to efficiently and effectively answer client questions and service their requests. Fortunately, this is quite easily fixable if you analyse the success metrics thoroughly enough and employ the right corrective measures.
• Bad user experience: the fundamental principle of design involves putting yourself in the position of your client. Understanding their needs and wants is and always will be the key to a successful business. How you design the user experience of the chatbot responses obviously matters a great deal, perhaps even more than the other chatbot metrics, since it isn’t really trackable through any chatbot analytics tool or platform. If the chatbot lacks personality and character, causes mistrust, or if interacting with it is simply frustrating due to poor conversation design choices, you might want to take a closer look at the user journey.
• Frequent maintenance issues: stuff breaks sometimes, and that’s normal. However, when your chatbot is more inaccessible than not, either due to being down completely, or frequently freezing or losing network connection, you’ll need to investigate and address this matter as quickly as possible.
• Improper channel implementation: a chatbot implemented on WhatsApp will behave differently in that environment vs. when implemented on Facebook’s Messenger or webchat. This seemingly innocuous mistake can be easily overlooked, so make sure you avoid it and properly adapt the chatbot’s code to the chosen channel’s interface.
• Limited functionality and integration: It is important for users to have the option to speak to an agent if they need to. We don’t recommend limiting this option. Chatbots are great in handling repetitive processes, but, at the end of the day, processes that require decision making, creativity or expert knowledge should be handled by humans. That is why when building bots you should ensure smooth integration with agent systems, CRMs, databases, etc. If your chatbot lacks that or other features, such as retrieving client account information from a database, processing transactions, or presenting your range of products or services, your clients will avoid interacting with the bot altogether and reach out to the human agents for even the most mundane tasks. So, if your chatbot metrics suggest a very high handover rate, look into it.

Collecting Relevant Data

Solutions to business ailments don’t grow on trees. You and your chatbot design team will have to rack your brains while poring over the right data to seek out areas for improvement, come up with adequate ways to address chatbot performance and prioritise all issues according to your business needs and the severity of the chatbot’s shortcomings. That is quite a handful, considering all this has to be done to prevent your chatbot from becoming dead weight.

So, you might be wondering… how do you actually check the accuracy of your bot?

For that, you need access to data. In an ideal world, you’re using a platform with a crystal-clear dashboard that, at a glance, offers you an overview of all needed KPIs for a virtual assistant. The ones that matter most for chatbot optimisation are:

• the rate of automation (or how many questions the bot handles in comparison to human agents),
• percentage of unrecognised questions,
• customer ratings, also known as NPS,
• average conversation length.

Assuming the platform you use can actually provide you with these datasets, for proper chatbot evaluation focus on tracking your conversation statistics and figure out where you can improve. A good starting point is looking at the most frequently and least frequently brought up topics. For the former, aim to increase the intent accuracy and automation rate as much as possible to reduce the load placed on your customer service reps. The latter, however, will require a more thorough look to understand your customers – do they simply never bring up the given subject, or do they omit your chatbot and ask to be transferred directly to a human agent to discuss it? Every lead is worth investigating to both capitalise on the benefits you’re missing out on and to continuously improve your chatbot.

Key KPIs to Track in Your Bot-Building Platform

Once you have access to a plethora of numbers, percentages, and graphs, picking out the right combination of trackable metrics can be overwhelming, let alone singling out and establishing the actual key performance indicators that will make your chatbot great.

Looking for a cheat sheet? Below you’ll find a great example of what an intuitive chatbot analysis dashboard should look like, based on our platform: SentiOne Automate:

1. Conversations volume

Gives you an overview of all conversations that occur in a given period, and lets you filter by user, bot, or all messages, as well as the total number of users.

SentiOne Analytics: Conversations volume

2. Tags

With a fully customisable tag system that lets you define and flag certain topics for future reference, you can track the prevalence of these tags, create funnels, and monitor your KPIs with extra precision.

SentiOne Analytics: Tags

3. Intents

A simple way to analyse all the different intents your chatbot has recognised over time.

SentiOne Analytics: Intents

4. Redirects

This statistic tracks whenever a bot is asked to perform a handover and allow the customer to speak to an agent, e.g., due to unsatisfactory intent recognition.

SentiOne Analytics: Redirects

5. Unrecognised messages

A quite crucial metric, giving you insight into those user messages the chatbot couldn’t recognise the intent of. This signals that you may need to program additional topics into your chatbot’s NLU capabilities.

SentiOne Analytics: Unrecognised messages

Tips for Chatbot Improvement

Let’s focus on actionable methods to enhance your chatbot’s success metrics.

First up, if you’re not already pouring heavy effort into the UX of your product, you might still be living in the early 2010s. Why? Let the numbers do the talking: every dollar invested into UX can bring up to 100 dollars in return. It goes without saying – crafting a chatbot that offers a pleasant user experience makes users come back, increasing the volume of conversations handled by the bot. Here, the importance of knowing your audience as an essential step of chatbot building comes back once again. Not only do you have to tailor your bot’s responses to your client base and their needs, but you also have to be aware of the potential design limitations of the channel they use most frequently and be ready to switch to another channel if they do so, too. Plus, don’t forget to promote and market your chatbot. To steadily increase your chatbot engagement rate, one email blast to commemorate deployment won’t do. Keep your customers tactfully informed, but don’t shove it down their throats.

To make your chatbot more independent and limit handovers, you’d do well to monitor conversations directly. Check for the most frequent reasons why customers end up talking to agents, and look for holes in your chatbot’s design to identify automation opportunities you’re potentially missing out on. Consult with your agents as well – they’re bound to offer a wealth of knowledge in this matter. If all your important dialogue flows are optimised, consider automating your transactional processes, since a lot of times, customers ask to be handed over simply because your bot lacks that functionality.

You can limit unrecognised messages by further training your chatbot in slightly more particular ways. By analysing conversation transcripts, you’ll recognise situations where your chatbot fumbles, for example when encountering slang expressions, tough to understand accents (if you’re working with a voice bot), or when non-native English speakers try to communicate with the chatbot, and neither party can easily understand each other’s intent. But before you dive head-first into an unproven solution to problems of this category, run some tests with real people first. An expert-led heuristic evaluation may often be hit-or-miss while testing with real users, though more time-consuming, helps avoid those misses and wasted funds. Don’t try to guess what your users talk like. Rather, reach directly to the source for help, and make data-driven decisions.

Overall, if your aim is to increase your NPS and customer satisfaction, employ these methods often, and constantly re-train your chatbot based on real feedback – a survey at the end of each conversation may seem trivial, but it does go a long way.

Improve Chatbot Metrics with SentiOne Automate

Want to learn more about chatbot analytics? SentiOne Automate offers you actionable insight into your customer’s tendencies when interacting with your chatbot. Our platform has robust built-in analytics with real-time data. Forget about external software or performance analytics – we’re dedicated to giving you the quickest access to all the data you need to evaluate your chatbot’s performance, and make changes easily to increase your ROI. If you have more questions about this or other Automate features – don’t hesitate to contact our team for help.

Please note that when discussing chatbots, we also mean voice bots. The primary distinction between them is the mode of communication, but they share the same underlying technology, with the addition of voice-specific technologies. Consequently, both solutions have similar use cases. Therefore, the principles discussed in the Bot Building Series can be applied to both chatbots and voice bots without any significant differences.

Article Summary

This article discusses the importance of monitoring and improving chatbot performance. It highlights the need for constant observation and iteration to ensure the chatbot meets expectations. The article explores possible reasons for underwhelming performance, such as poor training, bad user experience, maintenance issues, improper channel implementation, and limited functionality. It emphasises the collection of relevant data and suggests key performance indicators (KPIs) to track, including automation rate, unrecognised questions, customer ratings, and conversation length. The article provides tips for chatbot improvement, such as focusing on user experience, monitoring conversations, automating transactional processes, and training the chatbot to handle specific situations.