In this article of our blog series, AI Bots – A Manager’s Handbook, we’ll explore the most meaningful chatbot metrics to track in order to measure and optimise performance. We’ll assume you have a chatbot in production or in development, but if you’re just getting started, we have plenty of helpful tips for you.

Computer science pioneer Grace Hopper once said: “One measurement is worth a thousand expert opinions”. That’s just her opinion, of course, but it’s a wise one. As the owner of your organisation’s chatbot or voice bot, you will undoubtedly hear the opinions of others. Some will love it; some may hate it. Whatever the sentiment, your best defence, offence, and source of truth should be a handy KPI dashboard that doesn’t lie. But collecting meaningful conversational AI KPIs is no trivial task. Let’s deconstruct that chatbot analytics dashboard to find the best chatbot performance metrics.

If you are already owning/managing other digital channels, it may be tempting to cut-and-paste a dashboard from another digital channel such as your website, social media, or a mobile app. That would show traffic, uptime, and some form of conversions. The problem with that approach for bot evaluation is that it lacks depth in measuring the quality of interactions with bot users.

So, let’s try another take. What if you measured your chatbot/voice bot performance using the measures we use for an employee? Your customer service reps will undoubtedly have a clear role description, measurable goals, and detailed performance metrics. You can mirror some of these in the KPIs for a virtual assistant. We’re getting closer now.

The best collection of chatbot/voice bot metrics likely lies somewhere in the middle – measuring the effectiveness of a bot as a support channel, while also measuring the quality of customer interactions with your bot as a service representative.

Why Is it Important to Measure Voice Bot and Chatbot Performance?

Of any digital channel, chatbots and voice bots are likely the least set-it-and-forget-it. To harvest the most business benefit, it requires consistent chatbot analysis to monitor and improve your bot’s performance. The effort is worthwhile, as the potential benefits include:

• Improved customer experience, illustrated by high issue-resolution rates and positive customer feedback
• Customer insights, informed by data-rich logs of chat activity
• Cost savings, through servicing more customers with less staff
• Increased revenue, as you layer in promotional and upsell capabilities
• Measurable ROI, supporting your case for continued investment.

That’s the upside. It’s also important to measure chatbot effectiveness to avoid or contain potential risks. Issues with your chatbot can lead to some unpleasant consequences, including:

• Negative PR: Chatbots and voice bots don’t naturally have the sensitivity that humans do. With AI being a hot media topic, a bot mishap has the potential to become an embarrassing news item. This is particularly true when users broach risky or controversial topics, such as a healthcare chatbot handling questions on abortion.
• False or misleading information: The speed and scale of bots is such that if your chatbot or voice bot is drawing from incorrect information sources, a mistake can be replicated several times if undetected.
• Customer frustration: If your bot is unable to interpret a customer’s intent correctly, they can find themselves in a disappointing circular conversation that doesn’t solve their problem.
• Impact on support teams: Poor chatbot/voice bot performance can increase the burden on human reps, potentially negatively impacting morale and turnover.
• Low value and ROI: You likely identified some chatbot success metrics when building the original business case. Without chatbot KPIs, you’re unable to prove that your chatbot is delivering business results.

These factors can be avoided early by following best practices for building chatbots. But no matter how perfect your execution, you won’t know unless you measure.

Essential KPIs to Track Chatbot and Voice Bot Performance

Regardless of your industry, you will want your chatbot KPIs to capture two key areas. The first is the impact on customer experience, where you will use a range of statistics for measuring bot accuracy. The second is the impact on business results, which features chatbot analytics metrics for cost savings and revenue generation.

User Metrics: Improving Customer Experience

Volume & Engagement KPIs

Interaction rate measures the percentage of users who interact with your bot. So if 1,000 people came to your website and 200 interacted with your chatbot, your chatbot interaction rate would be 20%. This tells you how many people are using your chatbot.

Engagement rate measures the average number of messages shared between your chatbot and a user. Chatbot engagement rate shows how deeply visitors are interacting with your chatbot.

Bounce rate: Bounce rate measures how many visitors enter a chat with your chatbot but leave without interacting. Same applies to voice bots: when a user begins the call, hears they are speaking to a voice bot and hangs up.

Average conversation time measures the average duration of a chatbot or a voice bot session. For more meaningful insights, you can segment this by sessions that were successfully resolved and those that were not.

User retention: Measuring repeat visitors – those who used the bot more than once – can be a useful indicator of the quality of your voice bot/chatbot experience. Ideally, you should see a steady growth curve as more customers find value and revisit the bot.

Quality KPIs

Success rate: This KPI shows the number of chat or call sessions initiated compared to those completed successfully. This measure requires an input from the user, usually prompted by a bot question such as “Was I able to resolve your issue?” If you’re wondering how to check the accuracy of a chatbot or a voice bot, the success rate is often a great place to start.

Automation rate: This KPI measures how your chatbot is complementing your existing customer support team by illustrating how many customers the chatbot is handling independently. It typically includes some additional measures, such as:

• Fallback/containment rate: The fallback rate is the percentage of queries your chatbot could not resolve. Chatbot containment rate is the opposite, those that it did resolve.
• Deflection/handoff rate: The percentage of customers handed off to a live agent.

Call wait-time measures the delayed customer experience while waiting for phone or chat support. You’ll want to ensure you’re measuring this statistic in the context where your chatbot is available to reduce waiting times.

Customer satisfaction: Most organisations have some form of NPS survey or customer satisfaction score. Not only should bot interactions be included in the survey questions, but your chatbot or voice bot can help collect the survey feedback too.

Business Metrics – Increasing Business Efficiency

Cost Savings

Customer support cost reduction: To arrive at this impactful KPI, take the total number of tickets successfully resolved without human intervention, divide by the total number of tickets, and multiply by 100 for a percentage. If you have a benchmark for average cost-per-call with a human agent, you can multiply that amount to assign a business value to the resolutions your chatbot provided.

Operating cost reduction: Chatbot-delivered customer support can result in significantly reduced staffing costs. To truly obtain an accurate measure of  operational cost savings, you should consider the all-in costs of your customer support staff. That includes benefits, vacation, training, and sick time. Accounting for the cost of employee turnover may prove more difficult, but is nonetheless an important consideration, particularly as call centres often have the highest churn rate in an organisation.

Having a chatbot or a voice bot support the most common and repetitive customer inquiries can have many benefits for your customer support team that go beyond cost reduction. Bots can free agents to deal with more complex and higher-value customer support needs. With more challenging and rewarding work, and less repetitive tasks, you may find that employee engagement rates and morale improve significantly.

After-hours support: How many customer service tickets or calls did your chatbot or voice bot resolve outside of standard customer service hours?

Revenue Generation

Lead generation: If you haven’t already planned out how your chatbot can contribute to sales growth, start with our article on how to generate leads with a chatbot. Lead attribution can be an analytics headache. There are three loose categories of sophistication:

• Basic: A numeric count of a specific call-to-action (sales meetings booked, lead capture forms completed, etc.)
• Intermediate: Attribution and tracking to translate captured leads into actual sales, resulting in a dollar value for chatbot-driven sales
• Advanced: Linking client lifetime value (the full spend of an average client over time) with your chatbot sales in order to capture expected incremental sales in the future.

Upselling and cross-selling revenue: It’s usually true that your easiest next sale is from an existing customer. Whether you’re conducting outbound calls, or surfacing new promotional offers in chatbot or voice bot conversations, you’ll want your success metrics to measure the conversion rates and incremental sales lift.

Conversion rates: Your chatbot’s sales effectiveness will depend on the allure of the offers and campaigns it is equipped with. Measuring click-through rates and conversions on an offer or campaign basis will provide valuable input to improve your campaigns.

Channel metrics – assessing channel effectiveness

If you agree with our belief that chatbots are meaningful brand touchpoints, you may already have your chatbot active through multiple digital channels such as webchat, voice, social media, or collaboration tools.

In this case, several of the KPIs listed above are worthwhile segmenting by digital channel. This will give you valuable insights into the user behaviour and chatbot effectiveness in each respective channel. You will be able to tell at a glance, for example, that 80% of your leads are coming from your chatbot on Facebook Messenger.

Real-Time Bot Monitoring with SentiOne Automate

Gain access to actionable customer insights and quickly assess bot performance with SentiOne Automate. Our chatbot platform has rich analytics built into the platform, so you don’t need to fuss with extracting data and displaying it in external chatbot analytics tools. With real-time updates, your dashboard will always be up-to-the-minute. To learn more about our powerful conversational AI chatbots and voice bots, book a demo.

Article Summary

Discover the vital KPIs for measuring and enhancing chatbot and voice bot effectiveness. The article emphasises the significance of assessing both customer experience and business results. It suggests using metrics similar to those for evaluating employee performance. The importance of measuring voice bot and chatbot performance is highlighted, with benefits including improved customer experience, cost savings, increased revenue, and measurable ROI. The article outlines essential KPIs to track, encompassing user metrics, quality indicators, cost savings, and revenue generation. It also addresses channel effectiveness assessment.