Measuring chatbots, and what makes for success

Retention, in particular, is currently a particular challenge, says Ilker Köksal, co-founder and CEO of Botanalytics. This challenge is not only for “bot makers just starting out but also those with well over a million users”. 

Botanalytics is a conversational and engagement analytics platform for chatbots. “We are to bots today what web analytics were to websites a decade or so ago,” explains Ilker. The San Francisco-based company was set up only in early 2016, but is already well placed to identify, analyse and extrapolate chatbot user insights and trends across a number of sectors, in relative depth. “We launched shortly before Facebook announced its bots at the f8 conference in April 2016. After the f8 announcement, things speeded up everywhere,” he says. 

Shortly after f8, Botanalytics received funding from 500startups and now “has several enterprise [clients] with hundreds of bots” on their platform, according to Ilker. This includes areas such celebrity, education, media, finance, travel, utilities, and more, with bot examples including for Grey Group, Harper Collins Publishers, CNN, Domino’s Pizza and singer Robbie Williams’ bot.

The team is now 12-strong, focused on back-end development, product design, marketing and business development and account management. Their key focus with measurement and analyses is on enhancing human-bot communication. This is done mainly through:

● Filtering conversations for bot makers, based on engagement steps and tracking the details of conversations including rich media (video, audio, location and images);

● Measuring engagement and retention. Standards vary depending on the nature of the bot, for example with average session length for entertainment bots tending to be longer than for on-demand bots, such as for Uber. “Bot makers should think about their bots within a specific category to compare their engagement and retention rates,” he advises. 

● Identifying bottlenecks in interactions. Bots sometimes do not understand users due to varying Natural Language Processing (NLP) standards. “When people say, for example, ‘I would like to call a taxi’ or ‘could you get me a cab?’ the bot should understand these commands as a unified message.” Currently there are still limitations in this area and in order to improve their bots, bot makers need deep analysis of conversations to understand where “bottlenecks in conversations occur, identify patterns and design and iterate their bots accordingly.”

Through its multi-sector analytics and data, Botanalytics have unique insight into what currently works in the bot economy and what doesn’t, and where usage trends are pointing. In Ilker’s words, “We are creating, helping and educating the market with our insights.”

● Subscribe to our weekly FIPP World newsletter to receive more stories such as these directly to your inbox.

Having launched experimental bots, several of their clients (and prospective clients) are now at the point where they want to better know and deeper engage their audiences, with improved retention a particular objective. “This is where we come in,” explains Ilker, “to measure their bot performance in terms of conversation and user metrics, for them to be able to iterate and better design their bots on the back of robust data and automated analyses, adding to that AI learning that serve to make their bots more intelligent.”

Currently, he says, there are broadly two types of bots in the market:

● Designed with good UIs (user interfaces). These fulfil current market needs, but is limited in functionality compared to the second type.

● Smart NLPs (natural language processing), adapting to consumer expectations as they evolve. “Bots in banking are moving rapidly towards the NLP side, for example for use in their FAQ sections.”

Some of the most interesting chatbot developments for him are:

● Customer support in the banking sector, particularly around FAQs as mentioned above. An example is American Express on FB Messenger. 

● New acquisition channels for brands. For example Burberry. 

● Celebrity bots aimed at increasing fan engagement. For example singer Robbie Williams. 

● Offline to online use with messenger apps, where, for example pre-ordering a pizza, or booking a hotel or flight or ordering a cab are shifting from to chatbots (something Ilker says is poised to impact on general online habits soon). For example Domino’s Pizza.

According to Ilker a good chatbot has the following characteristics:

● It has extended capabilities in terms of comprehension. A good NLP score is a crucial metric you want to measure, getting it as high as possible.

● Sentiment is another, which is a vital consideration says Ilker. “When people are trying to communicate with bots, you’d want to know whether they are sad, happy, angry, etc. Bots should be able to detect sentiments, in order to ‘know’ the emotional context of conversation. People would want to manage their feelings on bots, and in order to do that bots need to be designed to understand the user sentiment.” 

● Conversational flows that are designed according to human behaviour patterns.

“All bot makers should be focused on these areas when designing their bots,” he says, adding that, like with the web, “it is about analysing your data and knowing your user, with the rest flowing from there”.

• Get the FIPP Innovation World Report 2017/18 which dedicates a whole chapter to bots

• We also have a long-read on developing chatbots strategy, here

In terms of useful user metrics, he believes the following are most important at this point:

● Average conversation/user

● Average session length

● Conversation steps/user

● Retention rates

● Some correlation analyses in order to evaluate combinations of metrics

His message for media companies is to get involved. “You have people who are only thinking about building bots now, but in reality many are already way beyond this point. They are thinking about things like how they can keep their audiences active, how they can be more useful, and so on”, allowing them to develop share of market and capture user time early on.

With the proliferation of chatbots people will start to bypass the normal web, getting their content from bots, and for publishers, this means “thinking about how their users can engage with their content without having (or wanting to) visit a website; what they get at different time intervals such as hourly, daily, weekly; what the notifications strategy is; and so on. And then start measuring the same things all others are looking at to see how effective your strategy is, to iterate and develop more value from there.”

He suggests retail, banking/finance and technology are good sector examples to consider for ideas, with brands such as Starbucks, Burberry, Bank of America, Capital One, Taco Bell, Estée Lauder, Sephora, Pizza Hut, American Express, and Whole Foods, and Transferwise some of his standouts.

More like this

Innovation ‘how to’ report gives the lowdown on chatbots

How to develop your chatbot strategy

John Wilpers showcases top trends included in new FIPP Innovation World Report

How chatbots are going to change the world and the media landscape