Chatbots and How to Test Them

Machines that are performing automated conversations are expected to bridge the gap between companies and end users via the most natural method for human interaction – a dialogue. It is estimated that more than 200,000 chatbots are available today and the expectations are that 2018 will mark a huge step forward in the domain.

Every new and innovative area places challenges that require a lot of creative and out of the box thinking. Chatbots are not an exception to this. They are much more than just applications with predefined questions and answers. Chatbots are highly complex software solutions that combine the advantages of Machine Learning algorithms, Natural Language Understanding and Artificial Intelligence in order to convincingly simulate how a human would behave as a conversational partner.

The user intent, the possible conversation flows, the domain specific terminology and the correct data extractions are things that have to be considered when defining tests for the product. One of the main goals of any chatbot is to provide a scalable way to utilize messaging as a channel for interaction with customers. That is why the product is constantly being enriched with new business logic. In those circumstances test automation is a must have in order to ensure a scalable delivery process. The main focus of the presentation will be on Functional, Non-Functional and Automation Testing and what unexpected challenges those testing types introduce in the context of chatbots.
I will share the experience of our QA practice in a project that is developing a cognitive advisor trained to answer any user questions related to the supply chain area.