Findings from the Employing Large Language Models in Hybrid Task-oriented Chatbots for Health

The potential applications of chatbots appear to be limitless, and there are a multitude of frameworks that are available to aid in the development of chatbots. Health-related domains and behaviour change counselling can both greatly benefit from the use of chatbot applications due to their readily accessible and low-cost nature, as well as their patience when interacting with users. However, these types of chatbots must remain impartial and refrain from making judgments about users and be capable of engaging with users over an extended period of time in order to successfully facilitate a sustained and permanent change in behaviour that promotes good health. In these days, large language models (LLMs) continue to make daily progress in their ability to generate natural language. Nonetheless, their unpredictable nature imposes potential risks on their implementation in chatbots utilized in sensitive domains, such as health communication. As a result, the current health-focused chatbots rely heavily on pre-scripted systems that employ rule-based and retrieval-based approaches. Although these methods enable the creation of highly controlled chatbots, the resulting dialogue can often be perceived as monotonous and impersonal, which causes a decrease in user engagement with the chatbot.

We introduce the Hybrid Long-term Engaging Controlled Conversational Agents (HyLECA) open-source chatbot development framework, which is designed with an aim to utilize LLMs in a safe and controlled manner. HyLECA framework relies on a hybrid architecture that combines pre-scripted dialogue flows in order to maintain control over the dialogue, while also implementing retrieval-based and generation-based approaches to increase the variety and flexibility of the chatbot's utterances. The framework has been implemented in experimental settings to evaluate the effectiveness of using chatbots for interventions in two health-related fields: smoking cessation and sexual health promotion. Although these experiments primarily relied on the pre-scripted features of the system, we leverage the collected human-bot conversational data, to simulate scenarios in which multiple LLMs produce alternative utterances to the retrieved human-authored responses from the original chatbot configuration. Through conducting a series of offline and online evaluation studies, we gain insight into the applicability of state-of-the-art LLMs in these two critical health domains, and report our findings.