User Experience with Conversational Agents that Understand Conversation Contexts

Abstract

With the advent of ChatGPT, the interaction capabilities of conversational artificial intelligence (CAI) with human daily life have gradually increased. CAI is innovatively changing the user experience (UX) by producing human-like conversations. In this study, we investigate how the newly introduced characteristics of CAI affect UX. We focused on context understanding levels and conversation types. Context understanding refers to the ability to comprehend preceding and subsequent situations within a conversation. Such a feature provides a continuous and expanded conversational experience to users, enhancing the level of anthropomorphism and further improving UX. A CAI-based agent can provide sophisticated answers as if it thinks and makes decisions similar to humans. It can considerably affect human–machine interaction, contributing to persuasiveness and logical validity. Meanwhile, excessive reliance on the decisions of a CAI can cause problems such as dependence for making critical decisions or accepting information provided by AI without a critical perspective. We examined the UX provided by the conversation types that generate human-like responses. The independent variables examined in this study included context understanding level and conversation type. The context understanding levels were categorized into “agents that do not understand the context,” “agents that understand the context of the current conversation,” and “long-term memory agents.” The conversation types were classified into “supplementary conversation” focusing on information exploration and “contributory conversation” focusing on opinion exchange. After holding a conversation with a conversational agent (CA), the participants evaluated the perceived engagement, perceived satisfaction, perceived trust, perceived human- likeness, and perceived effectiveness. The experiment was conducted using the Wizard-of-OZ technique with 30 participants. The participants provided high values for perceived engagement, perceived satisfaction, perceived trust, perceived human-likeness, and perceived effectiveness of agents when considering a high context understanding level. Furthermore, the perceived satisfaction, perceived trust, and perceived effectiveness were higher in supplementary conversations than in contributory conversations. No significant difference was observed in personalized answers based on long-term memory. One of the contributions of this study is that the characteristic of a CA to understand the context can improve UX. Furthermore, by understanding the effect of the prompts uttered by CAs that can influence the user’s judgment on UX, the research on the social relationships of CAs can be further advanced.