Description
Systematically sample live chatbot introductions and code identity, competence, conversational, communication and proactive cues with a reliability-tested codebook.
A chatbot's opening screen sets users' expectations of what it is and what it can do. This audit applies content analysis to that moment across a sector. The framework groups cues into identity (disclosure label, avatar type, name type), competence (claims of expertise or admissions of limits), conversational (greeting, self-introduction, giving the turn), communication (explaining purpose and how to interact) and, added during coding, proactive cues (privacy disclaimers, proactive information, handover to a human). Introductions are sampled by listing sectors and competing organisations, screenshotting the first screen without interacting, and coding header, welcome messages and response options. Coders train on a joint set, revise the codebook, then a subset is double-coded for Krippendorff's alpha. The original study of 88 introductions found humanlike avatars common, but purpose explained in only 12.5 percent and handover offered in 18 percent. Codebook and sample are on OSF.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Siloed
- Stage
- Discovery, Evaluation
- Data
- Qualitative
- Effort
- Weeks
Steps
- List sectors, the organisations in each and their competitors, and locate each customer-service chatbot on home, contact or support pages.
- Screenshot the first screen without interacting (interact only to confirm an unlabelled agent is a bot) and stop at saturation.
- Code header, welcome messages and response options with the codebook: avatar and type, name and type, disclosure, greeting, self-introduction, competence or limitation, purpose, interaction instructions, turn-giving, handover, privacy disclaimer, proactive information, response option type, number of buttons and character limits.
- Train coders on examples and a jointly coded set of about 10 introductions, resolving differences and revising the codebook.
- Split the sample among coders and have an independent coder double-code a subset (about 25) to compute Krippendorff's alpha per cue.
- Report cue prevalence and describe how organisations combine cues, noting cues with low alpha.
Sources
- Chatbot Research and Design: CONVERSATIONS 2022 (Følstad et al., 2023)
Also known as
- Social cues in chatbot introductions codebook
- Social cue framework and content analysis of chatbot introductions
Categories
- User Research
- Usability & Evaluation