Intent Taxonomy Development for Conversational AI

Description

Builds a hierarchical, mutually exclusive taxonomy of what users want from a conversational system, grounded in real utterances and validated for coder agreement.

Intent taxonomy development treats the set of user goals a conversational system must handle as a qualitative coding problem. Researchers sample real utterances from transcripts, search logs or support tickets, code them inductively, and organise codes into a hierarchy of intent families and specific intents with written definitions, inclusion and exclusion criteria, and example utterances. The taxonomy is tested by independent coders for agreement (e.g. Cohen's kappa above 0.7), with confusable pairs refined or merged. Frequency counts show coverage and the long tail, guiding which intents to support. Grounding in dialogue act theory helps distinguish what users want from how they phrase it. Rigour comes from real data, reliability testing and periodic re-coding as usage evolves.

Attributes

Loss type
Unmigrated
Data
Mixed
Effort
Weeks

Steps

  1. Step 1: Collect 500+ real user utterances from existing channels (support transcripts, search logs, app store reviews).
  2. Step 2: Open card sort of utterances — what does this user want to do?
  3. Step 3: Cluster into intent categories.
  4. Step 4: Test taxonomy completeness: for a new set of 100 utterances, can every one be classified into an existing intent?
  5. Coverage < 80% = taxonomy is incomplete.
  6. Step 5: Validate with users: show them the intent labels — do they recognize them as their goals?

Sources

  • Chatbot Research and Design — (various)

Also known as

  • Intent modelling
  • Utterance taxonomy
  • Conversational intent classification

Categories

  • Content
  • Linguistics, Semiotics & Discourse