Description
Audits system messages and conversational turns against Grice's maxims of quantity, quality, relation and manner to find where the system breaks conversational expectations.
Grice proposed that conversation rests on a cooperative principle, expressed as four maxims: say as much as needed and no more (quantity), say what you believe true and have evidence for (quality), be relevant (relation), and be clear, brief and orderly (manner). Listeners infer meaning, called implicature, when a speaker appears to flout a maxim. Users apply the same expectations to interfaces and especially to chatbots and AI assistants. A pragmatics audit codes each system utterance against the maxims and also looks for unintended implicatures, such as a vague error implying user fault. Rigour comes from coding transcripts with two coders, rating severity, and validating with users by asking what they inferred from messages. It is particularly useful for diagnosing why technically correct responses feel unhelpful or evasive.
Attributes
- Discipline
- Linguistics, Semiotics & Discourse
- Loss type
- Siloed
- Stage
- Design, Evaluation
- Data
- Qualitative
- Effort
- Days
Steps
- Collect system utterances: error messages, microcopy and samples of chatbot or assistant transcripts (e.g. 100-300 turns).
- Code each utterance for violations of quantity, quality, relation and manner, noting the likely implicature.
- Have a second coder code at least 20% and reconcile disagreements.
- Test a sample of flagged utterances with 6-10 users, asking what they understood and inferred.
- Rewrite violating utterances and prompt or response policies to satisfy the maxims.
- Re-evaluate with users or by measuring downstream outcomes such as resolution rates and repeat queries.
Sources
- Conversational Design
- The Development of Language
- Logic and Conversation (H. Paul Grice)
Also known as
- Pragmatics of Conversation
- Grice's maxims audit
- Cooperative principle analysis
Categories
- Content
- Linguistics, Semiotics & Discourse