AI Explainability Auditing for UX

Description

Audits whether AI explanations help users understand, calibrate trust in and correct AI outputs, testing explanation types against users' actual questions and decisions.

Machine learning teams generate explanations such as feature importance, SHAP values, LIME local explanations, saliency maps and counterfactuals mainly for developers. An explainability audit asks whether these serve users. It starts by eliciting users' explanation needs, for example why this output, why not another, what would change it, and how confident the system is, using Liao's XAI question bank. It then evaluates candidate explanations on comprehension, decision accuracy, trust calibration (do users rely on the AI when it is right and override it when wrong), time and perceived usefulness, ideally in controlled studies with known correct and incorrect AI outputs. Explanations that increase trust without improving calibration are flagged as harmful. Regulatory requirements, such as the right to explanation under GDPR and the EU AI Act, add urgency.

Attributes

Loss type
Unmigrated
Data
Mixed
Effort
Weeks

Steps

  1. Explainability UX audit:
  2. Identify every AI decision point visible to users.
  3. (2) For each, apply: a SHAP or LIME explanation (which features drove this decision?).
  4. (3) Translate the technical explanation into user-facing language.
  5. (4) Test: do users understand the explanation? Can they predict how to change the outcome? User explainability score = % who can accurately predict what change would change the AI's output.

Sources

  • AI for Humans — Jeff Heaton
  • Ribeiro, Singh & Guestrin, 'Why should I trust you?' Explaining the predictions of any classifier (KDD 2016)
  • Liao, Gruen & Miller, Questioning the AI: Informing design practices for explainable AI user experiences (CHI 2020)
  • Lundberg & Lee, A unified approach to interpreting model predictions (NeurIPS 2017)

Also known as

  • XAI audit
  • Explanation usability evaluation
  • Human-centred explainability evaluation

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning