Evidence-Graded Guideline Development

Description

Build a design guideline set from research and practice, then have expert panels rate each guideline's importance and the strength of its supporting evidence on defined scales.

HHS argued that most guideline sets rest on a few experts' opinions, without references, consensus or any sign of relative importance. Its procedure borrowed evidence grading from medicine. More than 500 candidate guidelines were culled by internal review, rated for importance by 16 external reviewers (half designers, half usability specialists), and rated for strength of evidence by eight doctoral-level researchers who first defined a five-point scale running from strong research support (5) to weak expert opinion (1). Guidelines were illustrated, organised into chapters by a 20-person card sort and the print layout was usability tested. The 2006 update re-surveyed importance and re-rated evidence (Cronbach's alpha .92). Ratings let managers mandate the top guidelines and show researchers where evidence is thin, and low-evidence guidelines indicate where testing effort should go.

Attributes

Loss type
Extinct
Stage
Strategy, Design
Effort
Months

Steps

  1. Gather candidate guidelines from style guides, published research, public usability reports and in-house test lessons.
  2. Review internally: merge duplicates, resolve conflicts and reword unclear guidelines.
  3. Have external reviewers (about half designers, half usability specialists) rate each guideline's importance to the product's success; drop low-importance items.
  4. Have a panel of research-qualified raters agree evidence criteria, form a 5-point strength-of-evidence scale (5 strong research support to 1 weak expert opinion) and rate each guideline.
  5. Illustrate each guideline with examples and organise the set, for example by card sort, then usability test the published format.
  6. Update continuously: search the literature, revise guidelines, re-rate importance and evidence, and check inter-rater agreement (e.g. Cronbach's alpha).

Sources

  • Research-Based Web Design & Usability Guidelines (HHS, 2006)

Also known as

  • Evidence-Graded Guideline Development (Relative Importance and Strength of Evidence Ratings)
  • Strength of Evidence rating
  • Relative Importance rating
  • Research-based guideline grading
  • HHS guideline rating procedure

Categories

  • Research Operations
  • Research Design & Rigor