Automated Usability Evaluation

Description

Software checks interface code against coded guidelines, flags items that need human manual checks, and reports compliance and interface statistics at scale.

Automated evaluation tools read interface code quickly and report conformance with coded guidelines, often suggesting fixes, along with statistics such as number of fonts, average font size, button size, menu depth and image load time. They cannot judge meaning: a tool can confirm an image has alternative text but not whether the text is meaningful, so where a criterion needs judgement the tool flags a manual check for a human reviewer. Tools suit large page sets with little time, and also high-level surveys across many sites, such as comparing accessibility features across government agencies. Rigour comes from combining the automated pass with systematic completion of the flagged manual checks and recording which guideline version was applied. Typical tools include WAVE, AChecker and Deque products.

Attributes

Loss type
Active
Effort
Hours

Steps

  1. Choose the tool and guideline set (for example WCAG 2.x level AA) and define the pages or screens in scope.
  2. Run the automated tool across the full set.
  3. Review the summary report, interface statistics and any suggested fixes.
  4. Carry out every manual check the tool flags, such as meaningfulness of alt text and labels.
  5. Fix failures, then rerun the tool to confirm.
  6. Optionally aggregate results across many sites or releases for a high-level picture.

Sources

  • Research Methods in HCI (Lazar et al., 2017)

Also known as

  • Automated usability testing with manual checks
  • Automated accessibility testing
  • Automated guideline checking

Categories

  • User Research
  • Usability & Evaluation