Classical Item Analysis: Difficulty and Discrimination Indices

Description

Try out a draft test or scale, compute each item's difficulty and its power to separate high and low total scorers, and keep the items that discriminate best.

Classical item analysis prunes a surplus pool of draft items using tryout data. After expert review, the preliminary form goes to a representative tryout sample, respondents are ranked by total score, and upper and lower groups are formed: the top and bottom 27% (Kelley), or thirds or quarters (Ebel). Difficulty is the percentage answering correctly; in achievement tests items answered correctly by under about 33% or over about 67% are usually dropped. Discrimination is (U - L) divided by the size of one group, where U and L are the numbers in each group answering correctly or endorsing the item. Items above a critical value are retained (.40 is often cited as very good, though thresholds vary). Item-total correlations serve the same purpose for rating scales. Qualitative review, reliability, validity and norms follow before the final form is assembled. Small samples give unstable indices, and difficulty is rarely relevant for attitude scales.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Draft surplus items and review them with experts.
  2. Administer the preliminary form to a representative tryout sample.
  3. Score and rank respondents by total score.
  4. Form upper and lower groups (top and bottom 27%, or thirds or quarters).
  5. Compute each item's difficulty (proportion correct or endorsing) and discrimination index (U - L)/n, or item-total correlations for rating items.
  6. Rank items and reject those below the critical discrimination level or outside the target difficulty range.
  7. Review retained items qualitatively, assemble the final form, and establish reliability, validity and norms.

Sources

  • Research Methodology in Behavioural Sciences (Mangal & Mangal, 2013)
  • How to Develop a Good Survey Scale in Quantitative UX Research? (Zhaoyang, 2024)

Also known as

  • Item analysis
  • Item difficulty index
  • Item discrimination index
  • Upper and lower groups method
  • Kelley 27 percent groups

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics