Description
Justify interpreting an existing metric as a construct with evidence on content, response process, internal structure, relations to other variables and consequences, iterating on weak assumptions.
Validity belongs to an interpretation and use, not to a measure. Argument-based validation, from Messick and Kane, builds a case for using an existing metric as evidence of something. Content evidence asks whether the measure covers the construct and excludes unwanted variance, for example aggregates that mix unrelated components. Response-process evidence asks whether the data were generated by a stable process: incentives to misreport, policy changes that altered recording, independent raters, units based on few records (weight by the square root of record counts), and imputation by the source. Internal structure checks factor patterns among multiple indicators. Relations to other variables test whether correlations match theory, ideally across sources. Consequences ask what happens when the metric is used, for example in rankings. Kane's procedure is to state the interpretation, gather evidence on the weakest assumptions, and iterate.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Stage
- Strategy, Post-launch
- Data
- Analytical / expert
- Effort
- Weeks
Steps
- State the intended interpretation and use of the measure (for example 'support tickets per user indicate task difficulty').
- Gather content evidence: does the measure cover the construct, and what unrelated variance does it include? Check definitions with the data owner.
- Examine the response process: incentives, policy or tracking changes, rater independence, small-count units and source imputation.
- Check internal structure where several indicators exist, watching for near-perfect correlations or group differences.
- Test relations to other variables against theory, preferably using independent sources.
- Consider the consequences of using the measure, such as gaming once it becomes a target.
- Collect further evidence on the weakest assumptions and iterate, documenting the argument.
Sources
- Cambridge Handbook of Research Methods (Nichols & Edlund, 2023)
- Validating the Interpretations and Uses of Test Scores (Kane, 2013)
- The Practice of Social Research (Babbie, 2013)
Also known as
- Validity argument for archival data
- Messick validity facets for secondary data
- Metric validation
Categories
- Research Operations
- Research Design & Rigor