Data Adequacy Model Set

Description

Build many small random models on training, test and evaluation splits and compare their behaviour; if performance differs across splits, the data are not adequate for modelling.

Before investing in a predictive model, Pyle checks whether the data are sufficient and representative. The data are split into training, test and evaluation sets. A set of quick models is built, each on a random group of about 11 inputs with one randomly chosen variable as the output, trained on the training set and scored on all three sets. If the models behave similarly across sets, the splits represent the same population and hold enough data; marked differences mean the data are insufficient or unrepresentative and more data or a new sample is needed. It is a rough, model-agnostic sanity check, and Pyle gives no threshold for 'dissimilar'.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Hours

Steps

  1. Split the data into training, test and evaluation sets.
  2. Draw random groups of about 11 variables, each with one randomly chosen output variable.
  3. Train a quick model for each group on the training set.
  4. Score every model on all three sets and compare performance distributions.
  5. Treat consistently dissimilar performance as a sign of inadequate or unrepresentative data and collect more or resample.

Sources

  • Handbook of Data Mining (Ye, 2003)

Also known as

  • DAMS
  • Data sufficiency check

Categories

  • Analytics & Behaviour
  • Statistics & Modelling