Description
Check a model by perturbing test inputs, swapping sensitive attributes and making changes with known expected effects, to see whether outputs behave robustly, fairly and sensibly before launch.
Aggregate accuracy on a held-out set says little about how a model will behave with real users. Behavioural testing probes specific expectations. Perturbation tests add realistic noise to test inputs, such as background noise, clipped audio, typos or poor lighting, and favour the model that degrades least. Invariance tests change only sensitive information, such as name, race or gender, while holding everything else fixed; outputs should not change, and if they do, the sensitive features or their proxies should be removed. Directional expectation tests apply changes whose effect direction is known, such as a larger lot size raising a predicted house price, and investigate any reversal. The tests are written from domain and user knowledge, which is where UX researchers contribute: they know which real-world variations, user groups and expectations matter. A model sensitive to small perturbations is hard to maintain and open to adversarial attack.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- List the real-world input variations users will produce (accents, noise, typos, device cameras, phrasing) and create perturbed copies of the test set that mimic them.
- Compare candidate models on the perturbed data as well as the clean test set.
- For sensitive attributes, swap values while holding other inputs constant and check that outputs do not change.
- Define input changes with known expected output directions and check that predictions move accordingly.
- Log and investigate every violation, fixing data, features or model before deployment.
- Keep the test suite and rerun it on every model update.
Sources
- Designing Machine Learning Systems (Huyen, 2022)
Also known as
- Behavioural testing of ML models (perturbation, invariance, directional expectation tests)
- Perturbation tests
- Invariance tests
- Directional expectation tests
- Model behavioural tests
- CheckList-style testing
Categories
- Analytics & Behaviour
- Computational & Machine Learning